Livestock and poultry behavior recognition method and system based on deep learning multi-modal time sequence fusion

Abstract

This invention discloses a method and system for livestock and poultry behavior recognition based on deep learning multimodal temporal fusion, relating to the field of livestock and poultry behavior recognition. The method includes: acquiring temporal sensing data and visual data to construct a multimodal temporal dataset; extracting motion sensing temporal features and visual posture temporal features for each individual livestock and poultry to obtain a multimodal behavior representation vector; constructing an inter-individual association graph and obtaining the group context-enhanced feature sequence of each individual livestock and poultry through a graph attention network; decoding and outputting the behavior category label of each individual livestock and poultry at each time step through a temporal decoding network; constraining the model using a group behavior consistency regularization term to obtain the final behavior recognition result; and constructing a comprehensive loss function to perform end-to-end joint training of all trainable network modules in the livestock and poultry behavior recognition model. This method enables behavior recognition of individual livestock and poultry under dynamic group context enhancement, effectively solving the problem of livestock and poultry behavior recognition and analysis in complex farming scenarios.

Description

Technical Field

[0001] This invention relates to the field of livestock and poultry behavior recognition, specifically to a method and system for livestock and poultry behavior recognition based on deep learning multimodal temporal fusion. Background Technology

[0002] With the increasing scale and intensification of livestock and poultry farming, the demand for precise monitoring of individual livestock and poultry health status and production performance is becoming increasingly urgent. Livestock and poultry behavior patterns directly reflect their physiological state and welfare level. For example, abnormal feeding behavior can predict digestive system diseases, reduced activity may indicate limb or hoof injuries or fever symptoms, and changes in respiratory rate are closely related to respiratory infections. Therefore, intelligent health monitoring based on behavior recognition has become a key technological means in modern precision farming. Currently, the mainstream technological approaches mainly fall into two categories: behavior monitoring based on wearable sensors and behavior recognition based on computer vision. Wearable sensors, such as accelerometers and gyroscopes, can continuously collect kinematic data of individuals, offering advantages such as high sampling frequency and excellent data accuracy. However, relying solely on wearable data makes it difficult to obtain information on the spatial position and social interaction of individuals within a group. Visual sensors, such as RGB cameras and depth cameras, can capture the limb posture, movement trajectory, and spatial distribution of livestock and poultry. However, in scenarios with severe occlusion, large changes in lighting, and similar individual appearances, maintaining individual identity and continuous tracking face significant challenges. The limitations of both indicate that single-modal data is insufficient to support the high-precision and robust behavior recognition requirements in complex farming scenarios.

[0003] In existing technologies, most multimodal fusion methods for livestock and poultry behavior recognition employ simple early or late fusion strategies. This involves directly concatenating features from different modalities and inputting them into a classifier, or using a voting process to determine the independent recognition results of each modality. Neither of these methods fully exploits the temporal dependencies and complementary features between modalities. Especially in group farming scenarios, individual behavior recognition depends not only on its own sensor data but also on the behavioral states of neighboring individuals. Existing methods lack effective modeling of this dynamic group interaction. Furthermore, existing technologies often employ fixed-time-window statistical features for temporal feature extraction, such as mean, variance, and peak frequency. These artificially designed features struggle to capture the nonlinear temporal dependencies and long-range correlations in behavioral patterns, thus limiting recognition accuracy. Summary of the Invention

[0004] To address the aforementioned technical challenges, this paper presents a method and system for livestock and poultry behavior recognition based on deep learning-based multimodal temporal fusion. This technical solution constructs a hierarchical fusion architecture for individual behavior recognition and group interaction perception, combining temporal attention mechanisms and graph neural networks to achieve group-level behavior reasoning based on accurate individual-level perception. This effectively solves the problems of deep fusion modeling of multimodal temporal data and accurate recognition of individual behavior under the influence of group interaction.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for livestock and poultry behavior recognition based on deep learning multimodal temporal fusion includes:

[0007] By deploying multimodal data acquisition devices on individual livestock and poultry and in the breeding environment, the temporal sensing data of each individual livestock and poultry and the visual data of the group scene are acquired simultaneously. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed.

[0008] Based on the multimodal temporal dataset, motion sensing temporal features and visual posture temporal features are extracted for each individual livestock and poultry. The two are then adaptively weighted and fused through temporal attention fusion to obtain the multimodal behavior representation vector of each individual livestock and poultry.

[0009] Based on the spatial location information of each livestock and poultry individual in the breeding scenario, an inter-individual association graph is constructed based on spatial distance and relative movement between individuals. The multimodal behavior representation vector of each livestock and poultry individual is used as the initial feature of the node. Group-level interactive information is propagated and aggregated through a graph attention network to obtain the group context-enhanced feature sequence of each livestock and poultry individual.

[0010] Based on the group context-enhanced feature sequence of each livestock and poultry individual, the behavior category label of each livestock and poultry individual at each time step is decoded and output through a temporal decoding network. The model is constrained during the training phase using a group behavior consistency regularization term, and the output result is corrected during the inference phase using a smoothing post-processing strategy to obtain the final behavior recognition result. The livestock and poultry behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence.

[0011] A comprehensive loss function is constructed, and all trainable network modules in the livestock and poultry behavior recognition model are jointly trained end-to-end based on the labeled sample set. The trained recognition model is then deployed in an online recognition system.

[0012] Preferably, the step of simultaneously acquiring the temporal sensing data of each individual livestock and poultry and the visual data of the group scene, performing spatiotemporal alignment and preprocessing, and constructing a multimodal temporal dataset includes:

[0013] Each individual livestock and poultry is equipped with a wearable sensor that integrates an accelerometer, a gyroscope, and a UWB positioning module. The sensor continuously collects individual triaxial acceleration data and triaxial angular velocity data at a preset sampling frequency to form motion sensing time-series data.

[0014] A visual acquisition device consisting of a depth camera and an RGB camera deployed above the breeding pens continuously acquires depth and color images of the group scene at the same sampling frequency as wearable sensors.

[0015] Wearable sensors and visual acquisition devices are synchronized with a unified time server to ensure that motion sensing data and visual data correspond one-to-one in the time dimension at each sampling moment.

[0016] The collected triaxial acceleration and triaxial angular velocity data are used to generate sensing time segments with a fixed time window length by using a sliding window truncation method.

[0017] The acquired depth and color images are aligned with the coordinates based on preset camera intrinsic and extrinsic parameters, so that the depth and color images correspond one-to-one in spatial pixel position.

[0018] For each sensing time segment within a time window, a Kalman filter algorithm is used for smoothing, and the filtered triaxial acceleration data is normalized to zero mean by window, mapping the acceleration values ​​to a distribution interval centered at zero.

[0019] For the depth and color images within each time window, extract the two-dimensional bounding box and instance segmentation mask of each individual livestock and poultry in each frame image, and obtain the three-dimensional spatial coordinates of each individual livestock and poultry.

[0020] The two-dimensional pose estimation model is used to extract the two-dimensional pose key point coordinate sequence of each livestock individual from color images;

[0021] Based on the wearable sensor device number of each livestock and poultry individual and the individual identification obtained by visual detection, the correspondence between the sensor data and visual data in the same time window in each livestock and poultry individual dimension is established by the nearest neighbor matching algorithm to complete the data association;

[0022] The motion sensing time series, visual posture time series, and three-dimensional spatial position coordinate sequence of each individual livestock and poultry are combined to form a multimodal time series sample, which constitutes a multimodal time series dataset.

[0023] Preferably, the step of extracting motion sensing temporal features and visual posture temporal features for each individual livestock and poultry, and adaptively weighting and fusing the two through temporal attention fusion to obtain the multimodal behavioral representation vector of each individual livestock and poultry includes:

[0024] Extract motion sensing time-series segments and visual pose time-series segments of any livestock or poultry individual within a single time window from a multimodal time-series dataset;

[0025] The motion sensing time sequence segment is a multi-channel time sequence signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first time sequence coding network.

[0026] The first temporal coding network is constructed based on a multi-layer gated recurrent unit. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window.

[0027] The visual pose temporal segment is a sequence of two-dimensional pose key point coordinates. The coordinates of each key point at each time step are flattened and stitched together before being input into the second temporal coding network.

[0028] The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window.

[0029] Construct a temporal attention fusion, taking the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs, and calculate the attention weights of the two modalities at the current time step by step.

[0030] The temporal feature sequence of motion sensing and the temporal feature sequence of visual posture are multiplied by the attention weights at the corresponding time points and then added element by element to obtain the temporal step-by-step multimodal behavior representation sequence of the livestock individual.

[0031] Preferably, the step of constructing an inter-individual association graph based on the spatial location information of each livestock and poultry individual in the breeding scenario, and using the multimodal behavior representation vector of each livestock and poultry individual as the initial node feature, and obtaining the group context-enhanced feature sequence of each livestock and poultry individual through a graph attention network for group-level interaction information propagation and aggregation includes:

[0032] For each moment within the same time window, the set of all livestock and poultry individuals at that moment is taken as the graph node set, and the Euclidean distance between any two nodes is calculated based on the three-dimensional spatial coordinates at that moment.

[0033] A spatial distance threshold is preset. When the Euclidean distance between any two nodes is less than the threshold, an undirected edge is established between them.

[0034] Based on the spatial proximity and relative motion state between two nodes, the interaction strength weight between any established node pair is defined at each time step, and the weights of all node pairs constitute the adjacency matrix at the corresponding time step.

[0035] The multimodal behavior representation vectors of all nodes at this moment are used as the initial node features of the graph attention network;

[0036] The graph attention network employs a multi-head attention mechanism, performing message passing time-by-time. In each layer, for any central node at that time, a linear transformation is performed on the node features at that time. The set of its neighboring nodes is determined by the non-zero element index of the corresponding row in the adjacency matrix at that time. Attention coefficients are calculated for each neighboring node through the attention mechanism.

[0037] Using the attention coefficient as the weight, the transformed features of all neighboring nodes of the central node are weighted and summed, and then nonlinearly activated to obtain the updated features of the central node under the current head.

[0038] Concatenate the outputs of all headers to complete the message passing for this layer;

[0039] After repeating the message passing process for a preset number of layers, the updated features output by the attention network layer of each graph are taken and residually connected and spliced ​​to obtain the group context enhancement feature sequence of each livestock individual at each time step.

[0040] Preferably, the step of decoding and outputting the behavior category label of each livestock individual at each time step based on the group context enhancement feature sequence of each individual livestock, using a temporal decoding network, and constraining the model during the training phase using a group behavior consistency regularization term, and correcting the output results during the inference phase using a smoothing post-processing strategy, to obtain the final behavior recognition result includes:

[0041] The group context enhancement feature sequence of each livestock and poultry individual is input into the temporal decoding network. The temporal decoding network is constructed based on a bidirectional gated recurrent unit. It performs bidirectional encoding on the group context enhancement feature sequence along the time step direction. After splicing the forward and backward hidden states of each time step, it is mapped to the probability distribution space of the preset number of behavior categories through a linear classification layer, and outputs the probability value of each livestock and poultry individual belonging to each preset behavior category at each time step.

[0042] The behavior category with the highest probability value at each moment is used as the initial decoding label for that moment, thus obtaining the initial behavior category sequence for each livestock individual at each moment within the time window;

[0043] Based on the idea of ​​graph Laplace regularization, a group behavior consistency regularization term is constructed. Spatial proximity weight is used to measure the interaction strength of any two livestock individuals at each time step, and KL divergence is used to measure the difference in the predicted probability distribution of two nodes. The sum of the weighted probability distribution differences of all nodes with spatial proximity relationships at all times is calculated as the group consistency loss.

[0044] During the inference phase, a post-processing correction strategy based on conditional random fields is constructed, defining an energy function that includes unary and binary potentials. The unary potential is based on the negative logarithm of the probability output of the decoding network, while the binary potential is based on spatial proximity weights to penalize inconsistent labels of adjacent nodes.

[0045] The energy function is minimized by the belief propagation algorithm, and the corrected behavior category sequence is obtained as the final behavior recognition result.

[0046] Preferably, the construction of the comprehensive loss function, which involves end-to-end joint training of all trainable network modules in the livestock and poultry behavior recognition model based on a labeled sample set, and deployment of the trained recognition model in an online recognition system, includes:

[0047] Obtain a set of manually labeled livestock and poultry behavior recognition samples. Each training sample contains multimodal time-series data within a time window and the real behavior category label of each individual livestock and poultry at each moment within that window.

[0048] Construct a comprehensive loss function, which is a weighted sum of behavior classification loss and group consistency loss;

[0049] The backpropagation algorithm is adopted to minimize the comprehensive loss function. All trainable network modules of the livestock and poultry behavior recognition model are jointly trained end-to-end on the training sample set, and the parameters with the minimum comprehensive loss on the validation set are selected as the final model parameters.

[0050] The trained livestock and poultry behavior recognition model is integrated and packaged into an online recognition model and deployed in an online recognition system;

[0051] In actual operation, the online recognition model receives streaming data from multimodal data acquisition in real time, performs spatiotemporal alignment, feature extraction, fusion and decoding, and outputs the real-time behavior recognition results of each individual livestock and poultry.

[0052] Furthermore, this solution proposes a livestock and poultry behavior recognition system based on deep learning multimodal temporal fusion, used to implement the livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion as described above, including:

[0053] The data acquisition module is used to simultaneously acquire the temporal sensing data of each individual livestock and poultry and the visual data of the group scene through multimodal data acquisition devices deployed in individual livestock and poultry and the breeding environment. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed.

[0054] The individual representation module is used to extract motion sensing temporal features and visual posture temporal features for each livestock individual based on a multimodal temporal dataset, and to adaptively weight and fuse the two through temporal attention fusion to obtain the multimodal behavior representation vector of each livestock individual.

[0055] The group aggregation module is used to construct an inter-individual association graph based on the spatial location information of each livestock and poultry individual in the breeding scenario, based on spatial distance and relative movement between individuals, and to use the multimodal behavior representation vector of each livestock and poultry individual as the initial feature of the node. Through the graph attention network, group-level interaction information is propagated and aggregated to obtain the group context-enhanced feature sequence of each livestock and poultry individual.

[0056] The decoding and correction module is used to enhance the feature sequence of each livestock individual based on the group context, decode and output the behavior category label of each livestock individual at each time step through the temporal decoding network, and use the group behavior consistency regularization term to constrain the model during the training phase. During the inference phase, the output result is corrected by combining the smoothing post-processing strategy to obtain the final behavior recognition result. The livestock behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence.

[0057] The training and deployment module is used to construct a comprehensive loss function, perform end-to-end joint training on all trainable network modules in the livestock and poultry behavior recognition model based on the labeled sample set, and deploy the trained recognition model on the online recognition system.

[0058] Preferably, the individual representation module includes:

[0059] The segment extraction unit is used to extract motion sensing time-series segments and visual pose time-series segments of any livestock individual within a single time window from a multimodal time-series dataset.

[0060] The motion sensing temporal feature unit is used when the motion sensing temporal segment is a multi-channel temporal signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first temporal coding network. The first temporal coding network is constructed based on a multi-layer gated recurrent unit and propagates forward along the time step direction in sequence, outputting the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window.

[0061] The visual posture temporal feature unit is used because the visual posture temporal segment is a sequence of two-dimensional posture key point coordinates. The coordinates of each key point at each time step are flattened and spliced, and then input into the second temporal coding network. The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window.

[0062] The attention fusion unit is used to construct temporal attention fusion. It takes the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs and calculates the attention weights of the two modalities at the current time step by step.

[0063] The multimodal behavior representation unit is used to multiply the time-step motion sensing temporal feature sequence and the visual posture temporal feature sequence by the attention weights at the corresponding time points, and then add them element by element to obtain the time-step multimodal behavior representation sequence of the livestock individual.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] This solution overcomes the information bottleneck of single modality in group farming scenarios by employing a multimodal data acquisition strategy that integrates wearable sensors and vision. It utilizes a deep temporal coding network (GRU / LSTM) to replace manual statistical features, fully exploring the nonlinear temporal dependencies in behavioral patterns. Furthermore, it achieves dynamic adjustment of modal contribution through a time-step adaptive attention fusion mechanism, overcoming the limitation of fixed fusion strategies in adapting to behavioral diversity. Building upon this, and considering the influence of neighbors on individual behavior in group farming scenarios, it dynamically constructs inter-individual association graphs at each time step and introduces a graph attention network for group interaction information propagation. Simultaneously, by adding self-loops to the adjacency matrix, it ensures the preservation of individual features during aggregation, effectively addressing recognition difficulties caused by occlusion and individual similarity. Moreover, by introducing a group consistency regularization term based on KL divergence during the training phase and employing conditional random fields for global smoothing correction during the inference phase, it forms a complete spatial constraint link from feature-level fusion to decision-level correction, effectively improving the accuracy and robustness of livestock behavior recognition in group farming scenarios. Attached Figure Description

[0066] Figure 1 This is a flowchart of the livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion of the present invention;

[0067] Figure 2 The present invention extracts motion sensing temporal features and visual posture temporal features for each individual livestock and poultry, and performs adaptive weighted fusion of the two through temporal attention fusion to obtain the multimodal behavior representation vector of each individual livestock and poultry.

[0068] Figure 3 The flowchart of the present invention is as follows: using graph attention network to propagate and aggregate group-level interactive information to obtain the group context-enhanced feature sequence of each livestock individual.

[0069] Figure 4 The flowchart of the present invention is as follows: the behavior category label of each individual animal is decoded and output by a temporal decoding network at each moment, and the model is constrained by a group behavior consistency regularization term to obtain the final behavior recognition result. Detailed Implementation

[0070] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0071] Reference Figure 1 As shown, a method for livestock and poultry behavior recognition based on deep learning multimodal temporal fusion includes:

[0072] By deploying multimodal data acquisition devices on individual livestock and poultry and in the breeding environment, the temporal sensing data of each individual livestock and poultry and the visual data of the group scene are acquired simultaneously. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed.

[0073] Based on the multimodal temporal dataset, motion sensing temporal features and visual posture temporal features are extracted for each individual livestock and poultry. The two are then adaptively weighted and fused through temporal attention fusion to obtain the multimodal behavior representation vector of each individual livestock and poultry.

[0074] Based on the spatial location information of each livestock and poultry individual in the breeding scenario, an inter-individual association graph is constructed based on spatial distance and relative movement between individuals. The multimodal behavior representation vector of each livestock and poultry individual is used as the initial feature of the node. Group-level interactive information is propagated and aggregated through a graph attention network to obtain the group context-enhanced feature sequence of each livestock and poultry individual.

[0075] Based on the group context-enhanced feature sequence of each livestock and poultry individual, the behavior category label of each livestock and poultry individual at each time step is decoded and output through a temporal decoding network. The model is constrained during the training phase using a group behavior consistency regularization term, and the output result is corrected during the inference phase using a smoothing post-processing strategy to obtain the final behavior recognition result. The livestock and poultry behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence.

[0076] A comprehensive loss function is constructed, and all trainable network modules in the livestock and poultry behavior recognition model are jointly trained end-to-end based on the labeled sample set. The trained recognition model is then deployed in an online recognition system.

[0077] To achieve accurate identification of livestock and poultry behavior, a modal data foundation containing complete kinematic and visual information is needed. Therefore, this solution synchronously collects multimodal data using wearable sensors and visual acquisition devices. After spatiotemporal alignment and data association, a multimodal temporal dataset is constructed to address the root cause of insufficient information from a single modality. Based on this, the motion sensing data and visual pose data of each individual livestock and poultry are independently encoded with temporal features, and deep temporal representations of the two modalities are extracted separately. Adaptive weighted fusion is then performed by calculating the contribution weights of each modality to the current behavior recognition through temporal attention fusion, enabling the model to dynamically adjust modal dependencies according to behavior type. This solves the problem of insufficient feature discrimination caused by the inability of traditional fixed fusion strategies to adapt to behavioral diversity. Finally, the multimodal behavior representation vectors of each individual livestock and poultry are used as the initial features of the nodes. By constructing an inter-individual association graph based on spatial distance and interaction intensity, and utilizing a graph attention network to achieve the propagation and aggregation of group-level interaction information, the feature representation of each individual can absorb the behavioral state information of neighboring individuals, thereby solving the recognition difficulties caused by occlusion or individual similarity in group farming scenarios at the feature level. Furthermore, a preliminary behavior category sequence is output through a temporal decoding network, and the recognition results are spatially correlated and corrected using group behavior consistency constraints, fully exploring the spatial prior rules of livestock and poultry group behavior to improve the consistency and reliability of the results. Finally, by constructing a comprehensive loss function that includes behavior classification loss and group consistency loss, end-to-end joint training is achieved on a labeled sample set to ensure the collaborative adaptation of each module. The trained model is then deployed on an online system to complete real-time recognition, thus forming a complete technical closed loop from data collection to model deployment.

[0078] Specifically, the temporal sensing data of each individual livestock and poultry and the visual data of the group scene are acquired synchronously. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed, including:

[0079] Each individual livestock and poultry is equipped with a wearable sensor that integrates an accelerometer, a gyroscope, and a UWB positioning module. The sensor continuously collects individual triaxial acceleration data and triaxial angular velocity data at a preset sampling frequency to form motion sensing time-series data.

[0080] A visual acquisition device consisting of a depth camera and an RGB camera deployed above the breeding pens continuously acquires depth and color images of the group scene at the same sampling frequency as wearable sensors.

[0081] Wearable sensors and visual acquisition devices are synchronized with a unified time server to ensure that motion sensing data and visual data correspond one-to-one in the time dimension at each sampling moment.

[0082] The collected triaxial acceleration and triaxial angular velocity data are used to generate sensing time segments with a fixed time window length by using a sliding window truncation method.

[0083] The acquired depth and color images are aligned with the coordinates based on preset camera intrinsic and extrinsic parameters, so that the depth and color images correspond one-to-one in spatial pixel position.

[0084] For each sensing time segment within a time window, a Kalman filter algorithm is used for smoothing, and the filtered triaxial acceleration data is normalized to zero mean by window, mapping the acceleration values ​​to a distribution interval centered at zero.

[0085] For the depth and color images within each time window, extract the two-dimensional bounding box and instance segmentation mask of each individual livestock and poultry in each frame image, and obtain the three-dimensional spatial coordinates of each individual livestock and poultry.

[0086] The two-dimensional pose estimation model is used to extract the two-dimensional pose key point coordinate sequence of each livestock individual from color images;

[0087] Based on the wearable sensor device number of each livestock and poultry individual and the individual identification obtained by visual detection, the correspondence between the sensor data and visual data in the same time window in each livestock and poultry individual dimension is established by the nearest neighbor matching algorithm to complete the data association;

[0088] The motion sensing time series, visual posture time series, and three-dimensional spatial position coordinate sequence of each individual livestock and poultry are combined to form a multimodal time series sample, which constitutes a multimodal time series dataset.

[0089] Precise alignment of multimodal data at the acquisition end is a prerequisite for subsequent fusion modeling. This requires establishing a one-to-one correspondence between different modalities in both the temporal and spatial dimensions. Specifically, in the temporal dimension, a unified time server provides a consistent clock reference for wearable sensors and visual acquisition devices, ensuring strict alignment between accelerometer sampling times and image frame exposure times. This ensures precise temporal correspondence between motion sensing data and visual data at the same timestamp, avoiding timing misalignment caused by clock drift. In the spatial dimension, spatial pixel alignment between depth maps and color maps is achieved through camera intrinsic and extrinsic parameter calibration, enabling depth information to be fully aligned. It accurately attaches to each image pixel; at the individual level, it extracts the instance segmentation mask and two-dimensional bounding box of each livestock individual from the image through the livestock individual detection model, and obtains the position coordinates of the individual in three-dimensional space by combining the depth image. Then, it uses the nearest neighbor matching algorithm to associate the device number of the wearable sensor with the individual identity identifier of the visual detection based on the spatial proximity and individual appearance characteristics, thereby establishing the correspondence between the motion sensing data and visual data of the same livestock individual, solving the technical problem of difficulty in associating and matching multi-source heterogeneous data at the individual level, and providing a data foundation for the subsequent extraction of accurate multimodal representations of each individual;

[0090] To further illustrate the approach of this scheme, the detection information, spatial location, and posture information of each individual livestock and poultry are extracted from the depth image and color image within each time window according to the following steps:

[0091] The length of the sliding window is determined based on the typical behavioral cycle duration of livestock and poultry, and is usually set to cover a time span of 2 to 5 seconds; the zero-mean normalization process makes the acceleration data distributed with zero as the center, effectively eliminating the static deviation caused by the difference in wearing position between different individuals;

[0092] The color image sequence within each time window is input frame by frame into a pre-trained livestock and poultry individual detection model. This model adopts an instance segmentation architecture based on MaskR-CNN, with ResNet-50 as the backbone network. It is pre-trained on an image dataset labeled with livestock and poultry individual bounding boxes and instance masks. For each frame of color image, the model outputs a set of detection results for all livestock and poultry individuals in that frame. Each detection result includes the individual's two-dimensional bounding box, binary instance segmentation mask, and individual identifier. The individual identifier is maintained by the SORT tracker to ensure that the same livestock and poultry individual has a consistent identifier number across different frames.

[0093] Based on the aligned depth and color images, the three-dimensional spatial coordinates of each individual livestock and poultry are calculated using the obtained instance segmentation mask. Specifically, for each individual in each frame, the set of all pixel positions covered by its segmentation mask is taken, and the depth value of the corresponding pixel position is read from the depth image. Considering that there may be noise in the depth sensor, the median depth of all effective depth pixels in the mask coverage area is used as the depth value estimate of the individual. At the same time, the weighted average pixel coordinates of all effective depth pixels in the individual mask are used as the image projection center of the individual.

[0094] Depending on the camera type, the image pixel coordinates and depth values ​​are back-projected to three-dimensional coordinates in the camera coordinate system using a pre-calibrated camera intrinsic parameter matrix. Specifically, the depth value is the Z-axis coordinate of the point in the camera coordinate system, and the image pixel coordinates, combined with the depth value and intrinsic parameters such as camera focal length and principal point, are used to calculate the X-axis and Y-axis coordinates through inverse projection transformation.

[0095] Furthermore, based on the pre-calibrated camera extrinsic parameters, namely the rotation matrix and translation vector, the coordinates in the camera coordinate system are transformed to the world coordinate system, namely the three-dimensional spatial position coordinates in the global coordinate system of the breeding pen. The origin of the world coordinate system can be set as a fixed reference point on the ground of the pen, with the Z-axis perpendicular to the ground and upward.

[0096] The color image is input into a pre-trained two-dimensional pose estimation model. The model adopts the HRNet architecture and can output a high-precision key point heatmap by maintaining high-resolution feature maps in parallel on multiple resolution branches. The model is pre-trained on an image dataset labeled with livestock and poultry pose key points and outputs two-dimensional coordinates of multiple preset semantic key points, covering parts such as head, ear roots, shoulders, spine, hips, and limb ends. For each individual detected in each frame, a sub-image of the individual region is cropped with its bounding box and input into the pose estimation model to obtain the set of pose key point coordinates of that individual.

[0097] To ensure that the pose keypoints and their 3D spatial locations belong to the same livestock individual, pose estimation is performed within the individual bounding box and is associated with the detection results through individual identification. When the pose estimation results of the same livestock individual are missing in consecutive frames due to occlusion, a linear interpolation method is used to fill in the keypoint coordinates of the missing frames. That is, based on the nearest valid pose estimation frames before and after the missing interval, linear interpolation is performed between the two according to the time ratio to obtain the estimated value of the keypoint coordinates at the missing time.

[0098] After visual information extraction is completed, for each moment within each time window, each individual livestock and poultry simultaneously possesses the following visual derived data: two-dimensional bounding box, instance segmentation mask, three-dimensional spatial position coordinates, and a set of two-dimensional pose key point coordinates. These data are indexed through individual identity identifiers.

[0099] Based on the pre-calibrated mapping relationship between the wearable sensor device number and the individual identification obtained by visual detection, the correspondence between the sensor data and visual data in each livestock individual dimension within the same time window is established through the nearest neighbor matching algorithm. Specifically, for each moment, the sensor spatial position of each livestock individual is obtained using the UWB positioning module integrated in the wearable sensor or an indoor positioning method based on sensor signal strength indication, and nearest neighbor matching is performed with the three-dimensional spatial position obtained by visual detection. The visually detected individual with the closest spatial distance is paired with the sensor individual as the same livestock individual.

[0100] After a successful match, the motion sensing time series, visual posture time series, and three-dimensional spatial position coordinate sequence of each livestock individual within the same time window are combined to form a complete time series sample containing multimodal information. The collection of all samples constitutes a multimodal time series dataset for subsequent model training and inference.

[0101] Reference Figure 2 As shown, temporal features of motion sensing and temporal features of visual posture are extracted for each individual livestock and poultry, and the two are adaptively weighted and fused through temporal attention fusion to obtain the multimodal behavioral representation vector of each individual livestock and poultry, including:

[0102] Extract motion sensing time-series segments and visual pose time-series segments of any livestock or poultry individual within a single time window from a multimodal time-series dataset;

[0103] The motion sensing time sequence segment is a multi-channel time sequence signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first time sequence coding network.

[0104] The first temporal coding network is constructed based on a multi-layer gated recurrent unit. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window.

[0105] The visual pose temporal segment is a sequence of two-dimensional pose key point coordinates. The coordinates of each key point at each time step are flattened and stitched together before being input into the second temporal coding network.

[0106] The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window.

[0107] Construct a temporal attention fusion, taking the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs, and calculate the attention weights of the two modalities at the current time step by step.

[0108] The temporal feature sequence of motion sensing and the temporal feature sequence of visual posture are multiplied by the attention weights at the corresponding time points and then added element by element to obtain the temporal step-by-step multimodal behavior representation sequence of the livestock individual.

[0109] Motion sensing data and visual data differ fundamentally in their temporal structure and information representation. To achieve effective fusion, each modality needs independent temporal encoding to extract high-level semantic features. Gated recurrent units (GRUs) effectively capture long-range dependencies in acceleration and angular velocity signals through their gating mechanism. Multilayer long short-term memory (LSTM) networks, through the synergistic effect of input gates, forget gates, and output gates, extract high-level semantic features containing spatial configuration evolution patterns from the posture keypoint sequence. The two temporal encoding networks are structurally adapted to the data characteristics of their respective modalities, and independent parameter spaces avoid gradient interference caused by differences in feature distribution between modalities. Furthermore, due to the different behavioral types of livestock and poultry affecting both motion sensing modalities and visual data... Different visual modalities have different levels of information dependence. For example, the motion signal of lying down behavior is weak but the visual posture changes significantly, while the motion signal of running behavior is intense and the visual contour changes significantly. Fixed-weight fusion strategies cannot adapt to the dynamic changes in the contribution of these modalities. Therefore, this scheme constructs a temporal attention fusion, which maps the temporal feature vectors of the two modalities to the same feature space and then performs element-wise addition and nonlinear activation. Then, the weight coefficients of the two modalities are calculated through a normalized exponential function. The modality that contributes more to the current behavior recognition is adaptively strengthened while the modality with redundant information is suppressed. Finally, the two mapped feature vectors are weighted and summed to obtain the fused multimodal behavior representation vector, realizing the dynamic adjustment and efficient fusion of modal contributions.

[0110] The expression for mapping the motion sensing temporal feature vector and the visual pose temporal feature vector to a common feature space of the same dimension through linear transformation is as follows:

[0111]

[0112] In the formula, This is the mapped motion sensing temporal feature vector. This is a motion sensing temporal feature vector. Mapped visual pose temporal feature vector This is a temporal feature vector of visual pose. , , , These are learnable parameters;

[0113] Scalar attention scores for each modality are computed using learnable context vectors:

[0114]

[0115] In the formula, For the attention score of the motor modality, For visual modality attention score, For the transpose of the learnable context vector, It is the hyperbolic tangent function;

[0116] The weighting coefficients are obtained by normalizing the exponential function:

[0117]

[0118] In the formula, For motion modality attention weights, For visual modality attention weights;

[0119] The two mapped feature vectors are multiplied by their respective weight coefficients and then summed element by element to obtain the multimodal behavioral representation vector of the livestock individual:

[0120]

[0121] In the formula, This is a multimodal behavior representation vector.

[0122] Reference Figure 3 As shown, a graph attention network is used to propagate and aggregate group-level interactive information, obtaining the group context-enhanced feature sequences of each livestock individual, including:

[0123] For each moment within the same time window, the set of all livestock and poultry individuals at that moment is taken as the graph node set, and the Euclidean distance between any two nodes is calculated based on the three-dimensional spatial coordinates at that moment.

[0124] A spatial distance threshold is preset. When the Euclidean distance between any two nodes is less than the threshold, an undirected edge is established between them.

[0125] Based on the spatial proximity and relative motion state between two nodes, the interaction strength weight between any established node pair is defined at each time step, and the weights of all node pairs constitute the adjacency matrix at the corresponding time step.

[0126] The multimodal behavior representation vectors of all nodes at this moment are used as the initial node features of the graph attention network;

[0127] The graph attention network employs a multi-head attention mechanism, performing message passing time-by-time. In each layer, for any central node at that time, a linear transformation is performed on the node features at that time. The set of its neighboring nodes is determined by the non-zero element index of the corresponding row in the adjacency matrix at that time. Attention coefficients are calculated for each neighboring node through the attention mechanism.

[0128] Using the attention coefficient as the weight, the transformed features of all neighboring nodes of the central node are weighted and summed, and then nonlinearly activated to obtain the updated features of the central node under the current head.

[0129] Concatenate the outputs of all headers to complete the message passing for this layer;

[0130] After repeating the message passing process for a preset number of layers, the updated features output by the attention network layer of each graph are taken and residually connected and spliced ​​to obtain the group context enhancement feature sequence of each livestock individual at each time step.

[0131] In group farming scenarios, individual behaviors are not isolated; an individual's behavior is influenced by the behavior of neighboring individuals. However, not all neighboring individuals have the same level of interaction. Effectively identifying key neighbors relevant to an individual's current behavioral decisions and extracting their information is the core issue in achieving group context enhancement. This solution constructs an inter-individual association graph through a two-step strategy: First, the Euclidean distance between any two nodes is calculated based on three-dimensional spatial coordinates. A basic connection relationship is established using a spatial distance threshold, including physically close individuals in the interaction candidate set, avoiding computational redundancy caused by message passing on a fully connected graph. Second, between established node pairs, the cosine similarity of multimodal behavior representation vectors is calculated to quantify the interaction strength. A higher value indicates that the behavioral characteristics of the two are closer and the interaction is stronger within the current time window. This value is used to weight edges, achieving refined modeling from spatial proximity to behavioral association. Based on this, the multimodal behavior representation vectors of each node are used as the initial node features of the graph attention network. An attention coefficient is calculated for each neighboring node using an attention mechanism, enabling the central node to adjust its behavior based on the neighboring nodes' interactions. The contribution of behavior recognition is dynamically allocated to information aggregation weights, which, together with temporal attention fusion, constitute the core of the temporal and spatial dual attention mechanism in this scheme. Through multi-layer message passing, the features of each node gradually integrate the behavioral information of multi-hop neighbors, while residual connections and splicing retain the group context information of different scales extracted from each layer. Finally, the group context enhancement features of each livestock individual contain not only its own multimodal temporal information, but also the interactive influence of other individuals in the group, effectively solving the problem of missing group context in individual behavior recognition in group farming scenarios. Among them, the setting of the spatial distance threshold is based on the spatial scale of the breeding pen and the typical social distance between livestock individuals. It is usually calibrated by statistically analyzing the distribution characteristics of the distance between individuals in a large number of samples and combining domain experience. The diagonal elements of the adjacency matrix are set to 1, that is, each node establishes a self-loop connection with itself to ensure that the node's own features are preserved in the aggregation process. The residual connection refers to adding the output of the current layer of the graph attention network to the input of the previous layer element by element. This connection method helps to alleviate the gradient vanishing problem in deep network training and retains the multi-scale information extracted from each layer.

[0132] For node pairs that have established spatial connections Its interaction strength weight Redefining:

[0133]

[0134] In the formula, For node pairs The Euclidean distance at the current moment, and For Gaussian kernel bandwidth parameters, and Each is a node pair The velocity vector at the current moment;

[0135] In each layer of the graph attention network, for the ... Layer The attention coefficient of a node to its neighbors is calculated as follows:

[0136]

[0137] In the formula, For the first Layer Head node To his neighbors raw attention score For activation function, For the first Layer Attention vector of size, for transpose, For the first Layer The linear transformation matrix of the head is a learnable parameter. and The first Layer node pairs eigenvectors, This involves concatenating vectors.

[0138] For the Layer Head node To his neighbors The raw attention scores are normalized to obtain the attention coefficients:

[0139]

[0140] In the formula, For the first Layer Head node For nodes The normalized attention coefficient;

[0141] The node update formula is:

[0142]

[0143] All K head outputs are concatenated to obtain , Using the exponential linear unit activation function, this attention mechanism enables each node to adaptively assign different weights to different neighbors when aggregating neighbor information, effectively identifying the key neighbors in the group that have the greatest impact on the current individual's behavioral decisions.

[0144] Reference Figure 4 As shown, the temporal decoding network decodes and outputs the behavior category label of each livestock individual at each time step, and the model is constrained by the group behavior consistency regularization term to obtain the final behavior recognition results, including:

[0145] The group context enhancement feature sequence of each livestock and poultry individual is input into the temporal decoding network. The temporal decoding network is constructed based on a bidirectional gated recurrent unit. It performs bidirectional encoding on the group context enhancement feature sequence along the time step direction. After splicing the forward and backward hidden states of each time step, it is mapped to the probability distribution space of the preset number of behavior categories through a linear classification layer, and outputs the probability value of each livestock and poultry individual belonging to each preset behavior category at each time step.

[0146] The behavior category with the highest probability value at each moment is used as the initial decoding label for that moment, thus obtaining the initial behavior category sequence for each individual livestock and poultry at each moment within the time window;

[0147] Based on the idea of ​​graph Laplace regularization, a group behavior consistency regularization term is constructed. Spatial proximity weight is used to measure the interaction strength of any two livestock individuals at each time step, and KL divergence is used to measure the difference in the predicted probability distribution of two nodes. The sum of the weighted probability distribution differences of all nodes with spatial proximity relationships at all times is calculated as the group consistency loss.

[0148] During the inference phase, a post-processing correction strategy based on conditional random fields is constructed, defining an energy function that includes unary and binary potentials. The unary potential is based on the negative logarithm of the probability output of the decoding network, while the binary potential is based on spatial proximity weights to penalize inconsistent labels of adjacent nodes.

[0149] The energy function is minimized by the belief propagation algorithm, and the corrected behavior category sequence is obtained as the final behavior recognition result.

[0150] Although the initial behavior category sequence output by decoding has already incorporated group context information at the feature level, this scheme introduces a group behavior consistency constraint at the decoding output for post-processing correction to further improve spatial recognition consistency. This constraint is based on the prior knowledge that livestock group behaviors are spatially correlated: when the initial identification result of a certain livestock individual is inconsistent with the behavior labels of most spatially neighboring individuals, the identification result is likely to be a misjudgment. This scheme first performs temporal decoding of the group context-enhanced feature sequence through a bidirectional gated recurrent unit, making full use of the context information before and after the current time step of the bidirectional network to improve decoding accuracy, and outputs the probability of each individual belonging to each preset behavior category at each time step. The model is initially labeled with the category of highest probability. During training, a soft constraint is applied to the model using a group consistency regularization term to encourage the predicted probability distributions of spatially neighboring individuals to converge. During inference, a Conditional Random Field (CRF) is used to globally optimize the output label sequence. Spatial smoothing correction is achieved through joint optimization of univariate and binary potentials. This post-processing strategy is consistent with the distribution-level consistency regularization objective in the training phase, leveraging the spatial correlation of the group to improve the rationality of the recognition results. The preset behavior categories include typical livestock and poultry behaviors such as standing, lying down, feeding, drinking, and walking. The spatial proximity weight is consistent with the edge weight in the graph construction phase, so that individuals that are closer in distance have a higher influence in the consistency constraint.

[0151] The group behavior consistency regularization term is defined as:

[0152]

[0153] in, The spatial proximity weight at time t, and Each is a node pair The three-dimensional spatial coordinates at time t, This represents the total number of time steps within the time window. Let be the set of edges in the graph. and Each is a node pair The predicted probability distribution at time t, This represents the Kullback–Leibler divergence, which measures the difference between two probability distributions. This regularization term encourages the predicted probability distributions of spatially neighboring individuals to converge during the training phase, providing prior constraints for individual identification at the group level.

[0154] During the inference phase, a conditional random field energy function is constructed for post-processing correction.

[0155]

[0156] In the formula, Let be the CRF energy function at time t. Let be the behavioral label vector of all individuals at time t. To decode the network output that individual i belongs to the label at time t The probability, As a unitary force, For indicator functions, when and The value is 1 when they are not equal, and 0 when they are equal. For a binary potential, the individual's behavior label vector refers to the vector formed by outputting the probability that a certain livestock individual i belongs to each preset behavior category at a certain time t.

[0157] The unary potential is the negative logarithm of the output probability of the decoding network, and the binary potential is the product of the spatial proximity weights and the label inconsistency indicator function. The energy function is minimized by the belief propagation algorithm to obtain the corrected label sequence. This post-processing strategy is smoother than hard voting replacement and can effectively avoid introducing excessive errors while utilizing the spatial correlation of the group. The belief propagation algorithm is a classic dynamic programming method for inference on a probabilistic graphical model. Its core idea is to repeatedly pass messages between adjacent nodes in the graph so that the state of each node can gradually integrate information from global observations. Specifically, in each iteration, each node calculates and sends a new message reflecting its confidence in the neighbor's state to each neighbor based on its own observation evidence (univariate potential) and messages received from all neighboring nodes. After multiple iterations of propagation, each node combines its own evidence with all incoming messages to approximately calculate the marginal probability or maximum posterior probability of each label under the current global energy function, thereby obtaining the most reasonable overall label configuration. In this scheme, the algorithm is used to solve for the minimum value of the conditional random field energy function, that is, by iteratively transmitting the label information of neighboring livestock and poultry in the individual graph, while taking into account the confidence of the model output (univariate potential) and the spatial consistency constraint (binary potential), and finally obtains a spatially smoothed group behavior label sequence.

[0158] Specifically, a comprehensive loss function is constructed, and end-to-end joint training is performed on all trainable network modules in the livestock and poultry behavior recognition model based on the labeled sample set. The trained recognition model is then deployed in an online recognition system, including:

[0159] Obtain a set of manually labeled livestock and poultry behavior recognition samples. Each training sample contains multimodal time-series data within a time window and the real behavior category label of each individual livestock and poultry at each moment within that window.

[0160] Construct a comprehensive loss function, which is a weighted sum of behavior classification loss and group consistency loss;

[0161] The backpropagation algorithm is adopted to minimize the comprehensive loss function. All trainable network modules of the livestock and poultry behavior recognition model are jointly trained end-to-end on the training sample set, and the parameters with the minimum comprehensive loss on the validation set are selected as the final model parameters.

[0162] The trained livestock and poultry behavior recognition model is integrated and packaged into an online recognition model and deployed in an online recognition system;

[0163] In actual operation, the online recognition model receives streaming data from multimodal data acquisition in real time, performs spatiotemporal alignment, feature extraction, fusion and decoding, and outputs the real-time behavior recognition results of each individual livestock and poultry.

[0164] This approach integrates temporal attention fusion, graph attention networks, and temporal decoding networks into a unified optimization framework for end-to-end joint training. Its core lies in constructing a comprehensive loss function that simultaneously reflects classification accuracy and group consistency. This drives the model to optimize both individual-level recognition accuracy and group-level spatial rationality during training. Specifically, the behavior classification loss uses a cross-entropy loss function to measure the difference between the predicted probability distribution and the true label for each livestock individual at each time step, ensuring the model possesses basic single-point classification capabilities. The group consistency loss quantifies and penalizes the degree of inconsistency in behavior labels among spatially neighboring nodes. A larger loss value is generated when the labels of spatially neighboring individuals diverge in the prediction results, thus driving the model to tend to output spatially consistent recognition results during parameter updates. This loss term also influences the graph attention network's feature aggregation process. The model has a clear optimization objective, pursuing not only accurate individual identification but also spatial consistency within the group, thereby making the propagation and aggregation of group interaction information more effective. During training, by monitoring the changes in the comprehensive loss value on the validation set and selecting the model parameters corresponding to the minimum value as the final parameters, the risk of overfitting is effectively avoided, ensuring the model's generalization ability. In the deployment phase, the trained modules are integrated and packaged into an online recognition model, capable of receiving multimodal data in real time and outputting behavior recognition results in a streaming manner. The calculation process between modules remains consistent with that in the training phase, ensuring a seamless transition from training to deployment. The weighting coefficients of the behavior classification loss and the group consistency loss are determined through cross-validation. The weight of the group consistency loss is typically set to 0.1 to 0.3 times the weight of the behavior classification loss to avoid excessive dominance of the classification objective by group constraints.

[0165] For each moment within each time window and for each individual livestock, the model outputs the probability distribution of the individual belonging to each preset behavior category through a temporal decoding network. At the same time, the true behavior category label of the individual at that moment is provided by a manually labeled sample set and represented in one-hot encoding form. The behavior classification loss is calculated based on the cross-entropy loss function, that is, for each individual at each moment, the probability value corresponding to the true category position in the model output probability distribution is taken, its negative logarithm is calculated, and then averaged over all individuals and all moments.

[0166] Furthermore, based on the same inventive concept as the aforementioned livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion, this solution proposes a livestock and poultry behavior recognition system based on deep learning multimodal temporal fusion, comprising:

[0167] The data acquisition module is used to simultaneously acquire the temporal sensing data of each individual livestock and poultry and the visual data of the group scene through multimodal data acquisition devices deployed in individual livestock and poultry and the breeding environment. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed.

[0168] The individual representation module is used to extract motion sensing temporal features and visual posture temporal features for each livestock individual based on a multimodal temporal dataset, and to adaptively weight and fuse the two through temporal attention fusion to obtain the multimodal behavior representation vector of each livestock individual.

[0169] The group aggregation module is used to construct an inter-individual association graph based on the spatial location information of each livestock and poultry individual in the breeding scenario, based on spatial distance and relative movement between individuals, and to use the multimodal behavior representation vector of each livestock and poultry individual as the initial feature of the node. Through the graph attention network, group-level interaction information is propagated and aggregated to obtain the group context-enhanced feature sequence of each livestock and poultry individual.

[0170] The decoding and correction module is used to enhance the feature sequence of each livestock individual based on the group context, decode and output the behavior category label of each livestock individual at each time step through the temporal decoding network, and use the group behavior consistency regularization term to constrain the model during the training phase. During the inference phase, the output result is corrected by combining the smoothing post-processing strategy to obtain the final behavior recognition result. The livestock behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence.

[0171] The training and deployment module is used to construct a comprehensive loss function, perform end-to-end joint training on all trainable network modules in the livestock and poultry behavior recognition model based on the labeled sample set, and deploy the trained recognition model on the online recognition system.

[0172] Specifically, the individual representation module includes:

[0173] The segment extraction unit is used to extract motion sensing time-series segments and visual pose time-series segments of any livestock individual within a single time window from a multimodal time-series dataset.

[0174] The motion sensing temporal feature unit is used when the motion sensing temporal segment is a multi-channel temporal signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first temporal coding network. The first temporal coding network is constructed based on a multi-layer gated recurrent unit and propagates forward along the time step direction in sequence, outputting the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window.

[0175] The visual posture temporal feature unit is used because the visual posture temporal segment is a sequence of two-dimensional posture key point coordinates. The coordinates of each key point at each time step are flattened and spliced, and then input into the second temporal coding network. The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window.

[0176] The attention fusion unit is used to construct temporal attention fusion. It takes the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs and calculates the attention weights of the two modalities at the current time step by step.

[0177] The multimodal behavior representation unit is used to multiply the time-step motion sensing temporal feature sequence and the visual posture temporal feature sequence by the attention weights at the corresponding time points, and then add them element by element to obtain the time-step multimodal behavior representation sequence of the livestock individual.

[0178] In summary, the advantages of this invention are: by constructing a hierarchical recognition architecture that integrates individual-level multimodal temporal perception and group-level graph interactive reasoning, and by combining temporal attention mechanisms and graph attention networks, the behavior recognition of each individual livestock and poultry under dynamic group context enhancement can be realized, effectively solving the problem of livestock and poultry behavior recognition and analysis in complex breeding scenarios.

[0179] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for livestock and poultry behavior recognition based on deep learning multimodal temporal fusion, characterized in that, include: By deploying multimodal data acquisition devices on individual livestock and poultry and in the breeding environment, the temporal sensing data of each individual livestock and poultry and the visual data of the group scene are acquired simultaneously. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed. Based on the multimodal temporal dataset, motion sensing temporal features and visual posture temporal features are extracted for each individual livestock and poultry. The two are then adaptively weighted and fused through temporal attention fusion to obtain the multimodal behavior representation vector of each individual livestock and poultry. Based on the spatial location information of each livestock and poultry individual in the breeding scenario, an inter-individual association graph is constructed based on spatial distance and relative movement between individuals. The multimodal behavior representation vector of each livestock and poultry individual is used as the initial feature of the node. Group-level interactive information is propagated and aggregated through a graph attention network to obtain the group context-enhanced feature sequence of each livestock and poultry individual. Based on the group context-enhanced feature sequence of each livestock and poultry individual, the behavior category label of each livestock and poultry individual at each time step is decoded and output through a temporal decoding network. The model is constrained during the training phase using a group behavior consistency regularization term, and the output result is corrected during the inference phase using a smoothing post-processing strategy to obtain the final behavior recognition result. The livestock and poultry behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence. A comprehensive loss function is constructed, and all trainable network modules in the livestock and poultry behavior recognition model are jointly trained end-to-end based on the labeled sample set. The trained recognition model is then deployed in an online recognition system.

2. The livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion according to claim 1, characterized in that, The process of synchronously acquiring temporal sensing data of each individual livestock and poultry and visual data of the group scene, performing spatiotemporal alignment and preprocessing, and constructing a multimodal temporal dataset includes: Each individual livestock and poultry is equipped with a wearable sensor that integrates an accelerometer, a gyroscope, and a UWB positioning module. The sensor continuously collects individual triaxial acceleration data and triaxial angular velocity data at a preset sampling frequency to form motion sensing time-series data. A visual acquisition device consisting of a depth camera and an RGB camera deployed above the breeding pens continuously acquires depth and color images of the group scene at the same sampling frequency as wearable sensors. Wearable sensors and visual acquisition devices are synchronized with a unified time server to ensure that motion sensing data and visual data correspond one-to-one in the time dimension at each sampling moment. The collected triaxial acceleration and triaxial angular velocity data are used to generate sensing time segments with a fixed time window length by using a sliding window truncation method. The acquired depth and color images are aligned with the coordinates based on preset camera intrinsic and extrinsic parameters, so that the depth and color images correspond one-to-one in spatial pixel position. For each sensing time segment within a time window, a Kalman filter algorithm is used for smoothing, and the filtered triaxial acceleration data is normalized to zero mean by window, mapping the acceleration values ​​to a distribution interval centered at zero. For the depth and color images within each time window, extract the two-dimensional bounding box and instance segmentation mask of each individual livestock and poultry in each frame image, and obtain the three-dimensional spatial coordinates of each individual livestock and poultry. The two-dimensional pose estimation model is used to extract the two-dimensional pose key point coordinate sequence of each livestock individual from color images; Based on the wearable sensor device number of each livestock and poultry individual and the individual identification obtained by visual detection, the correspondence between the sensor data and visual data in the same time window in each livestock and poultry individual dimension is established by the nearest neighbor matching algorithm to complete the data association; The motion sensing time series, visual posture time series, and three-dimensional spatial position coordinate sequence of each individual livestock and poultry are combined to form a multimodal time series sample, which constitutes a multimodal time series dataset.

3. The livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion according to claim 2, characterized in that, The process of extracting temporal features of motion sensing and temporal features of visual posture for each individual livestock and poultry, and then adaptively weighting and fusing the two through temporal attention fusion to obtain the multimodal behavioral representation vector of each individual livestock and poultry includes: Extract motion sensing time-series segments and visual pose time-series segments of any livestock or poultry individual within a single time window from a multimodal time-series dataset; The motion sensing time sequence segment is a multi-channel time sequence signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first time sequence coding network. The first temporal coding network is constructed based on a multi-layer gated recurrent unit. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window. The visual pose temporal segment is a sequence of two-dimensional pose key point coordinates. The coordinates of each key point at each time step are flattened and stitched together before being input into the second temporal coding network. The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward sequentially along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window. Construct a temporal attention fusion, taking the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs, and calculate the attention weights of the two modalities at the current time step by step. The temporal feature sequence of motion sensing and the temporal feature sequence of visual posture are multiplied by the attention weights at the corresponding time points and then added element by element to obtain the temporal step-by-step multimodal behavior representation sequence of the livestock individual.

4. The livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion according to claim 3, characterized in that, The process involves constructing an inter-individual relationship graph based on the spatial location information of each livestock and poultry individual in the breeding scenario, considering spatial distance and relative movement patterns between individuals. Using the multimodal behavioral representation vectors of each livestock and poultry individual as initial node features, a graph attention network is used to propagate and aggregate group-level interactive information, resulting in the acquisition of group context-enhanced feature sequences for each livestock and poultry individual. For each moment within the same time window, the set of all livestock and poultry individuals at that moment is taken as the graph node set, and the Euclidean distance between any two nodes is calculated based on the three-dimensional spatial coordinates at that moment. A spatial distance threshold is preset. When the Euclidean distance between any two nodes is less than the threshold, an undirected edge is established between them. Based on the spatial proximity and relative motion state between two nodes, the interaction strength weight between any established node pair is defined at each time step, and the weights of all node pairs constitute the adjacency matrix at the corresponding time step. The multimodal behavior representation vectors of all nodes at this moment are used as the initial node features of the graph attention network; The graph attention network employs a multi-head attention mechanism, performing message passing time-by-time. In each layer, for any central node at that time, a linear transformation is performed on the node features at that time. The set of its neighboring nodes is determined by the non-zero element index of the corresponding row in the adjacency matrix at that time. Attention coefficients are calculated for each neighboring node through the attention mechanism. Using the attention coefficient as the weight, the transformed features of all neighboring nodes of the central node are weighted and summed, and then nonlinearly activated to obtain the updated features of the central node under the current head. Concatenate the outputs of all headers to complete the message passing for this layer; After repeating the message passing process for a preset number of layers, the updated features output by the attention network layer of each graph are taken and residually connected and spliced ​​to obtain the group context enhancement feature sequence of each livestock individual at each time step.

5. The livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion according to claim 4, characterized in that, The process involves enhancing the group context feature sequences of each livestock individual, decoding and outputting the behavior category label of each individual at each time step using a temporal decoding network, and constraining the model during the training phase using a group behavior consistency regularization term. During the inference phase, a smoothing post-processing strategy is used to correct the output results, resulting in the final behavior recognition results, including: The group context enhancement feature sequence of each livestock and poultry individual is input into the temporal decoding network. The temporal decoding network is constructed based on a bidirectional gated recurrent unit. It performs bidirectional encoding on the group context enhancement feature sequence along the time step direction. After splicing the forward and backward hidden states of each time step, it is mapped to the probability distribution space of the preset number of behavior categories through a linear classification layer, and outputs the probability value of each livestock and poultry individual belonging to each preset behavior category at each time step. The behavior category with the highest probability value at each moment is used as the initial decoding label for that moment, thus obtaining the initial behavior category sequence for each livestock individual at each moment within the time window; Based on the idea of ​​graph Laplace regularization, a group behavior consistency regularization term is constructed. Spatial proximity weight is used to measure the interaction strength of any two livestock individuals at each time step, and KL divergence is used to measure the difference in the predicted probability distribution of two nodes. The sum of the weighted probability distribution differences of all nodes with spatial proximity relationships at all times is calculated as the group consistency loss. During the inference phase, a post-processing correction strategy based on conditional random fields is constructed, defining an energy function that includes unary and binary potentials. The unary potential is based on the negative logarithm of the probability output of the decoding network, while the binary potential is based on spatial proximity weights to penalize inconsistent labels of adjacent nodes. The energy function is minimized by the belief propagation algorithm, and the corrected behavior category sequence is obtained as the final behavior recognition result.

6. The livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion according to claim 5, characterized in that, The construction of the comprehensive loss function, the end-to-end joint training of all trainable network modules in the livestock and poultry behavior recognition model based on the labeled sample set, and the deployment of the trained recognition model in the online recognition system include: Obtain a set of manually labeled livestock and poultry behavior recognition samples. Each training sample contains multimodal time-series data within a time window and the real behavior category label of each individual livestock and poultry at each moment within that window. Construct a comprehensive loss function, which is a weighted sum of behavior classification loss and group consistency loss; The backpropagation algorithm is adopted to minimize the comprehensive loss function. All trainable network modules of the livestock and poultry behavior recognition model are jointly trained end-to-end on the training sample set, and the parameters with the minimum comprehensive loss on the validation set are selected as the final model parameters. The trained livestock and poultry behavior recognition model is integrated and packaged into an online recognition model and deployed in an online recognition system; In actual operation, the online recognition model receives streaming data from multimodal data acquisition in real time, performs spatiotemporal alignment, feature extraction, fusion and decoding, and outputs the real-time behavior recognition results of each individual livestock and poultry.

7. A livestock and poultry behavior recognition system based on deep learning multimodal temporal fusion, characterized in that, The method for implementing the livestock and poultry behavior recognition method based on deep learning multimodal temporal fusion as described in any one of claims 1-6 includes: The data acquisition module is used to simultaneously acquire the temporal sensing data of each individual livestock and poultry and the visual data of the group scene through multimodal data acquisition devices deployed in individual livestock and poultry and the breeding environment. After spatiotemporal alignment and preprocessing, a multimodal temporal dataset is constructed. The individual representation module is used to extract motion sensing temporal features and visual posture temporal features for each livestock individual based on a multimodal temporal dataset, and to adaptively weight and fuse the two through temporal attention fusion to obtain the multimodal behavior representation vector of each livestock individual. The group aggregation module is used to construct an inter-individual association graph based on the spatial location information of each livestock and poultry individual in the breeding scenario, based on spatial distance and relative movement between individuals, and to use the multimodal behavior representation vector of each livestock and poultry individual as the initial feature of the node. Through the graph attention network, group-level interaction information is propagated and aggregated to obtain the group context-enhanced feature sequence of each livestock and poultry individual. The decoding and correction module is used to enhance the feature sequence of each livestock individual based on the group context, decode and output the behavior category label of each livestock individual at each time step through the temporal decoding network, and use the group behavior consistency regularization term to constrain the model during the training phase. During the inference phase, the output result is corrected by combining the smoothing post-processing strategy to obtain the final behavior recognition result. The livestock behavior recognition model is composed of multimodal temporal coding, temporal attention fusion, graph attention network and temporal decoding network connected in sequence. The training and deployment module is used to construct a comprehensive loss function, perform end-to-end joint training on all trainable network modules in the livestock and poultry behavior recognition model based on the labeled sample set, and deploy the trained recognition model on the online recognition system.

8. The livestock and poultry behavior recognition system based on deep learning multimodal temporal fusion according to claim 7, characterized in that, The individual representation module includes: The segment extraction unit is used to extract motion sensing time-series segments and visual pose time-series segments of any livestock individual within a single time window from a multimodal time-series dataset. The motion sensing temporal feature unit is used when the motion sensing temporal segment is a multi-channel temporal signal. The acceleration channel data and angular velocity channel data of each time step are concatenated along the channel dimension and then input into the first temporal coding network. The first temporal coding network is constructed based on a multi-layer gated recurrent unit and propagates forward along the time step direction in sequence, outputting the hidden state sequence of all time steps, which serves as the motion sensing temporal feature sequence of the livestock individual within the time window. The visual posture temporal feature unit is used because the visual posture temporal segment is a sequence of two-dimensional posture key point coordinates. The coordinates of each key point at each time step are flattened and spliced, and then input into the second temporal coding network. The second temporal coding network is constructed based on a multi-layer long short-term memory network. It propagates forward along the time step direction and outputs the hidden state sequence of all time steps, which serves as the visual posture temporal feature sequence of the livestock individual within the time window. The attention fusion unit is used to construct temporal attention fusion. It takes the temporal feature sequence of motion sensing and the temporal feature sequence of visual pose as inputs and calculates the attention weights of the two modalities at the current time step by step. The multimodal behavior representation unit is used to multiply the time-step motion sensing temporal feature sequence and the visual posture temporal feature sequence by the attention weights at the corresponding time points, and then add them element by element to obtain the time-step multimodal behavior representation sequence of the livestock individual.

Images