Pig abnormal behavior recognition and early warning system based on internet of things and deep learning

Abstract

The application discloses a pig abnormal behavior recognition and early warning system based on Internet of Things and deep learning, comprising: a data acquisition layer deployed in the interior and surrounding area of a pig breeding house; an edge computing layer deployed in an industrial control room on site of the breeding house, in communication connection with the data acquisition layer, for lightweight real-time processing and initial screening of abnormal events on the collected multi-modal data; a cloud data processing layer connected with the edge computing layer through the Internet, for multi-modal deep fusion analysis and fine identification of the abnormal event data; and an early warning and application layer in communication connection with the cloud data processing layer, for generating and pushing multi-level early warning information and auxiliary decision suggestions. The application adopts multi-source collaborative perception, improves monitoring comprehensiveness and pertinence, realizes global coverage through fixed node grid layout, and rechecks suspected abnormal areas on demand by track-type mobile robots, and effectively overcomes the limitation of single sensor in combination with three-mode data acquisition of vision, audio and environment.

Description

Technical Field

[0001] This application belongs to the field of intelligent monitoring technology for livestock farming, specifically involving a system for identifying and warning of abnormal pig behavior based on the Internet of Things and deep learning. Background Technology

[0002] In modern pig farming, abnormal pig behaviors (such as lameness, prolonged immobility, frequent coughing, and abnormal feeding) are key indicators for early disease warning. Traditional monitoring methods rely heavily on manual inspections, which have inherent drawbacks such as low efficiency (limited number of pens that can be inspected per person per day), strong subjectivity (experience differences leading to missed diagnoses), and inability to provide 24-hour coverage. Furthermore, human contact can increase the risk of disease transmission.

[0003] In recent years, some farms have attempted to introduce single video monitoring systems, but they face multiple technical bottlenecks: First, the pigsty environment is complex (drastic changes in lighting, dust and ammonia interference, pigs occluding each other), resulting in low accuracy of single visual modality recognition and a false alarm rate of over 30%; Second, existing systems mostly adopt an "edge-to-cloud direct transmission" architecture, with massive amounts of raw video data uploaded, leading to high network bandwidth pressure and high cloud processing latency, making it difficult to meet the minute-level early warning requirements; Third, edge devices have limited computing power, and general deep learning models (such as the standard YOLOv4) have a large number of parameters, making it difficult to deploy lightweight real-time analysis; Fourth, the models lack continuous optimization mechanisms, resulting in poor adaptability to seasonal changes in pig behavior patterns or new diseases; Fifth, the system's decision-making process is "black box," disconnected from the experience of livestock experts, leading to insufficient credibility of early warning results and difficulty in generating actionable management recommendations.

[0004] Therefore, there is an urgent need for an intelligent early warning system that integrates multimodal perception, edge-cloud collaborative computing, self-evolution capabilities, and deep integration with professional knowledge to achieve accurate, efficient, and reliable monitoring of pig health status. Summary of the Invention

[0005] This application provides a system for identifying and warning of abnormal behavior in pigs based on the Internet of Things and deep learning. It aims to solve the problems of low accuracy of single visual modality recognition in existing technologies, difficulty in meeting the timeliness requirements of minute-level early warning, and difficulty in deploying lightweight real-time analysis.

[0006] A system for identifying and warning of abnormal pig behavior based on the Internet of Things and deep learning includes: The data acquisition layer is deployed inside and around the pig farm to achieve real-time, multi-source, all-weather data acquisition of the farming environment, individual pig behavior, and physiological acoustic characteristics. The edge computing layer is deployed in the industrial control room at the breeding farm site and communicates with the data acquisition layer to perform lightweight real-time processing and initial screening of abnormal events on the acquired multimodal data. The cloud data processing layer connects to the edge computing layer via the Internet to perform multimodal deep fusion analysis and fine identification of abnormal event data; The early warning and application layer communicates with the cloud data processing layer to generate and push multi-level early warning information and auxiliary decision-making suggestions. The data acquisition layer includes a fixed data acquisition node array and a track-mounted mobile acquisition robot, which synchronize data and interact with commands through an industrial-grade wireless self-organizing network protocol.

[0007] Optionally, the fixed data acquisition nodes are arranged in a grid pattern according to the layout of the pigsty's internal pens. Each node integrates a visual acquisition module, an environmental perception module, and an audio acquisition module, wherein: The visual acquisition module uses a wide dynamic range CMOS image sensor and has a built-in near-infrared fill light, which automatically switches to infrared mode at night. The environmental sensing module includes a temperature and humidity sensor, an ammonia concentration sensor, a carbon dioxide sensor, and a light intensity sensor. The audio acquisition module uses a MEMS microphone array and is equipped with a front-end digital signal processor for preliminary noise reduction.

[0008] Optionally, the track-mounted mobile data collection robot includes: The positioning and navigation module uses a low-frequency RFID reader in conjunction with electronic tags embedded on both sides of the track to achieve absolute position identification; The gimbal and robotic arm mechanism includes a high-definition visible light camera and an infrared thermal imager mounted on the gimbal, and a proximity sensor at the end of the robotic arm. The task triggering mechanism automatically plans a path to move to the target pen when a fixed node detects suspected abnormal behavior or receives a cloud scheduling instruction, and then tracks and photographs the suspected pigs from multiple perspectives.

[0009] Optionally, the edge computing layer is deployed with a lightweight deep learning model, including: An improved YOLOv4-Tiny target detector, in which the backbone network uses depthwise separable convolutions instead of standard convolutions, adds a third detection branch to handle small targets, and embeds a global context attention module; The improved DeepSORT multi-target tracker upgrades the motion model to a uniform acceleration model, replaces the re-identification network with a lightweight network built on the OSA module, and optimizes the cascade matching strategy.

[0010] Optionally, the global context attention module in the improved YOLOv4-Tiny object detector replaces the fully connected layer with a one-dimensional convolution, and the convolution kernel size y is adaptively determined by the number of channels C: ; in, This indicates taking the nearest odd number upwards, with the parameter... The slope of the logarithmic mapping between the number of control channels and the kernel size, where b is the bias term.

[0011] Optionally, the edge computing layer is also configured with a behavior screening module, which calculates gait cycle, dwell time and activity area based on trajectory time series data, and combines the key point information of the pig's face to set threshold rules to make preliminary judgments on prolonged immobility, lameness, coughing and sneezing, and abnormal eating, and only uploads the preliminary screening abnormal event data to the cloud data processing layer.

[0012] Optionally, the cloud data processing layer includes a multimodal deep fusion network, which includes: A single-modal feature encoder processes visual, audio, and environmental temporal data separately. The cross-modal feature fusion machine employs a multi-head cross-attention mechanism, using visual features as queries and audio and environmental features as keys and values. A multi-task decoder, including a fine-grained behavior classifier, an anomaly severity regressor, and a temporal locator; The joint decision-making module weights and fuses the outputs of the multi-task decoder to generate the final diagnostic result.

[0013] Optionally, the cloud data processing layer also includes a model self-evolution module, which automatically filters samples with confidence levels below a threshold or whose labels are corrected after manual review, adds them to a difficult example pool, periodically expands the training set by expert annotation, and retrains the multimodal deep fusion model based on the updated training set to achieve continuous improvement in recognition accuracy.

[0014] Optionally, the early warning and application layer includes: The large visual monitoring screen displays a panoramic map of the farm, real-time video previews, a list of abnormal events, and environmental parameter curves. The mobile app provides alert push notifications, event details, task orders, and data statistics. The multi-level early warning mechanism is divided into three levels: attention level, warning level, and severe level, based on the score of the degree of abnormality and the results of the risk assessment of transmission. Each level corresponds to a different processing time limit and responsible person.

[0015] Optionally, the early warning and application layer also includes an expert rule base, which transforms the experience and knowledge of livestock experts into executable rules, and makes collaborative decisions with the output of deep learning models. When specific conditions are met, the early warning level is automatically adjusted and auxiliary decision-making suggestions are generated, including environmental control suggestions, isolation suggestions, treatment plan recommendations, and vaccine booster reminders.

[0016] Compared with the prior art, this application has at least the following beneficial effects: This application adopts multi-source collaborative sensing to improve the comprehensiveness and relevance of monitoring. Fixed node grid deployment achieves full coverage, and track-type mobile robots review suspected abnormal areas as needed. Combining visual, audio, and environmental three-modal data acquisition, it effectively overcomes the limitations of single sensors.

[0017] This application adopts an edge-cloud collaborative architecture to optimize resource utilization and response time. The lightweight edge layer model (improved YOLOv4-Tiny+DeepSORT) completes more than 90% of the initial screening of routine data, and only uploads 5% of suspected abnormal fragments to the cloud. This reduces network bandwidth usage by 85%, shortens the early warning response time to within 3 minutes, and improves the real-time performance of the system.

[0018] This application enhances the robustness and accuracy of recognition through multimodal deep fusion. The cloud-based cross-modal attention fusion mechanism effectively associates visual behavior, cough acoustic features and environmental parameters (such as a sudden increase in ammonia). The recognition F1 value for complex abnormalities such as limping and respiratory diseases reaches 0.92, which is better than the single-modal solution. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the module connections of a pig abnormal behavior recognition and early warning system based on the Internet of Things and deep learning, provided as an embodiment of this application. Detailed Implementation

[0020] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1 The present application will be further described in detail with reference to the embodiments.

[0021] The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning provided in this application includes: a data acquisition layer, an edge computing layer, a cloud data processing layer, and an early warning and application layer; The data acquisition layer is deployed inside and around the pig farm to achieve real-time, multi-source, all-weather data acquisition of the farming environment, individual pig behavior, and physiological acoustic characteristics. This layer consists of a fixed data acquisition node array and a track-mounted mobile acquisition robot, which synchronize data and exchange commands via an industrial-grade wireless self-organizing network protocol. Specifically, the fixed data acquisition nodes are deployed in a grid pattern according to the layout of the pigsty, with each monitoring unit covering a standardized 8m × 10m pigsty. Each node integrates the following functional modules: The visual acquisition module employs a wide dynamic range CMOS image sensor with a resolution of at least 1920×1080 and a frame rate of 30fps. This module has a built-in near-infrared supplemental light (center wavelength 850nm), which automatically switches to infrared mode in nighttime or low-light conditions to ensure continuous 24-hour imaging. The lens has a 90° field of view and supports electronic pan-tilt control, used to acquire images of pigs' standing, lying, eating, and drinking behaviors, providing input for subsequent posture estimation. The environmental sensing module includes a temperature and humidity sensor (model SHT30, accuracy ±0.3℃, ±2%RH), an ammonia concentration sensor (model MQ-137, range 1~100ppm), a carbon dioxide sensor (model S8-0053, range 400~5000ppm), and a light intensity sensor. These sensors collect data at 5-minute intervals, which is then aggregated to the node controller via an RS485 bus, timestamped, and uploaded in a package. The audio acquisition module employs a MEMS microphone array with a sampling rate of 44.1kHz and 16-bit quantization. This module continuously acquires environmental audio from the pigs' activity area, including pig sounds such as grunts, coughs, sneezes, and chewing noises. The acquired data undergoes preliminary noise reduction (Butterworth low-pass filter, cutoff frequency 8kHz) by a front-end digital signal processor to remove background noise caused by equipment such as fans and manure removal devices. The edge preprocessing unit, with a low-power embedded processor built into the fixed node, is responsible for the initial encoding and compression of the acquired video, audio, and environmental data. The video stream is compressed in H.265 format and temporarily stored. Key segments are only pushed to the edge computing layer when a moving target is detected or a call instruction is received from the cloud / edge server. To address the issues of obstructed views and lost details at fixed nodes, the track-mounted mobile data collection robot utilizes an I-shaped aluminum alloy track deployed above the livestock sheds. The track is 2.5-3.0 meters above the ground and runs through each livestock pen. The track-mounted mobile robot, driven by wheels, is suspended below the track and possesses autonomous inspection and precise positioning capabilities along the track. Specifically, it includes: The positioning and navigation module uses a low-frequency RFID reader (operating frequency 125kHz) mounted on the front of the robot chassis, which works in conjunction with RFID electronic tags embedded on both sides of the track to achieve absolute position identification. Limit switches are also installed at both ends of the track for hard limiting of the parking position. The robot employs a fuzzy PID control algorithm to differentially control the drive wheels, with a low-speed inspection speed of 0.1m / s, medium-speed 0.25m / s, and high-speed 0.5m / s, and a parking positioning error controlled within ±12mm. The robot features a gimbal and robotic arm mechanism. A two-dimensional gimbal (360° horizontal, ±90° pitch) is mounted on the bottom of the robot. A high-definition visible light camera (2560×1440 resolution, supports 20× optical zoom) and an infrared thermal imager (640×512 resolution, temperature measurement accuracy ±2℃) are mounted on the gimbal. Additionally, a secondary robotic arm is located on the side of the robot, its end effector extending into the pen to collect close-up data on the pigs' body surface characteristics (such as skin erythema and injuries) and to attach smart ear tags (for reading physiological parameters such as body temperature and heart rate). The system includes a task triggering mechanism. Normally, the robot is docked at a charging station and ready to go. When a fixed node detects suspected abnormal behavior (such as pigs lying still for extended periods or frequent coughing) or receives a cloud-based dispatch command, the system automatically plans the optimal path and drives the robot to move directly above the target pen. Upon arrival, the gimbal automatically adjusts its angle to track and photograph the suspected pig from multiple perspectives. Simultaneously, the near-field sensors at the end of the robotic arm (including a miniature camera and an infrared temperature probe) collect detailed data to supplement information missing from the fixed node due to distance or angle variations. A local area network is established between fixed nodes and mobile robots via industrial-grade WiFi modules, covering the entire building. All collected data is encapsulated in the following format: Video stream data: in units of GOP (Group of Pictures), with timestamp, node ID, and camera pose parameters; Environmental parameter data: encapsulated in JSON format, including sensor type, value, and acquisition time; Audio data: saved in WAV format, with noise reduction flags and audio energy characteristic values; Data is first aggregated to a regional data aggregation gateway deployed in the control room of the livestock shed, and then connected to the Internet via fiber optic cable, and uploaded to the cloud data processing layer on a regular basis. When the network is interrupted, the data is temporarily stored locally, and automatically resumed when the network is restored, ensuring data integrity.

[0022] The edge computing layer is deployed in the industrial control room at the breeding farm site, serving as a bridge connecting the front-end data acquisition layer and the cloud data processing layer. It is responsible for real-time data processing, lightweight inference, and initial screening of abnormal events. This layer adopts a heterogeneous computing architecture, consisting of edge computing servers, GPU accelerator cards, and a real-time database. It is connected to the regional aggregation gateway of the data acquisition layer via gigabit Ethernet, enabling low-latency data processing and response. Specifically, the edge computing server uses an industrial-grade embedded industrial control computer to ensure at least 72 hours of data processing capability during network outages. The operating system is Ubuntu 20.04 LTS, and a Docker containerized environment is deployed to isolate the operation of different algorithm modules. The edge computing layer subscribes to multimodal data topics published by the regional aggregation gateway via the MQTT protocol, including: Compressed video stream (H.265 format): After being received by GOP, it is decoded in real time into YUV420P raw frames using the FFmpeg library and scaled to the model input size (such as 416×416 or 224×224) while maintaining the original aspect ratio for padding. Environmental time-series data (JSON format): After parsing, it is stored in the in-memory database Redis to form a sliding window (default window length 30 minutes) for subsequent anomaly correlation analysis. Audio feature data: If the front end has already extracted MFCC features, it will be received directly; otherwise, the original audio segment will be received for local feature extraction. The lightweight deep learning model deployed in the edge computing layer comprises two core components: an improved YOLOv4-Tiny object detector and an improved DeepSORT multi-object tracker. These two work together in a serial pipeline manner. The target bounding boxes and category information output by the detector serve as input to the tracker, which assigns a unique identifier to each target and establishes a temporal trajectory, providing a foundation for subsequent behavior analysis, as detailed below: To achieve real-time and accurate detection of pig targets in aquaculture scenarios, while meeting the computing power constraints of edge devices, a systematic lightweight transformation and accuracy optimization of the YOLOv4-Tiny basic network was carried out. The improvement strategy covers four dimensions: convolution operator replacement, feature pyramid reconstruction, attention mechanism embedding, and activation function optimization.

[0023] In the backbone network CSPDarknet53-Tiny, all standard 3×3 convolutions are replaced with depthwise separable convolutions. The depthwise separable convolution consists of two independent stages: the first stage is depthwise convolution, where each kernel is responsible for only one channel of the input feature map, completing spatial feature extraction, with a computational cost of W×H×M×K×K, where W and H are the width and height of the feature map, M is the number of input channels, and K is the kernel size; the second stage is pointwise convolution, using 1×1 convolutions to linearly combine the channel dimensions of the feature map output from the depthwise convolution, with a computational cost of M×N×K×K, where N is the number of output channels. The computational cost of standard convolution is W×H×M×N×K×K, and the ratio of the computational cost of the two is... + Taking an input feature map of size 416×416 and 64 channels as an example, the computational cost is reduced to approximately 18.7% of that of standard convolution after using depthwise separable convolution. Experimental results show that this replacement operation reduces the total number of detector parameters by 52% and improves GPU inference speed by 41%, freeing up computational resources for subsequent multi-channel video parallel processing. To address the issue of small targets like piglets occupying a small portion of the image and their features being easily lost, a third detection branch is added to the existing two detection branches (downsampling by 16 times to output a 26×26 feature layer and downsampling by 32 times to output a 13×13 feature layer). This branch takes the 52×52 feature layer output from the CSPBlock module in the backbone network. This layer has higher spatial resolution and contains richer detail information, as detailed below: The feature fusion path adopts a bidirectional feature pyramid structure that combines top-down and bottom-up approaches: Top-down approach: The 26×26 feature layer is compressed to 128 channels through 1×1 convolution, and then upsampled to restore the resolution to 52×52. It is then concatenated with the 52×52 feature layer of the backbone network in terms of channel dimension to obtain a fused feature containing high-level semantic information and shallow detail information. Bottom-up path: The above-mentioned fused features are downsampled through a 3×3 convolution with a stride of 2, and then fused again with a 26×26 feature layer to further enhance the feature representation capability; The fused 52×52 detection branch adopts a preset anchor box mechanism. The anchor box size is obtained by K-means clustering of small target bounding boxes in the training set. The cluster centers are set to three groups of (8×8), (16×16), and (24×24) to meet the detection needs of small targets such as piglet heads and young piglets. To enhance the model's robustness to complex scenes such as occlusion, pose changes, and uneven lighting, an improved Global Context Attention (GCA) module is inserted at the connection point between each detection branch and the backbone network. This module's design draws inspiration from coordinate attention, achieving lightweight global context modeling through decompositional modeling. Let the input feature map be The calculation process of the GCA module is as follows: Global context modeling: First, a 1×1 convolution is used. A linear transformation is performed on the input feature map to obtain the attention weight matrix. After normalization using the Softmax function, the feature map is multiplied with the original feature map to obtain the global context features. :

[0024] in =H*W represents the total number of spatial locations; Cross-channel interaction and feature recalibration: Traditional GCNet uses two 1×1 convolutions for fully connected transformations along the channel dimension, resulting in a large number of parameters. This solution uses one-dimensional convolutions instead of fully connected layers, compressing the number of parameters while capturing local cross-channel interaction information. Specifically, global context features are... One-dimensional convolution is performed along the channel dimension, and the kernel size y is adaptively determined by the number of channels C:

[0025] Where y=2 and b=1 are hyperparameters. This indicates taking the nearest odd number upwards, with the parameter... The slope of the logarithmic mapping between the number of channels and the kernel size is controlled by b, which is the bias term. After one-dimensional convolution, channel attention weights are generated using the Sigmoid function and multiplied element-wise with the original input feature map X to complete feature recalibration. Residual connection: The recalibrated feature map is added to the input feature map using residuals to ensure a smooth gradient backpropagation path.

[0026] in Use the Sigmoid activation function; Ablation experiments show that after inserting the GCA module, the mean accuracy (mAP) of the model on the validation set is improved by 3.2 percentage points, while the number of parameters increases by only 461, and the increase in computation is negligible. LeakyReLU is used instead of standard ReLU as the activation function to address the neuron death problem caused by ReLU's zero gradient on the negative half-axis. The mathematical expression for LeakyReLU is:

[0027] The slope parameter of the negative half-axis Setting it to 0.1 allows neurons to maintain small gradients even with negative inputs, promoting model convergence and accelerating the training process. The improved YOLOv4-Tiny detector outputs predicted feature maps at three scales, with dimensions of 52×52×(B×(5+C)), 26×26×(B×(5+C)), and 13×13×(B×(5+C)), respectively. Here, B represents the number of bounding boxes predicted for each grid (set to 3), and C represents the number of target categories (set to 3 categories in this system: pigs, feed troughs, and waterers). Each bounding box prediction includes center coordinates (…). , ), width and height ( , ), confidence level and category probability distribution; In the post-processing stage, the Soft-NMS algorithm is used instead of traditional non-maximum suppression to solve the problem of false suppression of adjacent targets in dense scenes. Soft-NMS decays the confidence of adjacent detection boxes based on the size of the overlapping region instead of directly setting them to zero; the decay function uses a Gaussian form.

[0028] Where M is the highest confidence detection box. The bounding box to be processed is shown, and IOU is the intersection-union ratio of the two bounding boxes. Set to 0.5. After Soft-NMS processing, the detection boxes with a confidence level higher than the threshold (0.5) are used as the final output, with targets of the category "pigs" being sent to the downstream tracking module; The improved DeepSORT multi-object tracker incorporates three core improvements based on the DeepSORT algorithm: motion model upgrade, lightweighting of the re-identification network, and optimization of the cascaded matching strategy, as detailed below: The original DeepSORT algorithm uses a constant velocity model (CV) to describe the target's motion, with an 8-dimensional state vector: , where (u,v) are the center coordinates of the detection box, γ is the aspect ratio, h is the box height, and the dotted variable is the corresponding first derivative (velocity). This model assumes that the target maintains uniform linear motion between adjacent frames, making it difficult to fit the frequent non-uniform behaviors such as pauses, turns, and sudden accelerations that occur during the pig's walking process, leading to an increase in identity switching (IDSwitch); Therefore, this scheme upgrades the motion model to a constant acceleration model (CA), and expands the state vector to 12 dimensions:

[0029] The variables with two dots represent the corresponding acceleration components. The state transition matrix F is correspondingly expanded into a 12×12 matrix:

[0030] in This is the inter-frame time interval (fixed at 40ms for a 25fps video stream). The observation matrix HH remains in 4×12 form, observing only the position component:

[0031] The process noise covariance matrix Q and the observation noise covariance matrix R were obtained through maximum likelihood estimation of manually labeled trajectories in the training set. Experimental results show that, after adopting the CA model, the trajectory prediction error for the target during turning and stopping is reduced by 37%, and the number of identity switching events is reduced by 28%. To reduce the computational overhead of the appearance feature extraction module, the re-identification network based on Wide ResNet in the original DeepSORT was replaced with a lightweight network built on the OSA module (One-Shot Aggregation), named VOVNetV1. The OSA module's design philosophy stems from an improvement on the inefficiency of DenseNet's dense connections. In DenseNet, each layer is densely connected to all preceding layers, resulting in a linear increase in the number of input channels with the number of layers, high memory access cost (MAC), and low GPU parallelism utilization. The OSA module aggregates features from all preceding layers only in the last layer of the module, with no direct connections between intermediate layers. This significantly reduces memory access overhead while retaining the advantages of multi-receptive-field feature fusion. The specific structure of VOVNetV1 is as follows: Input layer: Accepts a detection bounding box image of size 120×60 (to maintain the aspect ratio, the original detection bounding box is scaled proportionally and then filled to 120×60). First layer: 3×3 standard convolution, stride 1, output channels 32, output size 120×60; OSA Module 1: Stacks three 3×3 depth separable convolutional layers, each with 64 output channels, using downsampling with a stride of 2, and output size 60×30; OSA Module 2: Stacked three 3×3 depth separable convolutional layers, each with 128 output channels, stride 2 downsampling, and output size 30×15; OSA Module 3: Stack three 3×3 depth separable convolutional layers, each with 128 output channels, stride 1 to maintain size, and output size 30×15; Global average pooling layer: compresses the feature map to 1×1 and outputs a 128-dimensional feature vector; L2 normalization layer: Performs L2 normalization on the 128-dimensional feature vector to ensure , used for subsequent cosine distance calculation; Within each OSA module, the outputs of intermediate layers are not directly passed to subsequent layers. Instead, at the end of the module, all intermediate layer outputs are concatenated along the channel dimension, and then subjected to 1×1 convolution for feature fusion and dimensionality reduction. This design results in shorter gradient propagation paths and faster training convergence. VOVNetV1 was pre-trained on the Market-1501 pedestrian re-identification dataset and then fine-tuned on a self-built pig re-identification dataset. The pig dataset contains 50 individuals, with at least 50 images per pig, covering different ages, shooting angles, and lighting conditions. After fine-tuning, the model achieved a Rank-1 accuracy of 94.2% on the test set, with only 1 / 3 the number of parameters of the original Wide ResNet, and a 40% improvement in inference speed. The tracker uses the Hungarian algorithm to find the optimal match between the bounding box and the trajectory. The cost matrix is ​​composed of a weighted fusion of motion features and appearance features, as follows: Motion feature metric: The normalized distance between the j-th detection box and the Kalman prediction box of the i-th trajectory is calculated using Mahalanobis distance.

[0032] in The detection box position vector , The Kalman prediction location for the trajectory, The covariance matrix is ​​provided by the Kalman filter. Mahalanobis distance considers the correlation between dimensions and prediction uncertainty, making it more suitable for measuring motion similarity than Euclidean distance. Less than the threshold If the value is 9.4877 (determined by the 0.95 quantile of the chi-square distribution), then the motion feature is considered to be successfully matched.

[0033] Appearance feature measurement: The similarity between the feature vector of the detection box and the set of historical feature vectors of the trajectory is calculated using the least cosine distance.

[0034] in The detection bounding box is a 128-dimensional normalized feature vector extracted by VOVNetV1. Let i be the set of feature vectors from the most recent 100 frames that have been successfully associated with trajectory i. Less than the threshold If the score is 0.2 (determined by maximizing the F1-score on the validation set), then the appearance feature is considered to be successfully matched.

[0035] Cost matrix fusion: The total cost matrix is ​​a weighted combination of the two metrics.

[0036] in The weighting coefficient is set to 0.5 in this scheme, and is applied only when... and Only when the value is zero is the corresponding cost term included in the matrix; otherwise, it is set to infinity (indicating that matching is not allowed). Cascaded matching strategy: To address the increased uncertainty in trajectory prediction after prolonged occlusion, cascaded matching prioritizes recently matched trajectories. The specific steps are as follows: Trajectories are divided into different priorities based on their proximity to the most recent successful match time, with the most recently matched trajectory participating in the matching process first. For each priority level, a cost matrix is ​​constructed and the matching is solved using the Hungarian algorithm; Unmatched detection boxes proceed to the next priority level for further matching; The remaining unmatched detection boxes are used as candidates for initializing new trajectories. They need to be successfully matched for 3 consecutive frames to be confirmed as official trajectories. Tracks with more than a threshold of 30 consecutive unmatched frames are deleted; The tracker outputs the following information for each active target: Unique Identifier ID: Integer type, assigned incrementing from 1; Current frame bounding box: The format has been mapped back to the original image coordinates; Historical trajectory coordinates: The center point coordinate sequence of the most recent 30 frames, used for subsequent velocity calculation and trajectory smoothing; Motion state: including instantaneous velocity and acceleration (provided by the Kalman filter state vector); Appearance feature vector: 128-dimensional floating-point vector, cached in the feature library for subsequent re-identification; The above information is stored in a structured data format in the real-time database of the edge server for use by the downstream behavior analysis module. The behavior analysis module calculates features such as gait cycle, dwell time, and activity area based on trajectory time series data, and combines the posture estimation results to make preliminary judgments on behaviors such as prolonged immobility, limping, and abnormal eating. The judgment results trigger the edge-cloud collaboration mechanism to upload abnormal event data to the cloud for refined analysis. Subsequently, based on the detection and tracking results, the edge computing layer further extracts refined posture information of individual pigs and sets rules for preliminary abnormal behavior identification, as follows: A coordinate regression-based facial landmark detection algorithm (adapted to pig facial structure) was adopted to regress 68 pig facial landmarks (including nose tip, corners of eyes, corners of mouth, and ear roots). To achieve real-time inference, MobileNetV2 was used as the backbone network, and an adaptive Wing loss function was introduced to optimize landmark localization accuracy. The loss function is defined as follows:

[0037] in =10, =2, This loss function pays more attention to small errors, accelerates convergence, and reduces the normalized mean square error (NMSE) at key points to 3.01%. Based on the keypoint coordinates, calculate the following biometric features: Eye aspect ratio: used to determine the state of eye opening and closing, combined with time series analysis of blink frequency, to initially identify signs of fatigue; Head Euler angles: Pitch angle, yaw angle and roll angle are calculated by the geometric relationship of key points to determine whether the pig has been looking down or to the side for a long time (which may indicate lethargy). Trunk skeleton extraction: Key points are used to fit the outline of the pig's trunk, and the back bending angle, gait cycle, etc. are extracted for lameness detection. Establish initial screening rules for abnormal behavior based on thresholds and logical judgments to reduce the amount of data uploaded to the cloud while enabling real-time on-site alarms: Prolonged lying down detection: If a pig target moves less than a preset threshold (e.g., 0.5 m / min) within 30 consecutive frames, and its posture is classified as lying on its side or lying on its back, it is marked as "suspected prolonged lying down" and local recording is triggered; Limp detection: Extract the vertical movement trajectory of the head and hips during the gait cycle. If the left and right stride asymmetry coefficient exceeds 0.3, it is marked as "suspected limp". Cough and sneeze detection: Combining audio energy characteristics with changes in the opening and closing of key points of the mouth, if more than 3 high-frequency short-duration audio pulses are detected within 1 second, it is marked as "suspected cough"; Abnormal eating habits: Count the time each pig spends in the feeding area. If it is less than 30% or more than 150% of the normal feeding time, it is recorded as "abnormal eating habits". The initial screening results are stored as structured data (JSON format), including: timestamp, pig ID, anomaly type, confidence level, keyframe image (JPEG compressed), and associated environmental parameters (temperature, ammonia concentration). This type of data is first uploaded to the cloud, while the edge server locally retains the original video clips from the last 7 days for traceability. To balance real-time performance and recognition accuracy, a collaborative working mode is established between the edge computing layer and the cloud data processing layer: Uplink data filtering: Only event data containing initial screening anomalies (including keyframes, audio clips, and environmental parameters) are uploaded to the cloud, while normal data only retains metadata (such as statistical information and trajectory summaries), which greatly reduces bandwidth pressure; Model updates: Optimized model parameters are periodically distributed from the cloud to the edge server to enable online iterative updates of the model, adapting to dynamic factors such as seasonal changes and pig growth stages in the farm. Fault self-healing: When the edge server detects that the GPU is overheating or the algorithm process has crashed, it automatically restarts the backup container and records the exception log; if the network is interrupted, the data is temporarily stored locally and resumed according to the timestamp after the network is restored.

[0038] In one embodiment, the lightweight deep learning model deployed in the edge computing layer needs to be trained and validated offline before it can be used. The model training process involves several key technical aspects, including training dataset construction, hyperparameter setting, attention mechanism parameter optimization, and transfer learning strategies. The specific process is as follows: To achieve robust detection of pig targets under different farming scenarios, a training dataset containing heterogeneous samples from multiple sources was constructed. The data sources include: (1) raw video data collected continuously for 12 months by 23 fixed cameras and 5 track-mounted robots deployed in the cooperative farm, which were processed by frame extraction to obtain approximately 150,000 images; (2) 20,000 images selected from publicly available pig image datasets (such as the IPig dataset and the Nongda pig posture dataset) to enhance sample diversity; The dataset was constructed taking into full account the complexity of the aquaculture environment, specifically including the following dimensions: Age coverage: The sample covers four growth stages: suckling piglets (0-30 days old), nursery pigs (31-70 days old), fattening pigs (71-180 days old) and breeding sows. The sample size of each stage accounts for 15%, 25%, 45%, and 15% respectively, to ensure the model's adaptability to different body size characteristics. Breed diversity: It includes 8 common breeds such as Large White, Landrace, Duroc, Tibetan pig and local crossbred pig. Different coat colors (white, black, spotted) and facial features (deep wrinkles, drooping or erect ears) are covered in the sample. Lighting conditions: The data collection period covered three types of scenes: daytime (natural light), nighttime (infrared supplemental lighting), and dawn / dusk (low light). Histogram equalization and adaptive gamma correction were used to enhance the low-light samples to simulate different lighting conditions. Posture and Occlusion: Includes 12 types of postures such as standing, feeding, lying on one's side, lying on one's back, walking, and fighting, as well as partial occlusion (pen occlusion, occlusion by other pigs) and dense scene samples to improve the model's generalization ability in complex scenes; All images were manually annotated using the LabelImg tool. The annotations included: (1) target bounding boxes, defining the head or whole body area of ​​each pig; (2) key point annotations, with 68 additional facial key points annotated for 20,000 samples (adapted to the physiological structure of pigs); (3) attribute labels, including breed, age stage, posture category, etc. The dataset was randomly divided into training set, validation set and test set in a ratio of 8:1:1. The K-means clustering algorithm was used to perform cluster analysis on the bounding box sizes in the training set, and preset anchor box values ​​adapted to the pig targets were regenerated. Online data augmentation strategies were employed during training, including random horizontal flipping (probability 0.5), random brightness adjustment (-20%~+20%), random saturation adjustment (-20%~+20%), random cropping (preserving the original image area of ​​0.6~1.0), and mosaic enhancement (randomly stitching together 4 images before training) to simulate various changes that may occur in the actual aquaculture environment and improve the robustness of the model. The training parameters of the YOLOv4-Tiny model were improved. Model training was performed on a cloud server, employing synchronized batch normalization (BN) to ensure the consistency of batch normalization layer statistics during multi-GPU training. The training hyperparameters are set as follows: Optimizer: Adam, initial learning rate 0.001, momentum factor 0.937, weight decay 0.0005; Batch size: 256 (64 per card); Training epochs: 300; Learning rate adjustment strategy: Cosine annealing decay is adopted. The first 5 epochs are used for warm-up training, and the learning rate is linearly increased from 0.0001 to 0.001. After that, the learning rate is decayed to 0.1 times the original value every 30 epochs. Loss function: CIoU loss is used as the bounding box regression loss, and binary cross-entropy loss is used as the classification loss. The total loss function is the weighted sum of the two, with weight coefficients of 0.05 and 0.01, respectively. During training, model performance was evaluated on the validation set every 10 epochs, and the model weight file with the best mean average precision (mAP) was saved. Testing showed that the improved YOLOv4-Tiny model achieved an mAP of 93.7% on the validation set, a 5.2 percentage point improvement over the original model, and achieved a detection speed of 28fps on the Jetson AGX Xavier platform, meeting real-time requirements. The parameters of the GCA attention module are determined, including the parameters of the adaptive formula for the one-dimensional convolution kernel size in the GCA attention module. =2 and b=1 were determined through cross-validation experiments. The specific process is as follows: First, based on the improved YOLOv4-Tiny model, keeping other structures unchanged, only the calculation method of the one-dimensional convolutional kernel size in the GCA module is adjusted. The relationship between the convolutional kernel size y and the number of input channels C is defined as follows:

[0039] On the validation set, a grid search method is used. , A total of 12 parameter combinations were tested. Each parameter combination was trained 3 times, and the average mAP was used as the evaluation metric. Experimental results show that when When b=2 and b=1, the model achieves the highest mAP on the validation set while having the smallest parameter increment (only 461 parameters added). When =1, the kernel size is too large, which can capture more cross-channel interaction information, but introduces redundant parameters and leads to overfitting; when When b=3 or b=4, the kernel size is too small, resulting in insufficient information exchange between channels and a decrease in model accuracy. When b=0, truncation error is prone to occur when adjusting the kernel size to an odd number in low-channel layers (e.g., C=64). b=1 can effectively compensate for this deviation.

[0040] Furthermore, in terms of the trade-off between the number of parameters and accuracy, the GCA module achieves an accuracy improvement with only 461 additional parameters compared to the traditional SE module (approximately 3008 additional parameters) and GC module (approximately 4051 additional parameters), demonstrating the effectiveness of the selected parameter combination.

[0041] VOVNetV1 employs a two-stage training strategy to balance recognition accuracy and training efficiency, as detailed below: Phase 1: Pre-training. Since pig re-identification and pedestrian re-identification tasks share similarities in target pose variations and viewpoint diversity, pre-training was first performed on the large-scale pedestrian re-identification dataset Market-1501. The Market-1501 dataset contains 1501 pedestrian identities and 32668 bounding box images, covering various poses, lighting conditions, and backgrounds. The VOVNetV1 network was trained on 120×60 images as input, employing data augmentation techniques such as random horizontal flipping, random erasing, and random rotation (±10°). The network was trained for 60 epochs using the SGD optimizer with an initial learning rate of 0.1, decaying by a factor of 0.1 every 20 epochs. After pre-training, the network achieved an mAP of 79.4% and a Rank-1 accuracy of 88.6% on the Market-1501 query set, indicating that the network possesses good feature extraction capabilities. Phase 2: Fine-tuning. A pig re-identification dataset was constructed. Images of 50 pigs with identification tags were selected from our own dataset, with at least 50 images per pig (approximately 3000 images in total), covering different ages, shooting angles, and lighting conditions. The output dimension of the fully connected layer of the pre-trained model was modified to the number of pig identities (50 classes). The parameters of the first two OSA modules were frozen, and only the last two OSA modules and the fully connected layer were fine-tuned. The Adam optimizer was used with an initial learning rate of 0.001, a batch size of 32, and training for 30 epochs, with the learning rate decreasing by 0.1 times every 10 epochs. After fine-tuning, the model achieved a Rank-1 accuracy of 94.2% on the pig test set, validating the effectiveness of transfer learning. For tracking scenarios without identity labels, the classification layer is removed, and a 128-dimensional feature vector is extracted as appearance features. In the DeepSORT tracker, this feature vector is used to calculate the minimum cosine distance, which, together with the motion features, constructs the cost matrix. Compared to models trained directly from scratch, the pre-training + fine-tuning strategy can reduce training time by approximately 40% and exhibits better discriminative ability on challenging samples such as those with occlusion or drastic pose changes.

[0042] The cloud-based data processing layer, deployed in a public or private cloud data center, serves as the core computing and decision-making hub of the system, undertaking the functions of deep fusion, refined identification, knowledge accumulation, and model evolution of multimodal data. This layer receives key data and suspected abnormal fragments uploaded by the edge computing layer via the Internet, and utilizes the abundant computing resources in the cloud (CPU clusters, GPU arrays, and distributed storage systems) to run large-scale deep learning models, achieving accurate diagnosis of the health status of pigs and deep understanding of their behavioral semantics. The cloud-based data processing layer adopts a hybrid cloud architecture, with the master node deployed on cloud servers for large-scale model training and inference. The storage system uses distributed object storage (OSS) to store raw video clips, labeled data, and model version files. The resource scheduling layer is built on a Kubernetes containerized platform, supporting multi-tenant isolation and elastic scaling, dynamically adjusting computing resource allocation based on task load. Data transmission uses SSL / TLS encrypted channels to ensure data security and integrity during transmission. The cloud data processing layer receives data packets uploaded by the edge computing layer via a RESTful API interface. The data packets are formatted using Protocol Buffers serialization to reduce transmission overhead. Each data packet contains the following fields: Metadata: Farm ID, Pig House ID, Data Collection Node ID, Timestamp (Unix millisecond level), Data Packet Sequence Number; Abnormal event data: Abnormal type (prolonged lying down, lameness, coughing, abnormal eating, etc.), confidence level, and list of associated pig IDs; Keyframe images: JPEG format compressed image sequence, no less than 10 frames per event, resolution 1280×720; Audio clip: WAV format, sampling rate 44.1kHz, duration 3~5 seconds, including periods of abnormal sound; Environmental parameter time series data: JSON format, containing parameters such as temperature, humidity, ammonia concentration, and light intensity within 30 minutes before and after the anomaly occurred, with a sampling interval of 1 minute; Trajectory data: a sequence of trajectory coordinates associated with pig IDs (center point of each frame), spanning 5 minutes before and after the anomaly occurred; After receiving the data, an integrity check is first performed using MD5 hash comparison to ensure that the data packets are not corrupted. Then, clock calibration is performed, converting the local timestamps of each node to the standard time synchronized by the Network Time Protocol (NTP). For video stream data, FFmpeg is used to decode it into YUV420P raw format, and it is scaled to the model input size as needed. Audio data is pre-emphasized (pre-emphasis coefficient 0.97), then framed (frame length 25ms, frame shift 10ms), Hamming windows are added, and 40-dimensional Mel-frequency cepstral coefficients (MFCCs) and their first and second-order differences are extracted to form a 120-dimensional feature vector sequence. Environmental parameter time-series data is filled with missing values ​​using linear interpolation and normalized to the [0,1] interval. To overcome the misjudgment problem caused by insufficient information from a single modality, this solution constructs a multimodal deep fusion network based on multi-task learning to achieve joint modeling and collaborative decision-making of visual, audio, and environmental temporal data. This network is based on the Transformer architecture and uses a cross-modal attention mechanism to align and fuse heterogeneous features. The multimodal deep fusion network consists of four main modules: a single-modal feature encoder, a cross-modal feature fusion unit, a multi-task decoder, and a joint decision module. The single-modal feature encoder specifically includes: a visual encoder using ResNet50 as the backbone network, with input being a sequence of keyframe images (T frames, T=10). First, a 2048-dimensional feature vector is extracted from each frame, forming a T×2048 visual feature matrix. To capture temporal dynamic information, two layers of bidirectional long short-term memory (BiLSTM) are connected after ResNet50, with a hidden layer dimension of 512, outputting T×1024 visual temporal features. The audio encoder takes an MFCC feature sequence (frame number L, determined by the audio duration, approximately 150 frames) as input and employs a three-layer one-dimensional convolutional neural network (Conv1D) to extract local acoustic patterns. The kernel sizes are 3, 5, and 3, with a stride of 1 for each layer, resulting in output channels of 64, 128, and 256 respectively. A global average pooling layer follows the convolutional layers to obtain a 256-dimensional audio feature vector. To align the audio and visual features with the temporal dimension, the audio feature vector is copied T times, forming a T×256 audio feature matrix. An environmental encoder takes temporal data of environmental parameters (time points M, M=30) as input, including four dimensions: temperature, humidity, ammonia concentration, and light intensity, forming an M×4 environmental feature matrix. A two-layer Transformer encoder (4 heads, 128 hidden layer dimensions) is used to extract the temporal dependencies of the environmental parameters, outputting M×128 environmental temporal features. Linear interpolation is then used to align the M time points to T visual frames, obtaining T×128 aligned environmental features. The cross-modal feature fusion engine employs a multi-head cross-attention mechanism, using visual features as the query and audio and environmental features as the key and value, respectively, to achieve the interaction and fusion of multimodal information. The specific calculation process is as follows: visual features The query matrix is ​​obtained through linear transformation. ,in .

[0043] Audio features With environmental characteristics Concatenating along the feature dimensions yields multimodal key / value features. Then, the key matrix is ​​obtained through linear transformation. Sum matrix ,in , ; Calculate the cross-attention output:

[0044] in The scaling factor is used. An 8-head attention mechanism is employed, where each head computes independently, the data is concatenated, and then linearly transformed again to obtain the fused multimodal features. ; Fusion features With original visual features Residual connections are performed, and layer normalization (LayerNorm) is used to obtain the final multimodal fused feature representation:

[0045] Based on the fused multimodal features, three parallel task decoders are set up to complete recognition tasks of different granularities: Fine-grained behavior classifier: Global average pooling is performed along the time dimension to obtain a 512-dimensional global feature vector, which is then input into a three-layer fully connected network (512→256→128→C), where C is the number of behavior categories (set to 12 categories in this system: normal standing, eating, drinking, lying on one's side, lying on one's back, walking, fighting, limping, prolonged lying down, coughing, sneezing, and screaming). The output layer uses the Softmax activation function to output the probability distribution of behavior categories. Anomaly severity regressor: A two-layer fully connected network (512→128→1) is used. The output layer employs the Sigmoid activation function, mapping the output to the [0,1] interval to represent the severity score of the anomaly. This score is weighted by combining the behavior category and duration. During training, anomaly levels (continuous values ​​from 0 to 1) labeled by experts are used as supervision signals. Temporal Localizer: A one-dimensional convolutional neural network is used to classify FoutFout frames one by one to identify the start and end times of abnormal behavior. Convolutional Layer Configuration: Two Conv1D layers with a kernel size of 5, a stride of 1, and output channels of 256 and 128 respectively. A fully connected layer is then used to output the binary classification probability (normal / abnormal) for each frame. The system is trained using a frame-by-frame cross-entropy loss function. The joint decision-making module fuses the outputs of the multi-task decoder to generate the final comprehensive diagnostic result. The fusion strategy employs a weighted voting mechanism, with behavioral classification results accounting for 0.5 weights, anomaly score accounting for 0.3 weights, and temporal location reliability accounting for 0.2 weights. For samples with classification confidence scores below the threshold (0.7), a manual review process is triggered, pushing the data to the expert annotation platform. The specific process is as follows: Training strategy and loss function: The multimodal deep fusion model adopts a two-stage training strategy. Pre-training phase: Pre-training is performed on large-scale general behavior recognition datasets (such as Kinetics-400 and AudioSet) to enable the model to learn general visual-audio joint representations. During pre-training, only the visual encoder and audio encoder are trained, while the environment encoder and multi-task decoder are frozen. The cross-entropy loss function is used, the optimizer is Adam, the initial learning rate is 0.001, the batch size is 64, and the training lasts for 50 epochs. Fine-tuning phase: End-to-end fine-tuning was performed on a self-built multimodal pig dataset. The dataset contains 5000 labeled samples, each consisting of a 10-second video, corresponding audio, and a sequence of environmental parameters. Livestock experts labeled the behavioral category, anomaly level, and start and end times of the anomaly. A multi-task joint loss function was used during fine-tuning.

[0046] in Cross-entropy loss for behavior classification, The mean squared error loss is used for regression of anomaly severity. Frame-by-frame cross-entropy loss for temporal localization. Weighting coefficients. , , The values ​​were set to 1.0, 0.5, and 0.8 respectively, and then optimized and determined on the validation set using grid search. Data augmentation strategies were employed during training: random cropping, random flipping, and random brightness adjustment of video frames; addition of Gaussian white noise (signal-to-noise ratio 15~25dB) and time scaling (0.8~1.2 times) to audio; and random perturbation (±5%) to environmental parameters. An early stopping strategy was adopted, terminating training when the validation set loss did not decrease for 10 consecutive epochs. Fine-grained identification and anomaly confirmation: Based on the output of the multimodal deep fusion model, the cloud data processing layer further refines the identification and confirmation of suspected abnormal events uploaded by the edge computing layer, reducing the false alarm rate and generating interpretable diagnostic reports. Behavioral time-series analysis, for events marked as "suspected prolonged lying down" at the edge computing layer, the cloud model uses a time-series locator to accurately identify the start and end times of the lying down behavior, calculates the duration of lying down, and compares it with the normal lying down duration threshold for pigs of that age (derived from historical data). If the duration exceeds the threshold (e.g., fattening pigs lying down continuously for more than 2 hours) and the abnormality score is greater than 0.6, it is confirmed as "abnormal prolonged lying down," and further analysis of environmental parameters (such as whether the temperature is too high or whether the ammonia concentration exceeds the standard) is conducted to help determine the cause. For suspected limp events, the cloud-based model extracts the vertical movement trajectories of the head and hips during the gait cycle, and calculates the left-right stride symmetry coefficient and gait frequency stability index. If the symmetry coefficient is less than 0.7 and the gait frequency variation coefficient is greater than 0.2, it is confirmed as an "abnormal limp." The model also outputs the specific leg involved in the limp (left front, right front, left rear, right rear), achieved by analyzing the contact time between key point trajectories and the ground. For suspected cough incidents, the cloud-based model combines the MFCC features output by the audio encoder with the mouth opening and closing movements detected by the visual encoder to determine the authenticity of the cough. If a cough sound is detected in the audio features (classified by the AudioSet pre-trained model), and the mouth opening and closing amplitude in the visual features is synchronized with the audio pulse, it is confirmed as an "abnormal cough." The model further distinguishes between dry cough and wet cough (through audio spectrum energy distribution), providing a basis for determining the type of respiratory disease. Cross-event correlation analysis involves establishing a time-series database in the cloud-based data processing layer to store historical health event records for each pig. When a new event is confirmed as abnormal, the system automatically retrieves relevant events for that pig from the past 7 days and performs spatiotemporal correlation analysis. Frequency analysis of similar events: Count the number of similar abnormal events that occurred in the past 24 hours. If the number exceeds the threshold (e.g., coughing events > 10 times / hour), the warning level will be upgraded. Cross-event causal chain analysis: Detects the temporal correlation between abnormal events, such as whether respiratory symptoms (coughing, sneezing) appear within 24 hours after an abnormal environmental parameter (excessive ammonia concentration). If so, it is marked as "environmentally induced type" and it is recommended to adjust the environmental parameters first. Group transmission risk assessment: If multiple pigs in the same pen show similar abnormalities within a similar time period (e.g., more than 3 pigs coughing within 24 hours), it is marked as "group transmission risk", and the group isolation and comprehensive testing process is initiated. After detailed identification is completed, the cloud data processing layer generates a structured diagnostic report, which includes the following: Basic event information: time, location (farm - pigsty - pen), pig ID, event type; Identification results: Behavior category (including confidence level), anomaly score (0~1), and anomaly start and end time; Multimodal evidence: keyframe images (annotated abnormal areas), audio waveforms (annotated abnormal time periods), and time-series curves of environmental parameters; Assisted diagnostic suggestions: Suggestions generated based on historical data and expert rule base, such as "suggest checking feed palatability", "suggest testing for ammonia sensor malfunction", and "suggest isolation and observation and collection of blood samples". Warning levels: Based on the severity score and transmission risk assessment results, they are divided into three levels: Attention (0.4~0.6), Warning (0.6~0.8), and Severe (0.8~1.0). The diagnostic report is stored in a cloud database in JSON format and is sent to the early warning and application layers in real time via a push notification mechanism. The model self-evolution and knowledge accumulation, the cloud data processing layer has the ability to continuously learn and iterate the model, and continuously improves the recognition accuracy and generalization ability through a closed-loop feedback mechanism; Difficult example mining and active learning automatically select samples with confidence scores below a threshold (0.7) or whose labels are corrected after manual review and add them to the difficult example pool. Weekly sampling (using a combination of uncertainty sampling and diversity sampling) is performed from the difficult example pool, and the samples are manually labeled by livestock experts before being added to the training set. Monthly retraining of the multimodal deep fusion model is based on the updated training set, and performance improvements are evaluated on the validation set. If the new model improves the F1-score on the validation set by more than 1 percentage point, the cloud-based production model is automatically replaced, and updates are distributed to the edge computing layer through a model version control mechanism. The system integrates an expert rule base and a built-in expert rule engine, transforming the experience and knowledge of livestock experts into executable rules for collaborative decision-making with model output. Rule example: If the temperature is >30℃ and the humidity is >80% and the pigs are lying on their sides and panting, then the warning level is increased by 1, and it is recommended to "start cooling and ventilation". If ammonia concentration >20ppm and pigs cough more than 5 times / hour, then warning level +1, and it is recommended to "check the manure removal system"; If a pig is lame and its body temperature is >40℃, then the alert level is +2, and it is recommended to "immediately isolate the pig and sample it for foot-and-mouth disease testing." The rule engine is implemented using the Drools framework, supporting dynamic creation, deletion, modification, and querying of rules. The rule execution results serve as auxiliary information in the model output and are integrated into the final diagnostic report. The system constructs a digital profile for each partner farm, recording information such as farm size, breed composition, historical disease outbreaks, environmental characteristics, and management practices. Based on this profile information, the model is fine-tuned, such as adjusting behavioral classification thresholds for specific breeds (e.g., local black pigs) and increasing the weighting of respiratory disease detection for specific seasons (e.g., winter). After the personalized model is generated in the cloud, it can be distributed to the edge servers of the corresponding farms through an edge-cloud collaboration mechanism, achieving localized adaptation of the model.

[0047] The early warning and application layer is deployed in the farm monitoring center and on the mobile terminals of management personnel. As the final output port of the system, it is responsible for the visualization of abnormal events, multi-level early warning push notifications, and decision support functions. This layer receives diagnostic results from the cloud data processing layer in real time via the WebSocket protocol. Combined with the farm's electronic map and pig identification information, it generates an intuitive early warning interface and pushes it to relevant responsible persons through multiple channels. The early warning and application layer specifically includes a visual monitoring screen and a mobile APP; The large visual monitoring screen is deployed in the central control room of the farm, and the screen interface is divided into four functional areas: Panoramic map of the pig farm: The left-hand area displays the floor plan of the pigsty, with each pen marked with a color-coded block indicating its health status (green: normal, yellow: caution, orange: warning, red: serious). Clicking on a specific pen will bring up a list of the health status of all pigs in that pen, which can be filtered by abnormality type. Real-time video preview: The central area displays a carousel of real-time video streams uploaded by fixed nodes and mobile robots. When an abnormal event occurs, it automatically switches to the view from the associated camera, overlaying the pig's ID box, the abnormality type label, and the confidence level onto the screen. The video stream uses the HLS protocol for transmission, with latency controlled to within 2 seconds. Abnormal Event List: The upper right area displays the latest abnormal events in a time-series table format, including fields such as time, column number, pig ID, abnormality type, severity level, and processing status (pending / processing / processed). The list supports filtering by level, type, and status, and unprocessed events are highlighted. Environmental parameter curves: The lower right area displays historical curves of environmental parameters such as temperature, humidity, and ammonia concentration for the selected column in real time (the past 24 hours are displayed by default), and is associated with the timeline of abnormal events to help managers analyze the correlation between environmental factors and abnormal behavior. The mobile app is designed for breeders, veterinarians, and farm managers, supports both Android and iOS platforms, and is developed using the Flutter framework. The app's functional modules include: Alert Push Notifications: Integrating the JPush SDK, when a new abnormal event is generated in the cloud, a targeted notification is pushed based on the event level and the recipient's role (keepers receive alerts for this section, veterinarians receive alerts for the entire group). The notification includes the abnormality type, location, and a brief suggestion; clicking the notification will take you to the event details page. Event Details Page: Displays complete information about the abnormal event, including the pig's file (age, breed, immunization history), multimodal evidence (keyframe images, audio waveforms, environmental parameters), diagnostic recommendations, and treatment records. Veterinarians can submit treatment opinions online and upload image evidence; treatment records are synchronously updated to the cloud database. Task Orders: The system automatically generates processing orders and assigns them to the corresponding zookeepers, including processing time limits (general anomalies to be processed within 4 hours, severe anomalies to be processed within 1 hour). Zookeepers can mark the status of the work orders (received, dispatched, arrived, completed), and the system records the time of each milestone for performance evaluation. Data statistics: The farm's weekly / monthly health reports are displayed in chart form, including indicators such as abnormal event trends, percentage of various abnormalities, average response time, and processing completion rate. Data can be exported to Excel or PDF. The aforementioned early warning and application layer has a multi-level early warning mechanism. Based on the anomaly severity score and the propagation risk assessment results, the system can set up a three-level early warning mechanism: Attention Level (Score 0.4~0.6): A single pig exhibits minor abnormalities, with no risk of transmission. A notification is sent to the corresponding pen's APP; the corresponding pen will be displayed in yellow on the large screen. The system records this but does not require immediate action; close observation is recommended. Warning level (score 0.6~0.8): A single pig shows obvious abnormalities, or 2~3 pigs in the same pen show similar abnormalities. A notification is sent to the feeder and the on-duty veterinarian. The corresponding pen on the large screen displays orange, and the system generates a processing work order, requiring on-site verification and feedback to be completed within 4 hours. Severe Level (Score 0.8~1.0): A single pig exhibits severe abnormalities (such as high fever, severe lameness, persistent cough), or more than 3 pigs in the same pen exhibit similar abnormalities (risk of group transmission). The system pushes the information to the feeder, veterinarian, and farm manager. The corresponding pen on the large screen displays red and flashes. The system generates an expedited work order, requiring isolation and sampling to be completed within 1 hour, and automatically notifies the epidemic prevention department for record-keeping. The early warning and application layer provides early warning information that, in addition to displaying abnormal states, also includes auxiliary decision-making suggestions generated based on expert rule bases and historical data analysis. Environmental control recommendations: such as "The current ammonia concentration exceeds the standard, it is recommended to start the ventilation fan" and "The temperature is too high, it is recommended to turn on the water curtain to cool down"; Isolation recommendations: such as "If a respiratory infectious disease is suspected, it is recommended to immediately transfer the patient to an isolation ward and collect a blood sample for testing"; Recommended treatment plan: For example, "If bacterial pneumonia is suspected, it is recommended to use ceftiofur injection at a dose of 0.1 ml / kg of body weight." Vaccination booster reminder: such as "This pig is due for a foot-and-mouth disease booster immunization; it is recommended to complete the vaccination within 3 days"; Decision recommendations are displayed as pop-ups on the event details page. Veterinarians can adopt or reject them. If adopted, they are automatically linked to the processing work order. Warning notifications are not the end point. The system is designed with a closed-loop data mechanism: after the zookeeper or veterinarian completes on-site treatment, they need to upload the treatment results (such as "isolated," "medicated," or "recovered") to the APP. The system then feeds the results back to the cloud for model iteration and optimization. If the result is a "false alarm," the sample is added to the difficult case pool, triggering manual review and model correction processes.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A pig abnormal behavior recognition and early warning system based on Internet of Things and deep learning, characterized in that, include: The data acquisition layer is deployed inside and around the pig farm to achieve real-time, multi-source, all-weather data acquisition of the farming environment, individual pig behavior, and physiological acoustic characteristics. The edge computing layer is deployed in the industrial control room at the breeding farm site and communicates with the data acquisition layer to perform lightweight real-time processing and initial screening of abnormal events on the acquired multimodal data. The cloud data processing layer connects to the edge computing layer via the Internet to perform multimodal deep fusion analysis and fine identification of abnormal event data; The early warning and application layer communicates with the cloud data processing layer to generate and push multi-level early warning information and auxiliary decision-making suggestions. The data acquisition layer includes a fixed data acquisition node array and a track-mounted mobile acquisition robot, which synchronize data and interact with commands through an industrial-grade wireless self-organizing network protocol.

2. The pig abnormal behavior recognition and early warning system based on Internet of Things and deep learning according to claim 1, characterized in that, The fixed data acquisition nodes are arranged in a grid pattern according to the layout of the pigsty. Each node integrates a visual acquisition module, an environmental perception module, and an audio acquisition module, wherein: The visual acquisition module uses a wide dynamic range CMOS image sensor and has a built-in near-infrared fill light, which automatically switches to infrared mode at night. The environmental sensing module includes a temperature and humidity sensor, an ammonia concentration sensor, a carbon dioxide sensor, and a light intensity sensor. The audio acquisition module uses a MEMS microphone array and is equipped with a front-end digital signal processor for preliminary noise reduction. 3.The IoT and deep learning based pig abnormal behavior recognition and early warning system according to claim 1, characterized in that, The track-mounted mobile data collection robot includes: The positioning and navigation module uses a low-frequency RFID reader in conjunction with electronic tags embedded on both sides of the track to achieve absolute position identification; The gimbal and robotic arm mechanism includes a high-definition visible light camera and an infrared thermal imager mounted on the gimbal, and a proximity sensor at the end of the robotic arm. The task triggering mechanism automatically plans a path to move to the target pen when a fixed node detects suspected abnormal behavior or receives a cloud scheduling instruction, and then tracks and photographs the suspected pigs from multiple perspectives.

4. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 1, characterized in that, The edge computing layer is deployed with a lightweight deep learning model, including: An improved YOLOv4-Tiny target detector, in which the backbone network uses depthwise separable convolutions instead of standard convolutions, adds a third detection branch to handle small targets, and embeds a global context attention module; The improved DeepSORT multi-target tracker upgrades the motion model to a uniform acceleration model, replaces the re-identification network with a lightweight network built on the OSA module, and optimizes the cascade matching strategy.

5. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 4, characterized in that, The global context attention module in the improved YOLOv4-Tiny object detector replaces the fully connected layer with a one-dimensional convolution, and the convolution kernel size y is adaptively determined by the number of channels C: ; in, This indicates taking the nearest odd number upwards, with the parameter... The slope of the logarithmic mapping between the number of control channels and the kernel size, where b is the bias term.

6. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 1, characterized in that, The edge computing layer is also equipped with a behavior screening module, which calculates gait cycle, dwell time and activity area based on trajectory time data, and combines the key point information of the pig's face to set threshold rules to make preliminary judgments on prolonged immobility, lameness, coughing and sneezing, and abnormal eating. Only the preliminary screening abnormal event data is uploaded to the cloud data processing layer.

7. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 1, characterized in that, The cloud data processing layer includes a multimodal deep fusion network, which comprises: A single-modal feature encoder processes visual, audio, and environmental temporal data separately. The cross-modal feature fusion machine employs a multi-head cross-attention mechanism, using visual features as queries and audio and environmental features as keys and values. A multi-task decoder, including a fine-grained behavior classifier, an anomaly severity regressor, and a temporal locator; The joint decision-making module weights and fuses the outputs of the multi-task decoder to generate the final diagnostic result.

8. The abnormal behavior recognition and early warning system for pigs based on the Internet of Things and deep learning according to claim 7, characterized in that, The cloud data processing layer also includes a model self-evolution module, which automatically filters samples with confidence levels below a threshold or whose labels are corrected after manual review, adds them to a difficult example pool, periodically expands the training set by expert annotation, and retrains the multimodal deep fusion model based on the updated training set to achieve continuous improvement in recognition accuracy.

9. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 1, characterized in that, The early warning and application layer includes: The large visual monitoring screen displays a panoramic map of the farm, real-time video previews, a list of abnormal events, and environmental parameter curves. The mobile app provides alert push notifications, event details, task orders, and data statistics. The multi-level early warning mechanism is divided into three levels: attention level, warning level, and severe level, based on the score of the degree of abnormality and the results of the risk assessment of transmission. Each level corresponds to a different processing time limit and responsible person.

10. The abnormal behavior identification and early warning system for pigs based on the Internet of Things and deep learning according to claim 9, characterized in that, The early warning and application layer also includes an expert rule base, which transforms the experience and knowledge of livestock experts into executable rules and makes collaborative decisions with the output of deep learning models. When specific conditions are met, the early warning level is automatically adjusted and auxiliary decision-making suggestions are generated, including environmental control suggestions, isolation suggestions, treatment plan recommendations, and vaccine booster reminders.

Images