An intelligent pig farming management and control system based on multimodal data monitoring

Abstract

The present application discloses an intelligent pig farming management and control system based on multimodal data monitoring, which belongs to the field of pig farming management technology, including: a data acquisition and individual identification module including RFID ear tags, inspection robots, environmental sensors and intelligent feeders, which determines the individual identity information of pigs based on the data collected by RFID ear tags and inspection robots, and associates the multimodal data of pigs in the pig house for preprocessing through the MQTT protocol; a feature extraction module uses a multimodal deep learning fusion model to extract the physiological and behavioral characteristics of pigs; a key breeding indicator analysis module constructs a pig health index and an environmental quality index; an abnormal warning module uses a CUSUM control chart to accumulate index deviations and respond to abnormalities; a management and control model constructs a Markov decision process with energy consumption minimization and health index maximization, and intelligently adjusts the fan speed and feeding strategy to achieve precise environmental control and feeding optimization.

Description

Technical Field

[0001] The present application relates to the technical field of pig farming management, and in particular to an intelligent pig farming management and control system based on multimodal data monitoring. Background Art

[0002] The swine industry is a vital component of global agricultural production, providing a crucial guarantee for meeting the growing demand for meat. Currently, swine farming is gradually moving towards intensification, scale, automation, and factory-based operations. Pig health monitoring plays a crucial role in pig farming management. Currently, actual production still relies on manual inspections to monitor pig health. This traditional manual inspection method has significant drawbacks: it relies on experience, is inefficient, is highly subjective, and lacks the ability to achieve high-frequency, continuous monitoring. This often leads to early signs of disease or sub-health conditions being overlooked, leading to significant delays in prevention and control. For example, manual observation cannot accurately capture key indicators such as abnormal body temperature and subtle changes in behavior patterns, which are often early signs of infectious diseases such as African swine fever and blue ear disease. Furthermore, as my country's swine farming transitions towards intensification and factory-based operations, stocking density is increasing, and the cost and difficulty of manual management are skyrocketing. Intelligent solutions are urgently needed to replace traditional methods.

[0003] In recent years, researchers have tried to improve monitoring efficiency through technical means. For example, wearable sensors, cameras and other equipment are used to collect color images, infrared thermal images, motion data, etc. of pigs, and deep learning algorithms are used for feature extraction. However, these technologies are mostly limited to the analysis of single-dimensional data such as sound, behavior or body temperature, and lack a comprehensive assessment of the overall health status of pigs. In addition, the equipment is easily damaged and expensive, making it difficult to promote on a large scale.

[0004] While existing intelligent farming systems have achieved breakthroughs in some areas, such as Shuanghui's automated grading of pork carcasses through an AI imaging grading system and Muyuan's use of building-style pig farming combined with intelligent environmental control equipment to improve biosafety, these systems primarily focus on environmental control or production process optimization, without deeply integrating multiple sources of data, such as physiology, behavior, and the environment, to achieve real-time, dynamic monitoring of health status. Furthermore, domestic intelligent equipment relies on imports and lacks independent research and development capabilities, resulting in high technology application costs and poor adaptability. For example, while intelligent inspection robots can accurately collect environmental parameters and pig body heat data, their deployment requires integration with IoT platforms and algorithm models. Currently, most farms lack such systematic solutions. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present application provides an intelligent pig farming management and control system based on multimodal data monitoring, which solves the problems of incomplete and inaccurate data collection, independence of environmental control and health monitoring, reliance of health assessment methods on empirical threshold judgment, inadequate intelligent early warning models and decision-making systems, and poor timeliness of stress event handling due to reliance on manual intervention.

[0006] In order to achieve the above-mentioned invention objectives, the technical solutions adopted in this application are:

[0007] The present application provides an intelligent pig farming management and control system based on multimodal data monitoring, including: a data acquisition and individual identification module, a feature extraction module, a key farming indicator analysis module, an abnormality warning module, and a management and control module;

[0008] The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeders. It determines the individual identity information of pigs based on the data collected by the RFID ear tags and inspection robots, and associates the multimodal data of pigs in the pig house collected in real time based on the determined individual identity information of pigs. It uses the NTP protocol for data alignment and uploads the data to the edge computing device on the inspection robot via the MQTT protocol for preprocessing.

[0009] The feature extraction module includes a multimodal deep learning fusion model, which uses the multimodal deep learning model to extract the physiological and behavioral characteristics of pigs from the preprocessed multimodal data;

[0010] The key breeding index analysis module constructs a pig health index based on the physiological and behavioral characteristics of the pigs extracted by the feature extraction module and the data collected by the intelligent feeder; and constructs an environmental quality index based on the environmental parameters collected by the environmental sensor;

[0011] The abnormality warning module accumulates the deviations between the pig health index and the environmental quality index and the target values ​​based on the CUSUM control chart and responds to abnormalities;

[0012] The management and control model includes an intelligent environmental control system and a feeding control module. A Markov decision process is constructed with minimizing energy consumption and maximizing health index as multi-objective optimization functions, and the DDPG algorithm is used to adjust the fan speed and feeding frequency.

[0013] Furthermore, the data collection and individual identification module specifically includes:

[0014] The RFID ear tag integrates a temperature sensor and an acceleration sensor to identify the pig and collect the pig's body temperature, location and behavior data; the inspection robot is equipped with color, infrared, and depth cameras and an ear tag data acquisition base station to collect RGB images, infrared thermal maps and depth point cloud data of the pig, and combines the RFID ear tag to achieve individual identification and data association; the environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor to monitor the environmental parameters in the pig house in real time; the smart feeder integrates a weight sensor and an RFID reader to record the duration and amount of each feeding of the pig, and obtains weight information of the pig through the weight sensor when the pig is feeding.

[0015] Furthermore, the determination of individual pig identity information based on data collected by the RFID ear tags and the inspection robot specifically includes:

[0016] The Deepsort multi-target tracking algorithm is used to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence for each pig;

[0017] The ST-LSTM network is used to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot to form a 6-dimensional vector composed of the acceleration data and the visual coordinates to infer the movement trajectory of the pig;

[0018] The Hungarian algorithm is used to match the ear tag ID with the pig's movement trajectory to determine the individual identity information of the pig.

[0019] Furthermore, the ST-LSTM network adopts a dual-branch structure with time-space separation. The spatial branch uses Conv1D to process acceleration data, and the temporal branch uses a unidirectional LSTM to process the visual coordinate sequence, and the dual-branch outputs are fused through a gated attention mechanism.

[0020] Furthermore, the multimodal deep learning model is used to extract the physiological and behavioral characteristics of pigs from the pre-processed multimodal data, specifically including:

[0021] A multimodal pig weight estimation model based on the fusion of RGB images and depth images extracts pig weight information;

[0022] The pig body temperature detection model based on the fusion of RGB images and infrared images measures the pig's body temperature information;

[0023] Based on the multimodal pig behavior recognition model that integrates pig skeleton information and RGB video stream, the time and frequency of various pig behaviors are extracted.

[0024] Furthermore, the multimodal pig weight estimation model that integrates RGB images and depth images includes:

[0025] Both the RGB branch and the depth branch use Swin Transformer as the feature extraction network to extract depth features and RGB features to obtain depth feature maps and RGB feature maps;

[0026] Use the cross-modal attention module to perform modal feature interaction between the depth feature map and the RGB feature map;

[0027] Perform 1×1 convolution operation on the modal features after fusion interaction to generate a fusion feature map;

[0028] Based on the fused feature map, the weight information of the pig is obtained.

[0029] Furthermore, the pig body temperature detection model based on the fusion of RGB images and infrared images measures the body temperature information of the pig, specifically including:

[0030] Use key point detection algorithm to detect the location of the pig ear root area on the RGB image;

[0031] The position of the pig's ear root area on the infrared image is obtained using the affine transformation matrix;

[0032] An infrared temperature matrix is ​​obtained in the ear base area, and the highest temperature in the infrared temperature matrix is ​​collected as the temperature at the ear base of the pig, and the temperature is used as the body temperature of the pig.

[0033] Furthermore, the affine transformation matrix includes:

[0034] Based on the RGB images and infrared thermal images of the pigs collected by the data acquisition and individual recognition modules, a set of matching feature points is selected;

[0035] According to the coordinate information of the matching feature points, the affine transformation matrix is ​​calculated.

[0036] Furthermore, the multimodal pig behavior recognition model based on the fusion of pig skeleton information and RGB video stream specifically includes: a skeleton image processing branch and an RGB processing branch;

[0037] The skeleton graph processing branch uses the ST-GCN spatiotemporal graph convolutional network to extract the spatiotemporal features of the skeleton graph sequence to obtain skeleton features;

[0038] The RGB processing branch uses an R(2+1)D network to extract the spatiotemporal features of the video frame sequence to obtain RGB features, where R represents the residual and (2+1)D represents the decomposition of the 3D convolution into independent 2D spatial convolution and 1D temporal convolution.

[0039] Furthermore, the formula for adjusting the feeding frequency by the feeding control module is:

[0040]

[0041] in, is the control coefficient, As the basic feeding amount, is the currently measured health index, As the baseline health index.

[0042] Beneficial effects of the embodiments of the present application:

[0043] The present application provides an intelligent pig farming management and control system based on multimodal data monitoring. It combines RFID ear tags, inspection robots, environmental sensors, feeders and other equipment to collect and inspect data for the entire pig house. It aligns data for each device through the NTP protocol, establishes a data association framework with the pig ID as the unique identifier, combines edge computing devices and cloud computing platforms for multimodal feature fusion, and analyzes the growth status, behavioral patterns, body temperature changes, feed intake and other information of individual pigs in the pig house based on multimodal data. This ensures that even if the ear tag is damaged, the individual pig can still be accurately located, providing a guarantee for precise feeding regulation and health management. In order to solve the problem that a single data source is easily interfered with, a weight estimation model, a body temperature detection model and a pig behavior recognition model that combine different modal data are constructed to improve the accuracy of monitoring. At the same time, by analyzing the physiological information, behavioral information, feeding information and pig house environmental information of the pig, a pig health index and a pig house environmental index are constructed respectively, which realizes early warning of pig diseases and improves the abnormal response speed. In addition, the fan speed and feeding frequency are dynamically adjusted based on abnormal responses, reducing the energy consumption of environmental control. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0045] Figure 1 A schematic diagram of an intelligent pig farming management and control system based on multimodal data monitoring provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0047] Pig health monitoring is crucial for pig farming management. Currently, actual production still relies on manual inspections to monitor pig health. Farm staff make assessments based on observing pig behavior (eating, sleeping, exercise, lameness, estrus, etc.) and collecting physiological indicators (weight, temperature, and appearance). However, this method is time-consuming, costly, inefficient, and highly subjective. The lack of standardized standards also hinders industrialized implementation. More importantly, manual inspections typically only reveal late-stage symptoms, while sub-health or early-stage diseases (such as mild temperature abnormalities and behavioral changes) can easily be overlooked, leading to delayed prevention and control. Furthermore, traditional methods lack the ability to achieve high-frequency, continuous monitoring, making it difficult to meet the real-time and precision requirements of modern intensive farming.

[0048] In recent years, researchers have explored using various wearable sensors and cameras to collect pig data (color, infrared, 3D, and time-series motion data) to extract physiological and behavioral features of abnormal pig health, and have made some progress. For example, Cang Yan et al. used the ReSpeaker Core v2.0 development board and MobileNetV2 network technology to classify pig sounds, achieving a recognition rate of 97.3%. Researchers used the Yolov4 network to identify pig drinking behavior in images from different perspectives. To address the difficulty of identifying high-risk movements in lactating sows, researchers proposed a deep video recognition algorithm based on a hidden Markov model. Researchers also proposed an improved Ostu algorithm for detecting the pig's ear base region based on thermal infrared images. Researchers also leveraged facial landmark detection technology to automatically identify key areas of pig faces in infrared images. However, these studies only capture specific physiological or behavioral characteristics of pigs and fail to consider their overall health. Furthermore, the fragility of fixed equipment, including damage and fall, makes it less costly than manual labor, hindering widespread application.

[0049] Based on this, the embodiment of the present application provides a pig farming intelligent management and control system based on multimodal data monitoring, which can be seen in Figure 1 , Figure 1 The figure shows a schematic diagram of an intelligent pig farming management and control system based on multimodal data monitoring provided by an embodiment of the present application, including: a data acquisition and individual identification module, a feature extraction module, a key farming indicator analysis module, an abnormality warning module, and a management and control module;

[0050] The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeders. It determines the individual identity information of pigs based on the data collected by the RFID ear tags and inspection robots, and associates the multimodal data of pigs in the pig house collected in real time based on the determined individual identity information of pigs. It uses the NTP protocol for data alignment and uploads the data to the edge computing device on the inspection robot via the MQTT protocol for preprocessing.

[0051] The feature extraction module includes a multimodal deep learning fusion model, which uses the multimodal deep learning model to extract the physiological and behavioral characteristics of pigs from the preprocessed multimodal data;

[0052] The key breeding index analysis module constructs a pig health index based on the physiological and behavioral characteristics of the pigs extracted by the feature extraction module and the data collected by the intelligent feeder; and constructs an environmental quality index based on the environmental parameters collected by the environmental sensor;

[0053] The abnormality warning module accumulates the deviations between the pig health index and the environmental quality index and the target values ​​based on the CUSUM control chart and responds to abnormalities;

[0054] The management and control model includes an intelligent environmental control system and a feeding control module. A Markov decision process is constructed with minimizing energy consumption and maximizing health index as multi-objective optimization functions, and the DDPG algorithm is used to adjust the fan speed and feeding frequency.

[0055] Furthermore, the data collection and individual identification module specifically includes:

[0056] The RFID ear tag integrates a temperature sensor and an acceleration sensor to identify the pig and collect the pig's body temperature, location and behavior data; the inspection robot is equipped with color, infrared, and depth cameras and an ear tag data acquisition base station to collect RGB images, infrared thermal maps and depth point cloud data of the pig, and combines the RFID ear tag to achieve individual identification and data association; the environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor to monitor the environmental parameters in the pig house in real time; the smart feeder integrates a weight sensor and an RFID reader to record the duration and amount of each feeding of the pig, and obtains weight information of the pig through the weight sensor when the pig is feeding.

[0057] In one embodiment of the present application, the management and control system provided herein uses piggery pens as units. It utilizes monitoring equipment installed within the pens to determine the identity of the pigs within the pens and collect multimodal data from the pigs within the unit in real time. Each breeding unit is equipped with a patrol robot, RFID ear tags, smart feeders, and environmental sensors. The RFID ear tags, in addition to identifying the pigs, also collect their body temperature, location, and behavioral data. The patrol robot is equipped with color, infrared, and depth cameras, as well as an ear tag data acquisition base station, to collect visual images, infrared thermal images, and depth point cloud data of the pigs. Combined with the RFID ear tags, the robot achieves individual identification and data association. The smart feeder integrates a weight sensor and RFID reader to record the duration and amount of each feeding session. The weight sensor also captures the pigs' weight information during feeding, creating a personalized feeding profile. Environmental sensors, including temperature, humidity, and hazardous gas sensors, monitor environmental parameters within the piggery in real time, providing basic data for environmental quality assessment. This equipment collects real-time video image data, feeding data, and environmental data from live pigs. It synchronizes this data using the NTP protocol's timestamp alignment method and uploads it to the inspection robot's onboard edge computing device via the MQTT protocol. The data is pre-processed by the onboard edge computing device and then transmitted to the cloud for further analysis and decision support, creating a complete intelligent closed-loop management and control system.

[0058] This application designs a precise monitoring model for individual pigs that combines RFID signals and video tracking algorithms to monitor individual pigs, generate a unique number for each pig, and ensure that the individual pig can still be accurately located even if the ear tag is damaged, laying the foundation for precise feeding regulation and health management.

[0059] Furthermore, the determination of individual pig identity information based on data collected by the RFID ear tags and the inspection robot specifically includes:

[0060] The Deepsort multi-target tracking algorithm is used to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence for each pig;

[0061] The ST-LSTM network is used to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot to form a 6-dimensional vector composed of the acceleration data and the visual coordinates to infer the movement trajectory of the pig;

[0062] The Hungarian algorithm is used to match the ear tag ID with the pig's movement trajectory to determine the individual identity information of the pig.

[0063] In one embodiment of the present application, before collecting pig data, the inspection robot performs an inspection and uses the Deepsort multi-target tracking algorithm to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence for each pig: ; Simultaneously analyze ear tag acceleration sensor data , using the ST-LSTM network to process acceleration data and visual coordinate sequences, forming a 6-dimensional vector composed of acceleration data and visual coordinates to infer the movement trajectory of pigs ; The Hungarian algorithm is used to match the ear tag ID with the pig's movement trajectory to determine the individual pig's identity information, where Representative The moment of the sampling point, and Representative The spatial coordinates of the pigs at the sampling points, , is the total number of sampling points, 、 and Indicates Acceleration in different directions at any moment.

[0064] Among them, in order to adapt to the computing power of edge computing devices, the original ST-LSTM is split into a dual-branch structure with time and space separation. The spatial branch uses Conv1D to process acceleration data. The convolution kernel size of Conv1D is a one-dimensional convolution of 3. The temporal branch uses a unidirectional LSTM to process the visual coordinate sequence, and the dual-branch output is fused through the gated attention mechanism.

[0065] In one embodiment of the present application, after locating an individual pig, its corresponding data is preprocessed through an edge computing node to form a three-dimensional data cube containing visual features, infrared temperature, and depth point cloud, and then the physiological and behavioral information of the pig is extracted through a deep learning model.

[0066] Furthermore, the multimodal deep learning model is used to extract the physiological and behavioral characteristics of pigs from the pre-processed multimodal data, specifically including:

[0067] The multimodal pig weight estimation model based on the fusion of RGB images and depth images extracts pig weight information, specifically including:

[0068] Both the RGB branch and the depth branch use Swin Transformer as the feature extraction network to extract depth features and RGB features to obtain depth feature maps and RGB feature maps;

[0069] Use the cross-modal attention module to perform modal feature interaction between the depth feature map and the RGB feature map;

[0070] Perform 1×1 convolution operation on the modal features after fusion interaction to generate a fusion feature map;

[0071] Based on the fused feature map, the weight information of the pig is obtained.

[0072] In one embodiment of the present application, the extraction of pig weight information relies on the pig weight estimation model designed by the present invention. In order to achieve contactless pig weight estimation, it is necessary to combine the acquired pig RGB-D image data and the real pig weight data for model training. In order to achieve accurate pig weight estimation, the present invention designs a multimodal pig weight estimation model that fuses RGB images and depth images. The model includes two branches, RGB and depth. Both RGB and depth branches use Swin Transformer as the feature extraction network. The depth feature map and the RGB feature map realize modal feature interaction through the cross-modal attention module, and generate a fused feature map through a 1×1 convolution operation. By utilizing the information in the RGB and depth data, the overall performance of the model is greatly improved. Finally, the feature map after feature fusion is used as the output of the model to estimate the weight of the pig. .

[0073] It can be seen that a weight estimation model combining RGB and depth images designed for weight information is used to monitor the daily weight changes of pigs. By fusing data from the two modalities, more accurate pig weight estimation is achieved.

[0074] The pig body temperature detection model based on the fusion of RGB images and infrared images measures the pig's body temperature information, specifically including:

[0075] Use key point detection algorithm to detect the location of the pig ear root area on the RGB image;

[0076] The position of the pig's ear root area on the infrared image is obtained using the affine transformation matrix;

[0077] An infrared temperature matrix is ​​obtained in the ear base area, and the highest temperature in the infrared temperature matrix is ​​collected as the temperature at the ear base of the pig, and the temperature is used as the body temperature of the pig.

[0078] Wherein, the affine transformation matrix includes:

[0079] Based on the RGB images and infrared thermal images of the pigs collected by the data acquisition and individual recognition modules, a set of matching feature points is selected;

[0080] According to the coordinate information of the matching feature points, the affine transformation matrix is ​​calculated.

[0081] In one embodiment of the present application, the present application extracts the temperature at the base of the pig's ear as a representation of its body temperature based on the color and infrared images of the pig. In order to achieve the extraction of the pig's body temperature, it is necessary to collect the color and infrared image data of the pig in the early stage for training the body temperature extraction model. The data is used to train the key point detection algorithm to detect the position of the pig's ear from the image. Due to the different resolutions of color images and infrared images, the body temperature extraction model of the present invention first selects a set of matching feature points on the infrared image and the RGB image. and ,in The infrared image pixels, is the first pixels, , The affine transformation matrix M is calculated by matching the coordinate information of the feature points. The coordinates on the RGB image can be converted to the coordinates on the infrared image through the affine transformation matrix. The present invention uses the HigherHRNet key point detection algorithm to detect the position of the pig ear root area on the RGB image. , and then use the calculated affine transformation matrix to transform the position of the pig ear root area on the infrared image Then, a 3*3 infrared temperature matrix is ​​obtained in the ear root area. , the highest temperature in this area is taken as the temperature at the pig's ear root and use it as pig body temperature .

[0082] A pig temperature detection model designed for body temperature information combines RGB images and infrared images. It can accurately measure the body temperature of the pig's ear, eye, and buttocks by improving the positioning accuracy of the target area and the accuracy of temperature reading.

[0083] The multimodal pig behavior recognition model based on the fusion of pig skeleton information and RGB video stream extracts the time and frequency of various pig behaviors, specifically including: a skeleton image processing branch and an RGB processing branch;

[0084] The skeleton graph processing branch uses the ST-GCN spatiotemporal graph convolutional network to extract the spatiotemporal features of the skeleton graph sequence to obtain skeleton features; the RGB processing branch uses the R(2+1)D network to extract the spatiotemporal features of the video frame sequence to obtain RGB features.

[0085] In one embodiment of the present application, the R(2+1)D network is a convolutional network structure, where R stands for Residual and (2+1)D represents the decomposition of a 3D convolution into independent 2D spatial convolution and 1D temporal convolution.

[0086] In order to realize the recognition of pig behavior, it is necessary to collect the color video stream data of pigs in the early stage and mark the behaviors that appear in it. This application designs a multimodal pig behavior recognition model that integrates pig skeleton information and RGB video stream information. The model first uses the RTMDet target detection algorithm and the RTMPose key point detection algorithm to detect the pig individuals and their skeleton joints in the video frame, and then constructs the pig skeleton image sequence. , then the skeleton graph sequence And the corresponding RGB video frame sequence Input into the pig behavior recognition model, where Indicates the video frames, Indicates the A pig skeleton diagram, , The pig behavior recognition model includes a skeleton image processing branch and an RGB processing branch. The skeleton image processing branch uses the ST-GCN spatiotemporal graph convolutional network to extract the spatiotemporal features of the skeleton image sequence, and the RGB processing branch uses the R(2+1)D network to extract the spatiotemporal features of the video frame sequence. After processing, the skeleton features are obtained. and RGB features , , , , are the dimensions of RGB features, is the number of channels of the feature, is the number of frame sequences, and The length and width of the feature. This application designs a feature fusion module to fuse skeleton features and RGB features. The input skeleton features , firstly converted into a length of vector, is the number of channels of input skeleton features, is the number of frame sequences for input skeleton features, is the number of pig skeleton points. Vector Adjust to the dimension size through channel dimension replication (Broadcasting) Tensor , then the tensor With RGB features Follow the sequence from the first frame to the The frame is concatenated, and the concatenated tensor is input into a 1×1×1 convolution to adjust its channel size to obtain the RGB feature of the fused skeleton feature. , and the fusion features Input into the classification layer to obtain the behavior classification results, and finally generate a behavior statistics report.

[0087] A pig behavior recognition model designed for behavioral information combines time-series pig skeleton information and RGB video stream. The accuracy of pig behavior recognition is improved by guiding RGB feature learning through skeleton features. The behavior recognition model counts the time and frequency of various pig behaviors.

[0088] In one embodiment of the present application, after obtaining the physiological and behavioral information of pigs based on the described collection process, the environmental parameters collected by the environmental sensor are combined to construct the pig house environmental quality index. EQI , pig health index HI Specifically, for the Environmental Quality Index EQI , taking temperature T, humidity RH, NH3 concentration, H2S concentration and CO2 concentration as independent variables, and the environmental quality score marked by experts as the dependent variable, the random forest model is trained to calculate the mean square error percentage and node purity increment of each indicator, and the normalized average of the two is used as the weight of each indicator. For pig health index HI First, health indicators are constructed based on the collected information on pig physiology, behavior and feeding. Health indicators include pig body temperature deviation , Indicates the detected pig body temperature, Indicates normal pig body temperature and 24h exercise volume , 、 、 The acceleration value collected by the ear tag and the frequency of abnormal behavior , feed conversion rate , and Represents daily weight gain and daily feed intake, feeding frequency The above metrics are statistically analyzed over a 24-hour window. The mean and standard deviation of each feature are calculated using the Z-score method. Outliers are then removed using the 3σ principle, and missing values ​​are supplemented using the KNN algorithm. Feature importance is calculated using the Gini impurity measure, and the normalized feature weights are output.

[0089] Existing health assessments mostly rely on empirical thresholds, do not consider individual differences, have high false alarm rates, and lag in early warning information. This application constructs a pig health index by analyzing the pig's physiological information, behavioral information, feeding information, and pig house environment information. HI , Pig House Environment Index EQI By using the random forest algorithm to screen high-weight features and combining it with CUSUM dynamic window detection, we can achieve early warning of pig diseases and improve the response speed of environmental regulation.

[0090] In one embodiment of the present application, when generating HI and EQI After the index is generated, the system uses the CUSUM algorithm to detect cumulative deviations. The window size is dynamically adjusted according to the number of anomalies, and the calculation formula is: ,in To preset the window size, express The window size at the moment, is the number of alarms. EQI When the index exceeds the threshold continuously within a window size, the environmental control module starts the DDPG algorithm for dynamic adjustment. The system defines the state space as the time series change rate of temperature, humidity, and gas concentration, and the action space as the fan speed gear (0-100%). The reward function is designed as R =0.7*( EQI Improvement rate) - 0.3*(energy consumption increment / baseline value), after the model is trained with historical environmental data offline, the DDPG network (using a 3-layer 512-node MLP) can achieve minute-level response control. HI When the index is continuously abnormal within the window period, the system triggers the early warning processing mechanism. For abnormal body temperature and behavior, the early warning module automatically dispatches the inspection robot to conduct high-frequency monitoring of the pen, simultaneously collects infrared images to verify ear temperature data, and generates an individualized health report including body temperature curves and movement trajectory heat maps. For abnormal weight and feed intake, the feeding module adjusts the strategy and adopts a dynamic feeding formula. ,in is the control coefficient (the control parameter is obtained by training based on historical feeding data), As the basic feeding amount, is the currently measured health index, As the baseline health index.

[0091] Existing feeding models rely solely on age to set fixed feed amounts, ignoring individual differences in feeding behavior, leading to serious feed waste. This application designs a feeding control model that combines pig age, weight, feed intake, and feeding frequency. The model establishes a dynamic feeding adjustment formula that, through linkage with the feeder, dynamically adjusts the feed amount for each pig, reducing feed waste while ensuring daily weight gain.

[0092] In one embodiment of the present application, the intelligent environmental control system of the present application is based on the fan control strategy of deep reinforcement learning (DDPG algorithm), through Q The value function dynamically optimizes the start and stop thresholds, improves response speed, and reduces energy consumption for environmental control.

[0093] The present application provides an intelligent pig farming management and control system based on multimodal data monitoring. It combines RFID ear tags, inspection robots, environmental sensors, feeders and other equipment to collect and inspect data for the entire pig house. It aligns data for each device through the NTP protocol, establishes a data association framework with the pig ID as the unique identifier, combines edge computing devices and cloud computing platforms for multimodal feature fusion, and analyzes the growth status, behavioral patterns, body temperature changes, feed intake and other information of individual pigs in the pig house based on multimodal data. This ensures that even if the ear tag is damaged, the individual pig can still be accurately located, providing a guarantee for precise feeding regulation and health management. In order to solve the problem that a single data source is easily interfered with, a weight estimation model, a body temperature detection model and a pig behavior recognition model that combine different modal data are constructed to improve the accuracy of monitoring. At the same time, by analyzing the physiological information, behavioral information, feeding information and pig house environmental information of the pig, a pig health index and a pig house environmental index are constructed respectively, which realizes early warning of pig diseases and improves the abnormal response speed. In addition, the fan speed and feeding frequency are dynamically adjusted based on abnormal responses, reducing the energy consumption of environmental control.

[0094] It should be noted that those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of this application, and it should be understood that the scope of protection of this application is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in this application without departing from the essence of this application, and such variations and combinations are still within the scope of protection of this application.

Claims

1. An intelligent pig farming management and control system based on multimodal data monitoring, characterized in that: include: Data collection and individual identification module, feature extraction module, key breeding indicator analysis module, abnormal warning module and management and control module; The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeders. It determines the individual identity information of pigs based on the data collected by the RFID ear tags and inspection robots, and associates the multimodal data of pigs in the pig house collected in real time based on the determined individual identity information of pigs. It uses the NTP protocol for data alignment and uploads the data to the edge computing device on the inspection robot through the MQTT protocol for preprocessing. The feature extraction module includes a multimodal deep learning fusion model, which uses the multimodal deep learning model to extract the physiological and behavioral characteristics of pigs from the preprocessed multimodal data; The key breeding index analysis module constructs a pig health index based on the physiological and behavioral characteristics of the pigs extracted by the feature extraction module and the data collected by the intelligent feeder; and constructs an environmental quality index based on the environmental parameters collected by the environmental sensor; The abnormality warning module accumulates the deviations between the pig health index and the environmental quality index and the target values ​​based on the CUSUM control chart and responds to abnormalities; The management and control module includes an intelligent environmental control system and a feeding control module. It constructs a Markov decision process with minimizing energy consumption and maximizing health index as the multi-objective optimization function, and uses the DDPG algorithm to adjust the fan speed and feeding frequency. The method of extracting physiological and behavioral characteristics of pigs from pre-processed multimodal data using a multimodal deep learning model specifically includes: A multimodal pig weight estimation model based on the fusion of RGB images and depth images extracts pig weight information; The pig body temperature detection model based on the fusion of RGB images and infrared images measures the pig's body temperature information; A multimodal pig behavior recognition model based on the fusion of pig skeleton information and RGB video stream is used to extract the time and frequency of various pig behaviors. The multimodal pig behavior recognition model based on the fusion of pig skeleton information and RGB video stream specifically includes: a skeleton image processing branch and an RGB processing branch; The skeleton graph processing branch uses the ST-GCN spatiotemporal graph convolutional network to extract the spatiotemporal features of the skeleton graph sequence to obtain skeleton features; The RGB processing branch uses the R(2+1)D network to extract the spatiotemporal features of the video frame sequence to obtain RGB features, where R represents the residual and (2+1)D represents the decomposition of the 3D convolution into independent 2D spatial convolution and 1D temporal convolution; The formula for adjusting the feeding frequency by the feeding control module is: in, is the control coefficient, As the basic feeding amount, is the currently measured health index, As the baseline health index.

2. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 1 is characterized in that: The data collection and individual identification module specifically includes: The RFID ear tag integrates a temperature sensor and an acceleration sensor to identify the pig and collect the pig's body temperature, location and behavior data; the inspection robot is equipped with color, infrared, and depth cameras and an ear tag data acquisition base station to collect RGB images, infrared thermal maps and depth point cloud data of the pig, and combines the RFID ear tag to achieve individual identification and data association; the environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor to monitor the environmental parameters in the pig house in real time; the smart feeder integrates a weight sensor and an RFID reader to record the duration and amount of each feeding of the pig, and obtains weight information of the pig through the weight sensor when the pig is feeding.

3. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 2 is characterized in that: The method of determining individual pig identity information based on data collected by RFID ear tags and inspection robots specifically includes: The Deepsort multi-target tracking algorithm is used to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence for each pig; The ST-LSTM network is used to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot to form a 6-dimensional vector composed of the acceleration data and the visual coordinates to infer the movement trajectory of the pig; The Hungarian algorithm is used to match the ear tag ID with the pig's movement trajectory to determine the individual identity information of the pig.

4. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 3 is characterized in that: The ST-LSTM network adopts a dual-branch structure with time-space separation. The spatial branch uses Conv1D to process acceleration data, and the temporal branch uses a unidirectional LSTM to process the visual coordinate sequence. The dual-branch outputs are fused through a gated attention mechanism.

5. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 1 is characterized in that: The multimodal pig weight estimation model that integrates RGB images and depth images includes: Both the RGB branch and the depth branch use Swin Transformer as the feature extraction network to extract depth features and RGB features to obtain depth feature maps and RGB feature maps; Use the cross-modal attention module to perform modal feature interaction between the depth feature map and the RGB feature map; Perform 1×1 convolution operation on the modal features after fusion interaction to generate a fusion feature map; Based on the fused feature map, the weight information of the pig is obtained.

6. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 1 is characterized in that: The pig body temperature detection model based on the fusion of RGB images and infrared images measures the body temperature information of the pig, specifically including: Use key point detection algorithm to detect the location of the pig ear root area on the RGB image; The position of the pig's ear root area on the infrared image is obtained using the affine transformation matrix; An infrared temperature matrix is ​​obtained in the ear base area, and the highest temperature in the infrared temperature matrix is ​​collected as the temperature at the ear base of the pig, and the temperature is used as the body temperature of the pig.

7. The intelligent pig farming management and control system based on multimodal data monitoring according to claim 6 is characterized in that: The affine transformation matrix includes: Based on the RGB images and infrared thermal images of the pigs collected by the data acquisition and individual recognition modules, a set of matching feature points is selected; According to the coordinate information of the matching feature points, the affine transformation matrix is ​​calculated.

Images