Pig breeding intelligent management and control system based on multi-modal data monitoring

Abstract

The invention discloses a pig breeding intelligent management and control system based on multi-modal data monitoring, and belongs to the technical field of pig breeding management, and the system comprises a data collection and individual recognition module which comprises an RFID ear tag, an inspection robot, an environment sensor and an intelligent feeder, and is used for determining the identity information of a pig individual based on the data collected by the RFID ear tag and the inspection robot, associating the collected multi-modal data of the live pigs in the pig house, and uploading the multi-modal data to edge computing equipment through an MQTT protocol for preprocessing; the feature extraction module extracts physiological and behavior features of live pigs by using a multi-modal deep learning fusion model; the key breeding index analysis module constructs a live pig health index and an environment quality index; the abnormity early warning module adopts a CUSUM control chart to accumulate index deviation and respond to abnormity; the management regulation and control model constructs a Markov decision process by minimizing energy consumption and maximizing health indexes, intelligently adjusts the rotating speed of a fan and a feeding strategy, and realizes precise environmental control and feeding optimization.

Description

Technical Field

[0001] This application relates to the technical field of pig farming management, and particularly to an intelligent control system for pig farming based on multi-modal data monitoring. Background Art

[0002] The pig farming industry is an important part of global agricultural production, providing an important guarantee for meeting the growing demand for meat. At present, pig farming is gradually developing towards intensification, large-scale, automation, and industrialization, and pig health monitoring is of great significance to pig farming management. At present, in actual production, the health status of pigs is still monitored by manual inspections. Traditional manual inspection methods have significant defects: relying on manual experience, low efficiency, strong subjectivity, and it is difficult to achieve high-frequency continuous monitoring, resulting in the frequent neglect of early diseases or sub-healthy states, and prominent problems of lagging prevention and control. For example, manual observation cannot accurately capture key indicators such as abnormal body temperature and slight changes in behavior patterns, which are often early signals of infectious diseases such as African swine fever and porcine reproductive and respiratory syndrome. In addition, with the transformation of China's pig farming towards intensification and industrialization, the breeding density has increased, and the manual management cost and difficulty have increased sharply. There is an urgent need for intelligent solutions to replace the traditional model.

[0003] In recent years, researchers have tried to improve the monitoring efficiency through technical means. For example, devices such as wearable sensors and cameras are used to collect color images, infrared thermal images, motion data, etc. of pigs, and deep learning algorithms are combined for feature extraction. However, these technologies are mostly limited to the analysis of single-dimensional data such as sound, behavior, or body temperature, lacking a comprehensive assessment of the overall health status of pigs, and the devices are prone to damage and costly, making it difficult to promote on a large scale.

[0004] Although existing intelligent breeding systems have made breakthroughs in some local links, such as Shuanghui realizing automatic grading of white-striped pigs through an AI image grading system, and Muyuan adopting building-style pig farming combined with intelligent environmental control equipment to improve biosecurity, most of these systems focus on environmental control or production process optimization, and do not deeply integrate multi-source data such as physiology, behavior, and environment to achieve real-time dynamic monitoring of health status. In addition, domestic intelligent equipment relies on imports and lacks independent R & D capabilities, resulting in high technology application costs and poor adaptability. For example, although intelligent inspection robots can accurately collect environmental parameters and pig body heat data, their deployment requires cooperation with the integration of the Internet of Things platform and algorithm models, and most current farms still lack such systematic solutions. Summary of the Invention

[0005] In view of the above deficiencies in the prior art, an intelligent control system for pig farming based on multi-modal data monitoring provided by this application solves the problems of incomplete and inaccurate data collection, independent environmental regulation and health monitoring, the health assessment method relying on empirical threshold judgment, the intelligent early warning model and decision-making system being insufficiently sound, and the disposal of stress events relying on manual intervention resulting in poor timeliness.

[0006] To achieve the above invention purpose, the technical solution adopted by this application is as follows: An intelligent control system for pig farming based on multi-modal data monitoring provided by this application includes: a data collection and individual identification module, a feature extraction module, a key breeding index analysis module, an abnormal early warning module, and a management and regulation module; The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeding devices. Based on the data collected by the RFID ear tags and inspection robots, the individual identity information of pigs is determined. Based on the determined individual identity information of pigs, the multi-modal data of pigs in the pigsty collected in real time is associated, and data alignment processing is performed using the NTP protocol, and then uploaded to the edge computing device carried on the inspection robot through the MQTT protocol for preprocessing; The feature extraction module includes a multi-modal deep learning fusion model, which uses the multi-modal deep learning model to extract the physiological and behavioral characteristics of pigs from the preprocessed multi-modal data; The key breeding index analysis module constructs a pig health index based on the physiological and behavioral characteristics of pigs extracted by the feature extraction module and combines the data collected by the intelligent feeding device; constructs an environmental quality index based on the environmental parameters collected by the environmental sensors; The abnormal early warning module accumulates the deviation between the pig health index and the environmental quality index and the target value based on the CUSUM control chart and responds to abnormalities; The management and regulation model includes an intelligent environmental control system and a feeding regulation module. A Markov decision process is constructed with minimizing energy consumption and maximizing the health index as the multi-objective optimization function, and the DDPG algorithm is used to adjust the fan speed and feeding frequency.

[0007] Furthermore, the data collection and individual identification module specifically includes: The RFID ear tag integrates a temperature sensor and an acceleration sensor, which is used to identify the identity of live pigs and collect the body temperature, location and behavior data of live pigs; the inspection robot is equipped with color, infrared, depth cameras and an ear tag data collection base station, which is used to collect the RGB images, infrared thermal maps and depth point cloud data of live pigs, and realizes individual identification and data association in combination with the RFID ear tag; the environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor, which real-time monitors the environmental parameters in the pigsty; the intelligent feeder integrates a weight sensor and an RFID reader, records the feeding duration and feed intake of each live pig each time, and obtains the body weight information when the live pig is feeding through the weight sensor.

[0008] Further, determining the individual identity information of live pigs based on the data collected by the RFID ear tag and the inspection robot specifically includes: Using the Deepsort multi-object tracking algorithm to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence of each live pig; Using the ST-LSTM network to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot, forming a 6D vector composed of acceleration data and visual coordinates, and inferring the movement trajectory of the live pig; Using the Hungarian algorithm to match the ear tag ID with the movement trajectory of the live pig to determine the individual identity information of the live pig.

[0009] Further, the ST-LSTM network adopts a spatio-temporal separation double-branch structure. The spatial branch uses Conv1D to process the acceleration data, and the temporal branch uses a unidirectional LSTM to process the visual coordinate sequence, and fuses the outputs of the double branches through a gated attention mechanism.

[0010] Further, using the multi-modal deep learning model to extract the physiological and behavioral characteristics of live pigs from the preprocessed multi-modal data specifically includes: Extracting the body weight information of live pigs based on a multi-modal live pig body weight estimation model that fuses RGB images and depth images; Measuring the body temperature information of live pigs based on a live pig body temperature detection model that fuses RGB images and infrared images; Based on a multi-modal live pig behavior recognition model that fuses live pig skeleton information and RGB video stream, extracting the time and frequency of various behaviors of live pigs.

[0011] Further, the multi-modal live pig body weight estimation model that fuses RGB images and depth images includes: Both the RGB branch and the depth branch use the Swin Transformer as the feature extraction network to extract depth features and RGB features, and obtain depth feature maps and RGB feature maps; Perform modal feature interaction between the depth feature map and the RGB feature map using a cross-modal attention module; Perform a 1×1 convolution operation on the modal features after fusion interaction to generate a fused feature map; Based on the fused feature map, obtain the weight information of the live pig.

[0012] Furthermore, the live pig body temperature detection model based on the fused RGB image and infrared image measures the body temperature information of the live pig, specifically including: Use the key point detection algorithm to detect the position of the live pig's ear root area on the RGB image; Use the affine transformation matrix to obtain the position of the live pig's ear root area on the infrared image; Obtain the infrared temperature matrix in the ear root area, collect the highest temperature in the infrared temperature matrix as the temperature at the live pig's ear root, and use the temperature as the body temperature of the live pig.

[0013] Furthermore, the affine transformation matrix includes: Based on the RGB image and infrared thermal map of the live pig collected by the data acquisition and individual recognition module, select a set of matching feature points; According to the coordinate information of the matching feature points, calculate the affine transformation matrix.

[0014] Furthermore, the multi-modal live pig behavior recognition model based on the fused live pig skeleton information and RGB video stream specifically includes: a skeleton graph processing branch and an RGB processing branch; The skeleton graph processing branch uses the ST-GCN spatio-temporal graph convolutional network to extract the spatio-temporal features of the skeleton graph sequence to obtain skeleton features; The RGB processing branch uses the R(2+1)D network to extract the spatio-temporal features of the video frame sequence to obtain RGB features, where R represents residual, and (2+1)D represents decomposing the 3D convolution into independent 2D spatial convolutions and 1D temporal convolutions.

[0015] Furthermore, the formula for the feeding regulation module to adjust the feeding frequency is:

[0016] where, is the regulation coefficient, is the basic feeding amount, is the currently measured health index, is the benchmark health index.

[0017] Beneficial effects of the embodiments of the present application: An intelligent control system for pig farming based on multi-modal data monitoring provided by this application combines devices such as RFID ear tags, inspection robots, environmental sensors, and feeders to collect data and conduct inspections on the entire pigsty. The data of each device is aligned through the NTP protocol, and a data association framework with the pig ID as the unique identifier is established. Combining edge computing devices and cloud computing platforms for multi-modal feature fusion, analyzing information such as the growth status, behavior patterns, body temperature changes, and feed intake of individual pigs in the pigsty based on multi-modal data, ensuring that individual pigs can still be accurately located in case the ear tags are damaged, and providing guarantee for precise feeding regulation and health management. Aiming at the problem that a single data source is easily interfered, a weight estimation model, a body temperature detection model, and a pig behavior recognition model that combine different modal data are constructed to improve the accuracy of monitoring. At the same time, by analyzing the physiological information, behavior information, feeding information, and pigsty environmental information of pigs, a pig health index and a pigsty environmental index are respectively constructed to realize the early warning of pig diseases and improve the abnormal response speed. In addition, based on the abnormal response, the fan speed and feeding frequency are dynamically adjusted, reducing the energy consumption of environmental control. Brief Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0019] Figure 1 It is a schematic diagram of an intelligent control system for pig farming based on multi-modal data monitoring provided by an embodiment of the present application. Detailed Embodiments

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0021] The health monitoring of live pigs is of great significance to pig farming management. At present, in actual production, the method of manual inspection is still relied on to monitor the health status of pigs. The staff in the pig farm judge by observing the behaviors of live pigs (such as eating, sleeping, moving, lameness, estrus, etc.) and collecting some physiological indicators of pigs (weight, body temperature, and body surface appearance characteristics). However, this method not only takes a long time, has high costs, low efficiency, and strong subjectivity, but also lacks a unified standard and cannot be industrially promoted. More critically, manual inspection can usually only detect late symptoms, while sub-healthy or early diseases (such as slight body temperature abnormalities and behavior changes) are extremely easy to be ignored, resulting in a lag in prevention and control. In addition, the traditional method cannot achieve high-frequency and continuous monitoring, and it is difficult to meet the requirements of modern intensive farming for real-time and accuracy.

[0022] In recent years, researchers have explored the use of various wearable sensors and cameras to collect pig data (such as color images, infrared images, 3D images, time-series motion data, etc.) for extracting abnormal physiological and behavioral characteristics of pigs, and certain progress has been made. For example, Cang Yan et al. used the ReSpeaker Core v2.0 development board and MobileNetV2 network technology to classify pig sounds, and the recognition rate could reach 97.3%; researchers achieved the recognition of pig drinking behaviors in images from different perspectives through the Yolov4 network; in view of the problem of difficult recognition of high-risk actions of lactating sows, researchers proposed a deep video recognition algorithm based on the hidden Markov model; researchers proposed an improved Ostu algorithm based on thermal infrared images to detect the ear root area of pigs; researchers automatically recognized key parts of the infrared images of pig faces by referring to the face key point detection technology. However, the above studies only obtained a certain physiological or behavioral characteristic of pigs, and did not consider their health status from the perspective of the overall condition of pigs. At the same time, the problem of easy damage and dropping of fixed devices makes there is no obvious advantage in cost compared with manual work, and it is difficult to be promoted and applied.

[0023] Based on this, the embodiments of this application provide a smart control system for pig farming based on multi-modal data monitoring. This system can be seen Figure 1 , Figure 1 Figure 10 shows a schematic diagram of a smart control system for pig farming based on multi-modal data monitoring provided by the embodiments of this application, including: a data collection and individual identification module, a feature extraction module, a key farming index analysis module, an abnormal warning module, and a management and control module; The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeders. Based on the data collected by the RFID ear tags and inspection robots, the individual identity information of pigs is determined, and the multi-modal data of pigs in the pigsty collected in real time is associated based on the determined individual identity information of pigs. The NTP protocol is used for data alignment processing, and the data is uploaded to the edge computing device carried on the inspection robot through the MQTT protocol for preprocessing; The feature extraction module includes a multi-modal deep learning fusion model, which extracts physiological and behavioral characteristics of live pigs from the preprocessed multi-modal data using the multi-modal deep learning model; The key breeding index analysis module constructs a live pig health index based on the physiological and behavioral characteristics of live pigs extracted by the feature extraction module and combines the data collected by the intelligent feeder; constructs an environmental quality index based on the environmental parameters collected by the environmental sensor; The anomaly warning module accumulates the deviation between the live pig health index and the environmental quality index and the target value based on the CUSUM control chart and responds to anomalies; The management and control model includes an intelligent environmental control system and a feeding control module. A Markov decision process is constructed with minimizing energy consumption and maximizing the health index as the multi-objective optimization function, and the DDPG algorithm is used to adjust the fan speed and feeding frequency.

[0024] Furthermore, the data collection and individual identification module specifically includes: The RFID ear tag integrates a temperature sensor and an acceleration sensor, which is used to identify the identity of live pigs and collect the body temperature, position and behavior data of live pigs; the patrol robot is equipped with color, infrared, depth cameras and an ear tag data collection base station, which is used to collect RGB images, infrared thermal maps and depth point cloud data of live pigs, and realizes individual identification and data association in combination with the RFID ear tag; the environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor, which real-time monitors the environmental parameters in the pigsty; the intelligent feeder integrates a weight sensor and an RFID reader, records the duration and food intake of each feeding of live pigs, and obtains the body weight information when live pigs are feeding through the weight sensor.

[0025] In one embodiment of the present application, the management and control system provided by the present application uses the breeding pens in the pig house as units, uses the monitoring equipment configured in the pens to determine the identity of the pigs in the pens, and collects the multimodal data of the pigs in the unit in real time. Each breeding unit is equipped with inspection robots, RFID ear tags, intelligent feeders, environmental sensors and other equipment, wherein the RFID ear tags are used to collect the body temperature, position and behavior data of the pigs in addition to identifying the identity of the pigs. The inspection robot is equipped with color, infrared, depth cameras and ear tag data acquisition base stations, which are used to collect visual images, infrared thermal images and depth point cloud data of the pigs, and combine the RFID ear tags to realize individual identification and data association. The intelligent feeder is integrated with a weight sensor and an RFID reader, which can record the duration and amount of each feeding of the pigs, and obtain their weight information when the pigs are feeding through the weight sensor, and build an individualized feeding file. The environmental sensor includes a temperature sensor, a humidity sensor and a harmful gas sensor, which monitors the environmental parameters in the pig house in real time and provides basic data for environmental quality assessment. The above equipment collects video image data, feeding data and environmental data of live pigs in real time, synchronizes the data using the timestamp alignment method of the NTP protocol, and uploads it to the edge computing device on the inspection robot through the MQTT protocol. The data is pre-processed by the edge computing device on the inspection robot and then transmitted to the cloud for further analysis and decision support, building a complete intelligent control closed loop.

[0026] This application designs a precise monitoring model for individual pigs that combines RFID signals and video tracking algorithms to monitor individual pigs, generates a unique number for each pig, and ensures that the individual pig can still be accurately located even if the ear tag is damaged, laying the foundation for precise feeding regulation and health management.

[0027] Furthermore, the determination of individual pig identity information based on the data collected by the RFID ear tag and the inspection robot specifically includes: The Deepsort multi-target tracking algorithm is used to process the RGB video stream collected by the inspection robot to generate the visual coordinate sequence of each pig; The ST-LSTM network is used to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot to form a 6-dimensional vector composed of the acceleration data and the visual coordinates to infer the movement trajectory of the pig; The Hungarian algorithm is used to match the ear tag ID with the pig's movement trajectory to determine the individual identity information of the pig.

[0028] In one embodiment of the present application, before collecting pig data, the inspection robot performs inspections, and uses the Deepsort multi-target tracking algorithm to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence for each pig: ; At the same time, parse the ear tag acceleration sensor data , use the ST-LSTM network to process the acceleration data and visual coordinate sequence, form a 6D vector composed of acceleration data and visual coordinates, and estimate the movement trajectory of the live pig ; Use the Hungarian algorithm to match the ear tag ID with the movement trajectory of the live pig to determine the individual identity information of the live pig. Among them, represents the time of the th sampling point, and represent the spatial coordinates of the live pig at the th sampling point, is the total number of sampling points, , and represent the accelerations in different directions at the time.

[0029] Among them, in order to adapt to the computing power of edge computing devices, the original ST-LSTM is split into a spatio-temporal separation dual-branch structure. The spatial branch uses Conv1D to process the acceleration data. The Conv1D is a one-dimensional convolution with a kernel size of 3. The temporal branch uses a unidirectional LSTM to process the visual coordinate sequence, and the outputs of the two branches are fused through a gated attention mechanism.

[0030] In an embodiment of the present application, after the individual live pig is located, the corresponding data is preprocessed through an edge computing node to form a three-dimensional data cube including visual features, infrared temperature, and depth point cloud, and then the physiological and behavioral information of the live pig is extracted through a deep learning model.

[0031] Furthermore, the use of the multi-modal deep learning model to extract the physiological and behavioral characteristics of the live pig from the preprocessed multi-modal data specifically includes: Based on the multi-modal live pig weight estimation model that fuses RGB images and depth images, extract the weight information of the live pig, specifically including: Both the RGB branch and the depth branch use the Swin Transformer as the feature extraction network to extract depth features and RGB features, and obtain the depth feature map and the RGB feature map; Use the cross-modal attention module to perform modal feature interaction on the depth feature map and the RGB feature map; Perform a 1×1 convolution operation on the fused and interacted modal features to generate a fused feature map; Based on the fused feature map, obtain the weight information of the live pig.

[0032] In one embodiment of the present application, the extraction of pig weight information relies on the pig weight estimation model designed by the present invention. In order to achieve contactless pig weight estimation, it is necessary to combine the acquired pig RGB-D image data and the real pig weight data for model training. In order to achieve accurate pig weight estimation, the present invention designs a multimodal pig weight estimation model that fuses RGB images and depth images. The model includes two branches, RGB and depth. Both RGB and depth branches use Swin Transformer as the feature extraction network. The depth feature map and the RGB feature map realize modal feature interaction through the cross-modal attention module, and generate a fused feature map through a 1×1 convolution operation. By utilizing the information in the RGB and depth data, the overall performance of the model is greatly improved. Finally, the feature map after feature fusion is used as the output of the model to estimate the weight of the pig. .

[0033] It can be seen that a weight estimation model combining RGB and depth images designed for weight information is used to monitor the daily weight changes of pigs. By fusing data from the two modalities, a more accurate pig weight estimation is achieved.

[0034] The pig body temperature detection model based on the fusion of RGB images and infrared images measures the body temperature information of pigs, including: The key point detection algorithm is used to detect the location of the pig ear root area on the RGB image; The position of the pig ear root area on the infrared image is obtained using the affine transformation matrix; An infrared temperature matrix is obtained in the ear base area, and the highest temperature in the infrared temperature matrix is collected as the temperature at the ear base of the pig, and the temperature is used as the body temperature of the pig.

[0035] Wherein, the affine transformation matrix includes: Based on the RGB images and infrared thermal images of pigs collected by the data collection and individual recognition modules, a set of matching feature points are selected; According to the coordinate information of the matching feature points, the affine transformation matrix is calculated.

[0036] In one embodiment of the present application, the present application extracts the temperature at the base of the pig's ear as a representation of its body temperature based on the color and infrared images of the pig. In order to achieve the extraction of the pig's body temperature, it is necessary to collect the color and infrared image data of the pig in the early stage for the training of the body temperature extraction model. The data is used to train the key point detection algorithm to detect the position of the pig's ear from the image. Due to the different resolutions of color images and infrared images, the body temperature extraction model of the present invention first selects a set of matching feature points on the infrared image and the RGB image. and ,in The infrared image a pixel point is the pixel point on the RGB image , is the total number of pixel points. The affine transformation matrix M is calculated by matching the coordinate information of the feature points. The coordinates on the RGB image can be transformed into the coordinates on the infrared image through the affine transformation matrix. In the present invention, the HigherHRNet key point detection algorithm is used to detect the position of the pig ear root area on the RGB image , and then the position of the pig ear root area on the infrared image is obtained by using the calculated affine transformation matrix . Subsequently, a 3×3 infrared temperature matrix is obtained in the ear root area , and the highest temperature in this area is used as the temperature at the pig ear root , and it is used as the body temperature of the pig .

[0037] A pig body temperature detection model combining RGB images and infrared images designed for body temperature information realizes the accurate measurement of the body temperature of the pig ear root, eye, and hip areas by improving the positioning accuracy of the target area and the accuracy of temperature reading

[0038] The multi-modal pig behavior recognition model based on the fusion of pig skeleton information and RGB video stream extracts the time and frequency of various pig behaviors, specifically including: a skeleton map processing branch and an RGB processing branch The skeleton map processing branch uses the ST-GCN spatio-temporal graph convolutional network to extract the spatio-temporal features of the skeleton map sequence to obtain skeleton features; the RGB processing branch uses the R(2 + 1)D network to extract the spatio-temporal features of the video frame sequence to obtain RGB features

[0039] In an embodiment of the present application, the R(2 + 1)D network is a convolutional network structure, where R represents Residual, and (2 + 1)D represents decomposing the 3D convolution into independent 2D spatial convolutions and 1D temporal convolutions

[0040] In order to realize the recognition of pig behaviors, it is necessary to collect the color video stream data of pigs in the early stage and label the behaviors that appear therein. The present application designs a multi-modal pig behavior recognition model that fuses pig skeleton information and RGB video stream information. The model first uses the RTMDet object detection algorithm and the RTMPose key point detection algorithm to detect pig individuals and their skeleton joint points in the video frames, thereby forming a pig skeleton map sequence , and then the skeleton map sequence and the corresponding RGB video frame sequence are input into the pig behavior recognition model, where represents the th video frame, represents the th pig skeleton diagram, , where is the total number of pig skeleton diagrams. The pig behavior recognition model includes a skeleton diagram processing branch and an RGB processing branch. The skeleton diagram processing branch uses the ST-GCN spatio-temporal graph convolutional network to extract the spatio-temporal features of the skeleton diagram sequence, and the RGB processing branch uses the R(2+1)D network to extract the spatio-temporal features of the video frame sequence. After processing, the skeleton features and RGB features are obtained. , , , are the dimensions of the RGB features respectively, where is the number of channels of the feature, is the number of frame sequences, and are the length and width of the feature. This application designs a feature fusion module to fuse the skeleton features and RGB features. The input skeleton feature is first transformed into a vector of length through a global average pooling operation (Global Average Pooling), where is the number of channels of the input skeleton feature, is the number of frame sequences of the input skeleton feature, is the number of pig skeleton points. The vector is adjusted to a tensor of dimension size through channel dimension replication and extension (Broadcasting), and then the tensor is concatenated with the RGB feature from the first frame to the th frame according to their sequences through a concatenate operation. The concatenated tensor is input into a 1×1×1 convolutional layer to adjust its channel size, and the RGB feature of the fused skeleton feature is obtained. The fused feature is input into the classification layer to obtain the behavior classification result, and finally a behavior statistical report is generated.

[0041] A pig behavior recognition model that combines temporal pig skeleton information and RGB video stream for behavior information, which improves the accuracy of pig behavior recognition by guiding RGB feature learning with skeleton features, and statistically counts the time and frequency of various pig behaviors through the behavior recognition model.

[0042] In one embodiment of the present application, after obtaining the physiological and behavioral information of live pigs based on the described acquisition process, the environmental quality index of the pigsty is constructed by combining the environmental parameters collected by the environmental sensors EQI , and the pig health index HI . Specifically, for the environmental quality index EQI , taking temperature T, humidity RH, NH3 concentration, H2S concentration, and CO2 concentration as independent variables, and the environmental quality score marked by experts as the dependent variable, training is carried out through a random forest model, calculating the percentage of mean squared error and the increment of node purity of each index, and taking the normalized average of the two as the weight of each index . For the pig health index HI , first construct health indicators according to the collected physiological, behavioral, and feeding information of live pigs. The health indicators include the deviation degree of pig body temperature , represents the detected body temperature of the live pig, represents the normal body temperature of the live pig, 24h exercise volume , , , are the acceleration values collected by the ear tag, the frequency of abnormal behavior , the feeding conversion rate , and represent the daily weight gain and daily feed intake respectively, and the feeding times . The above indicators use the Z-score method to calculate the mean and standard deviation of each feature with a 24-hour statistical window, and then use the 3σ principle to remove outliers in the features and use the KNN algorithm to supplement missing values. The importance of the features is calculated based on the Gini impurity, and finally the normalized feature weights are output

[0043] Existing health assessments mostly rely on empirical thresholds, do not consider individual differences, have a high false alarm rate and lagging warning information. In this application, by analyzing the physiological information, behavioral information, feeding information, and pigsty environmental information of live pigs, the pig health index HI and the pigsty environmental index EQI are constructed respectively. By screening high-weight features through the random forest algorithm and combining CUSUM dynamic window detection, early warning of pig diseases is realized, and the response speed of environmental regulation is improved

[0044] In one embodiment of the present application, after generating HI and EQI indexes, the system uses the CUSUM algorithm to detect the cumulative deviation. The window size is dynamically adjusted according to the number of anomalies, and the calculation formula is , where is the preset window size, represents The window size at the moment, is the number of alarms. EQI When the index exceeds the threshold continuously within a window size, the environmental control module starts the DDPG algorithm for dynamic adjustment. The system defines the state space as the time series change rate of temperature, humidity, and gas concentration, and the action space as the fan speed gear (0-100%). The reward function is designed as R =0.7*( EQI Improvement rate) - 0.3*(energy consumption increment / baseline value). After the model is trained offline with historical environmental data, the DDPG network (using a 3-layer 512-node MLP) can achieve minute-level response control. HI When the index is abnormal continuously within the window period, the system triggers the early warning processing mechanism. For abnormal body temperature and behavior, the early warning module automatically dispatches the inspection robot to conduct high-frequency monitoring of the pen, simultaneously collects infrared images to verify ear temperature data, and generates an individualized health report including body temperature curves and motion trajectory heat maps. For abnormal body weight and feed intake, the feeding module links to adjust the strategy and adopts a dynamic feeding formula ,in is the control coefficient (the control parameter is obtained by training based on historical feeding data), As the basic feeding amount, is the currently measured health index, The baseline health index.

[0045] The existing feeding model only relies on the age to set a fixed feed amount, ignoring the differences in individual feeding behavior, resulting in serious feed waste. This application designs a feeding control model that combines the age, weight, feed intake and feeding frequency of pigs. The model establishes a dynamic feeding adjustment formula, which dynamically adjusts the feed amount for each pig by linking with the feeder, reducing feed waste while ensuring daily weight gain.

[0046] In one embodiment of the present application, the intelligent environmental control system of the present application is based on a fan control strategy based on deep reinforcement learning (DDPG algorithm), through Q The value function dynamically optimizes the start and stop thresholds, improves response speed, and reduces energy consumption for environmental control.

[0047] An intelligent control system for pig farming based on multi-modal data monitoring provided by the present application combines devices such as RFID ear tags, inspection robots, environmental sensors, and feeders to collect data and conduct inspections on the entire pigsty. The data of each device is aligned through the NTP protocol, and a data association framework with the pig ID as the unique identifier is established. Multi-modal feature fusion is carried out in combination with edge computing devices and cloud computing platforms. Information such as the growth status, behavior patterns, body temperature changes, and feed intake of individual pigs in the pigsty is analyzed based on multi-modal data, ensuring that individual pigs can still be accurately located even when the ear tags are damaged, providing a guarantee for precise feeding regulation and health management. Aiming at the problem that a single data source is easily interfered, a weight estimation model, a body temperature detection model, and a pig behavior recognition model that combine different modal data are constructed, improving the accuracy of monitoring. At the same time, by analyzing the physiological information, behavior information, feeding information, and pigsty environmental information of pigs, a pig health index and a pigsty environmental index are respectively constructed, realizing early warning of pig diseases and improving the abnormal response speed. In addition, the fan speed and feeding frequency are dynamically adjusted based on abnormal responses, reducing the energy consumption of environmental regulation.

[0048] It should be noted that those of ordinary skill in the art will realize that the embodiments described herein are for helping readers understand the principles of the present application, and it should be understood that the protection scope of the present application is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present application based on the technical revelations disclosed in the present application, and these deformations and combinations are still within the protection scope of the present application.

Claims

1. An intelligent control system for pig farming based on multi-modal data monitoring, characterized in that, It includes: A data collection and individual identification module, a feature extraction module, a key breeding index analysis module, an anomaly warning module, and a management and control module; The data collection and individual identification module includes RFID ear tags, inspection robots, environmental sensors, and intelligent feeders. Based on the data collected by the RFID ear tags and inspection robots, the individual identity information of live pigs is determined. Based on the determined individual identity information of live pigs, the multi-modal data of live pigs in the pigsty collected in real time is associated, and data alignment processing is performed using the NTP protocol, and it is uploaded to the edge computing device carried on the inspection robot through the MQTT protocol for preprocessing; The feature extraction module includes a multi-modal deep learning fusion model, and uses the multi-modal deep learning model to extract the physiological and behavioral characteristics of live pigs from the preprocessed multi-modal data; The key breeding index analysis module constructs a live pig health index based on the physiological and behavioral characteristics of live pigs extracted by the feature extraction module and combines the data collected by the intelligent feeder; constructs an environmental quality index based on the environmental parameters collected by the environmental sensors; The anomaly warning module accumulates the deviation between the live pig health index and the environmental quality index and the target value based on the CUSUM control chart and responds to anomalies; The management and control model includes an intelligent environmental control system and a feeding control module. A Markov decision process is constructed with the minimum energy consumption and the maximum health index as the multi-objective optimization function, and the DDPG algorithm is used to adjust the fan speed and feeding frequency.

2. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 1, characterized in that, The data collection and individual identification module specifically includes: The RFID ear tag integrates a temperature sensor and an acceleration sensor, which is used to identify the identity of live pigs and collect the body temperature, position, and behavior data of live pigs; the inspection robot is equipped with a color, infrared, depth camera, and an ear tag data collection base station, which is used to collect the RGB images, infrared thermal maps, and depth point cloud data of live pigs, and realizes individual identification and data association in combination with the RFID ear tag; the environmental sensors include temperature sensors, humidity sensors, and harmful gas sensors, which real-time monitor the environmental parameters in the pigsty; the intelligent feeder integrates a weight sensor and an RFID reader, records the feeding duration and feed intake of each live pig each time, and obtains the body weight information of the live pig through the weight sensor when the live pig is feeding.

3. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 2, wherein Determining the individual identity information of live pigs based on the data collected by the RFID ear tag and the inspection robot specifically includes: Using the Deepsort multi-object tracking algorithm to process the RGB video stream collected by the inspection robot to generate a visual coordinate sequence of each live pig; Using the ST-LSTM network to process the acceleration data collected by the RFID ear tag and the visual coordinate sequence collected by the inspection robot to form a 6D vector composed of acceleration data and visual coordinates, and infer the movement trajectory of live pigs; Using the Hungarian algorithm to match the ear tag ID with the movement trajectory of live pigs to determine the individual identity information of live pigs.

4. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 3, wherein The ST-LSTM network adopts a spatio-temporal separation double-branch structure. The spatial branch uses Conv1D to process the acceleration data, and the temporal branch uses a unidirectional LSTM to process the visual coordinate sequence, and fuses the outputs of the double branches through a gated attention mechanism.

5. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 1, characterized in that, Extracting physiological and behavioral characteristics of live pigs from the preprocessed multi-modal data using a multi-modal deep learning model specifically includes: Extracting the weight information of live pigs based on a multi-modal live pig weight estimation model that fuses RGB images and depth images; Measuring the body temperature information of live pigs based on a live pig body temperature detection model that fuses RGB images and infrared images; Extracting the occurrence time and frequency of various behaviors of live pigs based on a multi-modal live pig behavior recognition model that fuses live pig skeleton information and RGB video streams.

6. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 5, wherein, The multi-modal live pig weight estimation model that fuses RGB images and depth images includes: Both the RGB branch and the depth branch use the Swin Transformer as the feature extraction network to extract depth features and RGB features, obtaining a depth feature map and an RGB feature map; Using a cross-modal attention module to perform modal feature interaction on the depth feature map and the RGB feature map; Performing a 1×1 convolution operation on the fused and interacted modal features to generate a fused feature map; Based on the fused feature map, obtaining the weight information of live pigs.

7. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 5, characterized in that, The live pig body temperature detection model that measures the body temperature information of live pigs by fusing RGB images and infrared images specifically includes: Using a key point detection algorithm to detect the position of the live pig's ear root area on the RGB image; Using an affine transformation matrix to obtain the position of the live pig's ear root area on the infrared image; Obtaining an infrared temperature matrix in the ear root area, collecting the highest temperature in the infrared temperature matrix as the temperature at the live pig's ear root, and using the temperature as the live pig's body temperature.

8. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 7, characterized in that, The affine transformation matrix includes: Based on the RGB images and infrared thermal maps of live pigs collected by the data acquisition and individual recognition module, selecting a set of matching feature points; Calculating the affine transformation matrix according to the coordinate information of the matching feature points.

9. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 5, characterized in that, The multi-modal live pig behavior recognition model that fuses live pig skeleton information and RGB video streams specifically includes: a skeleton graph processing branch and an RGB processing branch; The skeleton graph processing branch uses the ST-GCN spatio-temporal graph convolutional network to extract the spatio-temporal features of the skeleton graph sequence, obtaining skeleton features; The RGB processing branch uses the R(2+1)D network to extract the spatio-temporal features of the video frame sequence, obtaining RGB features, where R represents residual and (2+1)D represents decomposing 3D convolution into independent 2D spatial convolution and 1D temporal convolution.

10. The intelligent control system for pig farming based on multi-modal data monitoring according to claim 1, characterized in that The formula for the feeding regulation module to adjust the feeding frequency is: Among them, is the regulation coefficient, is the basic feeding amount, is the currently measured health index, is the reference health index.

Images