Pig health monitoring method based on multi-source information fusion

Abstract

The invention provides a live pig health monitoring method based on multi-source information fusion, and the method comprises the steps: collecting the sound data of a live pig in a natural environment, carrying out the classification and preprocessing of the sound data, forming a sound sample data set, inputting an LSTM neural network model, and carrying out the training, and obtaining a sound detection model; acquiring a live pig attitude image to obtain an attitude sample data set, and inputting the live pig attitude sample data set into the YOLOv5s model for training to obtain an attitude detection model; inputting live pig sound data collected in real time, preprocessing the data, and inputting the data into the sound detection model to obtain sound features; inputting live pig attitude images collected in real time into the attitude detection model to obtain attitude features; measuring the body temperature of the live pig by using an infrared temperature sensor, measuring the weight of the live pig by using a weight sensor, and fusing the sound feature, the posture feature, the body temperature and the weight to obtain a fusion vector; and inputting the fusion vector into a random forest classifier for decision making, and determining the health state of the live pig. According to the method, multi-modal data are fused, comprehensive evaluation and diagnosis of the health condition of the live pigs are facilitated, and the accuracy and robustness of the whole monitoring system are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of healthy livestock and poultry breeding, and in particular to a pig health monitoring method based on multi-source information fusion. Background Art

[0002] As pig farming continues to develop in the direction of scale and intensiveness, it is impractical to use traditional manual monitoring methods to quickly analyze the health status of pigs. Moreover, when the monitoring personnel enter the farm, the pigs will have a stress response, resulting in large errors in the measurement results, which cannot accurately reflect the health status of the pigs. At the same time, too much human-animal contact will not only disrupt the daily life of pigs and affect the accuracy of monitoring, but also greatly increase the probability of transmission of zoonotic diseases. If the epidemic is out of control one day, it will cause serious economic losses to the entire breeding enterprise. Therefore, in order to improve the production capacity, quality, economic benefits, and intelligent management level of pig breeding enterprises, the domestic small and medium-sized pig breeding enterprises use manual subjective judgment on pig abnormalities, health evaluation, disease monitoring and early warning in the breeding process, which has the problems of high labor intensity, low efficiency, and untimely abnormal situation discovery and disease early warning, resulting in serious economic losses for breeding enterprises. This overcomes the current problems of single monitoring technology used in farms to monitor abnormal behavior recognition, rapid health evaluation, and low accuracy of disease diagnosis of pigs.

[0003] The invention patent with application number 202211355314.4 discloses a method for tracking the posture of pigs based on edge computing and improved yolo, including the following steps: collecting pig posture behavior images to obtain a behavior monitoring data set; dividing the behavior monitoring data set to obtain a training set and a test set; constructing a target recognition model, inputting the training set into the target recognition model for recognition, and obtaining a recognition detection result; constructing a target tracking model, inputting the recognition detection result into the target tracking model to perform recognition and tracking processing on the pig group, and obtaining a pig behavior tracking result. The above invention meets the requirements of real-time and stable tracking of the postures of multiple pigs in group farming through edge computing and improved algorithms, and the algorithm has very good recognition and tracking effects on pigs of different numbers and different orientations, which is helpful to promote the unmanned and healthy farming of group-raised pigs, and lays a solid foundation for early warning of diseases in pigs in the later stage and taking countermeasures against them. However, the above patent only uses images as a monitoring method, which has certain limitations compared with multimodal data fusion, resulting in incomplete and untimely health evaluation and disease warning of pigs, which are mainly reflected in the following aspects:

[0004] 1. Unable to fully detect the health status of pigs: Although the posture and behavior of pigs can reflect some of their health status, their physiological signs (such as body temperature, heart rate, and coughing) are often important precursors to diseases or health problems. It is difficult to fully assess the health status of pigs based on images alone.

[0005] 2. Limited monitoring accuracy: Pig posture detection in a group-raising environment is easily affected by external factors such as crowding, occlusion, and changes in external light, which affects the detection accuracy of the algorithm and leads to a decrease in the accuracy of posture recognition and tracking.

[0006] 3. Failure to detect early health problems in time: Some pig health problems (such as fever and coughing) may only manifest as changes in body temperature or voice in the early stage, and changes in posture may not be obvious. Relying on image monitoring may lead to the failure to detect health problems in time and miss the opportunity for early intervention. Summary of the invention

[0007] In view of the existing single monitoring means and the contact sensors used in pig health monitoring, which have stress reactions on pigs, large errors, limited transmission distances, and poor equipment endurance, the present invention proposes a pig health monitoring method based on multi-source information fusion, which improves the accuracy and reliability of health monitoring through multimodal monitoring, combined with improved YOLOv5s posture recognition, LSTM sound recognition, and vital sign sensor detection. The present invention integrates monitoring parameters such as pig posture, sound, and vital signs, and uses deep learning and multi-source information fusion technology to build an intelligent pig health monitoring and disease warning platform, which helps managers to detect pig abnormalities, disease warnings, and provide technical support for precise feeding, healthy breeding, and scientific management.

[0008] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a pig health monitoring method based on multi-source information fusion, the steps of which are as follows:

[0009] Step 1: Collect the sound data of pigs in the natural environment, classify and preprocess the sound data to form a sound sample data set, input the sound sample data set into the LSTM neural network model for training, and obtain the sound detection model;

[0010] Step 2: Collect pig posture images at different angles and different time periods, perform data enhancement and posture labeling on the pig posture images to obtain a posture sample data set, input the pig posture sample data set into the YOLOv5s model for training, and obtain a posture detection model with optimized parameters;

[0011] Step 3: Input the real-time collected pig sound data into the sound detection model for preprocessing to obtain sound features; input the real-time collected pig posture image into the posture detection model with optimized parameters to obtain posture features; use an infrared temperature sensor to measure the pig's body temperature features, use a weight sensor to measure the pig's weight features, and fuse the sound features, posture features, body temperature features and weight features to obtain a fusion vector;

[0012] Step 4: Input the fusion vector into the trained random forest classifier to make a decision and determine the health status of the pig.

[0013] Preferably, the pig health monitoring system used includes a vital sign monitoring component, a sound monitoring component and a video monitoring component, which are all connected to a data acquisition system, which is connected to a host computer, which is provided with a data storage system and an alarm system, which is connected to a processing end via Ethernet, which is provided with a real-time monitoring system, which is respectively connected to the data storage system and the alarm system, and which processes and analyzes the data collected by the data acquisition system to realize online prediction of pig abnormalities, disease warnings and health diagnosis models, and transmits the obtained health status data of the pigs to the host computer, which displays the health status data of the pigs through a graphical user interface, and issues an alarm through the alarm system when abnormal pig data is detected;

[0014] The real-time monitoring system uses an improved YOLOv5s model to process pig status images for posture recognition and analysis; the real-time monitoring system uses an LSTM neural network model to process sound signals to achieve sound recognition and analysis.

[0015] Preferably, the vital sign monitoring component includes an infrared temperature sensor and a weight sensor, the infrared temperature sensor is used to measure the body temperature of the pigs in real time, and the weight sensor is used to monitor the weight of the pigs; the sound monitoring component includes a microphone, which is installed in the pig house to capture the sound signals of the pigs including breathing, calling and other sound characteristics; the video monitoring component includes a camera, which is installed above or on the side of the pig house to capture the activities of the pigs in real time and provide high-resolution posture images of the pigs;

[0016] The host computer is connected to the infrared temperature sensor, the weight sensor, the microphone and the camera in the same local area network to receive the raw data collected in real time;

[0017] The health status data includes image recognition results, sound recognition results, vital sign data and health status information.

[0018] Preferably, the sound data is classified into: destruction, eating, running, surprise, long cough, short cough;

[0019] The preprocessing is to sequentially downsample, segment and filter the sound data, input the filtered sound signal into a Mel filter bank to convert the filtered sound signal into a Mel spectrum graph, and form a sound sample data set to be trained according to the Mel spectrum graph;

[0020] The posture labels include: eating, crawling, dog sitting, side lying and standing.

[0021] Preferably, the downsampling is to reduce the sampling rate of the sound data from the original frequency to a sampling rate of 16000 Hz; the segmentation is to segment the downsampled sound data into smaller samples, and the segmentation time interval is set to 1 second; the filtering is to filter out the sound signals with a threshold value lower than 20 from the segmented sound data;

[0022] The method for converting the filtered sound signal into a Mel spectrum graph is as follows: dividing the filtered sound signal into frames of 25ms each; adding a Hamming window to each frame; performing a short-time Fourier transform on each frame of the sound signal after adding the Hamming window to obtain a spectrum graph; filtering the spectrum graph of each frame through a Mel filter bank to obtain a Mel spectrum graph; performing mean normalization processing on the Mel spectrum in the Mel spectrum graph to obtain a processed Mel spectrum graph;

[0023] The Mel filter bank is a triangular filter;

[0024] Perform timed frame extraction or dynamic frame extraction based on motion detection on the video captured by the camera to extract image frames and obtain representative pig posture images; use mosaic-9 to enhance the pig posture images;

[0025] The target detection and annotation tool is used to annotate each pig posture image with posture labels. The annotation content includes defining anchor boxes for each posture and assigning corresponding classification labels.

[0026] Preferably, the method for the LSTM neural network model to process the mel-spectrogram in the sound sample data set is:

[0027] 1) Flatten and arrange the Mel-spectrograms according to each time step and feature dimension to form a one-dimensional sequence;

[0028] 2) Input the one-dimensional sequence into the time-distributed fully connected layer, activate it using the tanh function, and generate the output sequence S;

[0029] 3) Use a bidirectional LSTM layer to perform bidirectional LSTM processing on the output sequence S to generate forward and reverse hidden states to obtain the complete output sequence X;

[0030] 4) Concatenate the output sequence S and the bidirectional output sequence X along the feature axis to obtain a feature vector;

[0031] 5) The first fully connected layer uses the ReLU activation function to perform weighted summation on the feature vectors, and the maximum pooling layer downsamples the data output by the first fully connected layer; the second fully connected layer applies the ReLU activation function to the downsampled data for weighted summation to obtain multi-dimensional data;

[0032] 6) Set the discard rate to 0.2 and flatten the multidimensional data into a one-dimensional vector;

[0033] 7) The last fully connected layer uses the ReLUrelu activation function to perform weighted summation on the one-dimensional vector and uses L2 regularization to add a penalty term to the network weights;

[0034] 8) The output layer is a fully connected layer with 10 units, using the softmax activation function for multi-class classification.

[0035] Preferably, the YOLOv5s model is an improved YOLOv5s model, and the improved YOLOv5s model is:

[0036] A new attention mechanism CBAM is introduced after the seventh layer of the backbone network. The new attention mechanism CBAM includes a spatial attention module and a channel attention module. The channel attention module processes the input feature map F to obtain the channel weight. The channel weight is multiplied by the input feature map F to obtain the feature map F′. The feature map F′ is passed as input to the spatial attention mechanism. The spatial attention module processes the feature map F′ to obtain the spatial weight. The spatial weight is multiplied by the feature map F′ to obtain the output feature map.

[0037] The FPN module in the neck part of the YOLOv5s model is modified to a BiFPN module, and the BiFPN module is used to improve the bidirectional cross-scale connection and perform weighted feature fusion.

[0038] Preferably, a new attention mechanism CBAM is added after the third feature extraction layer C3;

[0039] The channel attention module performs global average pooling and global maximum pooling on the input feature map, generating two pooling results representing the average value and maximum value of the feature map respectively; uses a shared fully connected layer to build the correlation between channels, adds the outputs of the two fully connected layers to obtain a feature map, and activates the feature map through the Sigmoid function to obtain the channel weight M c ;

[0040] The spatial attention mechanism performs global average pooling and global maximum pooling on the feature map F′ to obtain a pooling result, uses a 7×7 convolution kernel to perform a convolution operation on the pooling result, and activates the feature map after the 7×7 convolution kernel operation through a Sigmoid function to obtain a spatial weight;

[0041] The shared fully connected layer is a multi-layer perceptron, which includes two fully connected layers. The first fully connected layer reduces the dimension of the pooling result to 1 / 16 of the original number of feature channels, and the second fully connected layer restores it to the original number of channel features. The output of the second fully connected layer is added to the output of the first fully connected layer to obtain a feature map, which is passed to the Sigmoid activation function for activation to obtain the normalized channel weight:

[0042]

[0043] Among them, W0 and W1 are the weights of the first and second fully connected layers respectively, σ() is the Sigmoid activation function, MLP() represents the multi-layer perceptron, AvgPool and MaxPool represent the global average pooling function and the global maximum pooling function respectively. represents the feature map obtained by global average pooling, Represents the feature map obtained by global maximum pooling;

[0044] The global average pooling of the spatial attention mechanism performs global average on each spatial position to generate a two-dimensional feature map, which represents the average value on all channels; the global maximum pooling of the spatial attention mechanism takes the maximum value on all channels for each pixel position to generate another two-dimensional feature map, which represents the strongest response in the spatial dimension; the 7×7 convolution operation in the spatial attention mechanism performs feature extraction and fusion on the entire space;

[0045] The feature map after the 7×7 convolution kernel operation is activated by the Sigmoid function to obtain the normalized spatial weight:

[0046] M s =σ(f 7×7 (AvgPool(F′))+f 7×7 (MaxPool(F′)))

[0047] Among them, f 7×7 Represents the 7×7 convolution kernel operation function;

[0048] The BiFPN module is as follows: deleting nodes that do not contribute to the standard in the two fusion feature networks of the 7th and 5th layers in the non-adjacent feature pyramid; adding additional edges from the original input to the output node in the 6th layer of the feature pyramid to fuse more features without increasing too much cost; treating a pair of paths as a feature layer, and then repeating multiple times to obtain more high-level feature fusion, and using a fast normalized fusion method with weights for fusion, the calculation formula is:

[0049]

[0050] Among them, I i is the input feature, O is the output feature, ω i and ω j is a learnable weight, ε = 0.0001 is a small value;

[0051] The BiFPN module first performs top-down feature fusion and then bottom-up feature fusion.

[0052] Preferably, the infrared temperature sensor is installed on the side of the pig. When it is used for the first time or when the environmental conditions change, the infrared temperature sensor is calibrated for the ambient temperature, and the influence of the external temperature on the measurement accuracy is compensated by the internal algorithm; the conversion between the infrared radiation energy and the actual temperature is ensured to be accurate according to the emissivity ε of the pig's skin, and the real temperature is obtained;

[0053] The infrared temperature sensor collects infrared images of pigs in real time, captures infrared radiation from the pigs’ body surface, converts infrared radiation energy into temperature data, and generates infrared thermal maps. It uses pixel-by-pixel scanning and temperature compensation algorithms to scan the entire infrared thermal map to identify the highest temperature point on the pigs’ body surface, and records and transmits the temperature data of the identified highest temperature point to the data storage system, continuously collecting temperature data and recording changes in pigs’ body temperature.

[0054] Use a filtering algorithm to filter the temperature data of the highest temperature point to eliminate abnormal temperature readings that may be caused by environmental changes or sensor noise;

[0055] The temperature difference ΔT is obtained based on the pig's body temperature after filtering and the normal body temperature. If the temperature difference ΔT>T threshold , then trigger the alarm;

[0056] The method for calibrating the weight sensor is: in the absence of live pigs, record the output signal V output (no load), as the no-load reference point V empty =V output (no load); place a standard weight of known weight on the weight sensor, record the output signal of the weight sensor, and calculate the calibration factor based on the known weight Among them, W known is the known weight of the standard weight, V output (Standard Weight) is the weight of the standard weight measured;

[0057] Converted to actual pig weight W by calibration factor C pig =C·(V output (Pig)-V empty ), where V output (Pig) is the weight of the pig measured by the weight sensor;

[0058] The weighing data is smoothed by moving average filtering to obtain a stable weight reading: Among them, W filtered [n] is the weight data after filtering, N is the size of the filter window; W pig [ni] represents the weight data collected at i time points before the current moment;

[0059] Apply the error correction algorithm for further adjustment to obtain the corrected weight data W final =W filtered +δ(T,H,θ); where δ(T,H,θ) is the error correction term that takes into account body temperature T, humidity H, and tilt angle θ.

[0060] Preferably, the sound feature includes the confidence of each type of sound, and the posture feature includes the confidence of each posture;

[0061] The judgment method of the random forest classifier for decision making is:

[0062] Priority judgment: If the body temperature is abnormal, it is directly judged as 'unhealthy'; if the weight is abnormal and accompanied by abnormal posture or voice, it is judged as 'unhealthy'; if the body temperature is normal and the weight is slightly abnormal, it is judged as 'sub-healthy';

[0063] If the posture is "dog sitting" or the sound is "coughing", and the confidence level is higher than 0.7, they are marked as "sub-healthy" respectively. If the posture and sound are marked as "sub-healthy" at the same time, and the confidence level is higher than 50%, the comprehensive judgment is "unhealthy";

[0064] The random forest classifier dynamically adjusts the weight of each feature based on the confidence level, and comprehensively judges the health status of pigs through the weighted and nonlinear decision function f(X): f(X) = ω1·T body +ω2·S+ω3·P+ω4·ΔW

[0065] Among them, f(X) is the prediction result output by the model, which indicates the confidence of the model in the health of pigs; ω1, ω2, ω3, and ω4 are the weights of weight change ΔW, posture P, body temperature T, and sound feature S, respectively;

[0066] The classification threshold of pig health status is set to θ. If the predicted results of the health status f(X) are mapped to the classification threshold values ​​of 0.7<θ≤1, 0.4≤θ<0.7, and 0≤θ<0.4, respectively, the comprehensive evaluation results of the corresponding pig health diagnosis model are "healthy", "sub-healthy", and "unhealthy".

[0067] Compared with the prior art, the present invention has the following beneficial effects: the present invention integrates the multi-dimensional data of pig image posture recognition, voice recognition and body temperature monitoring, which is conducive to more comprehensive monitoring of the health status of pigs and comprehensive analysis and evaluation of posture, voice and body temperature. In view of the interference of pig posture recognition in complex scenes, the present invention integrates pig sound and vital sign detection data to effectively make up for the limitations of single posture detection, which is helpful for comprehensive evaluation and diagnosis of the health status of pigs, and further improves the accuracy and robustness of the overall monitoring system. The present invention is more conducive to timely discovery, evaluation and diagnosis of the health status of pigs by pig sound and vital sign detection data than single pig posture detection data. Through voice recognition, LSTM prediction model and vital sign detection sensor, the present invention can detect potential health problems and provide timely warnings before the posture of pigs changes significantly.

[0068] The present invention combines a non-contact infrared temperature sensor to detect the body temperature of pigs in real time, uses a double shear beam weighing sensor to measure the weight of pigs when eating, and transmits the measured body temperature and weight data to the host computer with the help of a wireless sensor network. The host computer combines the body temperature and weight detection data to warn of abnormal health and diseases of pigs; the present invention combines a deep learning model based on LSTM to analyze and extract the temporal characteristics of the sounds of pigs such as coughing and eating, and judges the emotions or health status of pigs by identifying the sound characteristics of pigs; the posture recognition of pigs adopts an improved YOLOv5s target detection model to analyze the posture behaviors of pigs standing, crawling, eating, etc. in the video in real time. The present invention integrates the abnormal behavior characteristics of pigs' physical signs (body temperature, weight), sounds (coughing, eating) and postures (standing, crawling, eating, etc.) to extract, and combines the big data analysis model to warn and evaluate the abnormal behavior and health status of pigs, and promptly reminds the breeding personnel to carry out timely and scientific treatment, reduce losses, and improve breeding benefits.

[0069] The present invention combines vital signs, audio, video perception detection technology, wireless sensor networks, cloud servers, deep learning and big data analysis technology to develop a three-dimensional pig vital signs information collection, health assessment, and abnormal behavior diagnosis and monitoring platform based on vital signs, acoustic and visual behaviors, providing a scientific basis for further improving the intelligent and precise management of pig farming and epidemic prevention and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 It is the overall flow chart of the present invention.

[0072] Figure 2 A process diagram constructed for the inventive gesture processing.

[0073] Figure 3 This is the overall block diagram of the improved YOLOv5 model in the present invention.

[0074] Figure 4 This is a data processing flowchart of the new attention mechanism CBAM of the present invention.

[0075] Figure 5 Process diagram constructed for the sound processing of the present invention.

[0076] Figure 6 The figure is a flow chart of data processing by the LSTM neural network model of the present invention.

[0077] Figure 7 Schematic diagram of random forest decision making of the present invention.

[0078] Figure 8 This is a structural diagram of the BiFPN module in the improved YOLOv5 model of the present invention.

[0079] Fig. 9 This is the network structure diagram of EfficientDet after the BiFPN module is introduced into the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] like Figure 1 As shown, a pig health monitoring method based on multi-source information fusion specifically includes the following steps:

[0082] Step 1: Collect the sound data set of live pigs in the natural environment, classify and preprocess the sound data set to obtain a sound sample data set, and input the sound sample data set into the LSTM neural network model for training to obtain a sound detection model.

[0083] The pig health monitoring system used in this application includes three parts: the pig's vital signs monitoring component, the sound monitoring component and the video monitoring component. The vital signs monitoring component includes an infrared temperature sensor and a weight sensor, which adopts non-contact infrared temperature measurement technology and high-precision weighing technology. The infrared temperature sensor is used to measure the body temperature of the pig in real time. The surface temperature is accurately detected by infrared technology to ensure that the data is accurate and reliable. The weight sensor is used to monitor the weight of the pig. It is embedded in the ground or a specific platform. The weight changes of the pig are recorded in real time through precise weighing technology to achieve real-time monitoring of the weight.

[0084] The sound monitoring component includes a highly sensitive microphone and a sound processing unit. The microphone is connected to the sound processing unit, which uses an LSTM (Long Short-Term Memory) neural network model for sound recognition and analysis. The highly sensitive microphone is installed in the pig house and can capture various sound signals of pigs, including breathing, calls and other sound characteristics. The collected sound data is transmitted to the LSTM neural network model for analysis to identify abnormal sounds that may represent health problems of pigs, such as coughing, screaming, etc.

[0085] The video monitoring component includes a high-definition camera and an image processing unit. The image processing unit uses an improved YOLOv5s model for posture recognition and behavior analysis. The high-definition camera is installed above or on the side of the pig house, which can capture the activities of pigs in real time and provide high-resolution video data. The image processing unit uses the improved YOLOv5s model to process the video data, identify the posture of pigs (such as standing, lying, abnormal behavior) and analyze their health status.

[0086] This application uses multimodal fusion of posture, sound, body temperature and weight data to comprehensively judge the health status of pigs through feature extraction and data analysis. This multimodal data fusion technology can bring the following significant effects:

[0087] 1. The system combines multimodal detection data (such as body temperature, weight, sound and posture) to provide different dimensions of information on the health status of pigs. By comprehensively analyzing these data, the overall health status of pigs can be more comprehensively and accurately assessed. Through the fusion analysis of multiple modal data, the misjudgment phenomenon that may occur from a single data source is avoided. For example, the combination of weight changes and posture can more accurately judge nutritional problems, and the combination of sound and body temperature can quickly identify disease signals.

[0088] 2. Multimodal data fusion can detect slight changes in body temperature, weight, voice, posture and other characteristics of pigs before they show obvious external symptoms. This helps breeding managers to provide early and accurate warnings of abnormal conditions such as increased body temperature, sudden weight loss, coughing, and eating habits of pigs, helping breeders to detect problems early and intervene to reduce losses.

[0089] 3. Through the complementarity of multimodal data such as posture, sound, and physical signs, the limitations of abnormal diagnosis and disease warning of a single data source are avoided, and the accuracy and reliability of the system's recognition of abnormal behavior and disease warning of pigs are improved. For example, by combining elevated body temperature, abnormal sounds (such as long coughs, short coughs), and abnormal postures (such as lying down for a long time or not eating), the system can comprehensively identify and diagnose health problems of pigs, such as fever, pneumonia, and other diseases.

[0090] 4. Multimodal data fusion and automated analysis ensure that the system can automatically and in real time monitor the health status of pigs, reducing reliance on manual monitoring and intervention frequency, saving labor costs, and improving the management efficiency and economic benefits of breeding enterprises.

[0091] like Figure 1 As shown, the present invention includes five steps: data collection, posture processing, sound processing, feature fusion and classifier. Data collection is achieved through infrared temperature sensor, weight sensor, microphone and camera. The infrared temperature sensor, weight sensor, microphone and camera are all connected to the data collection system, and the data collection system is connected to the host computer. The functions of the host computer include:

[0092] Data collection and caching: The host computer is connected to the same local area network as the infrared temperature sensor, weight sensor, microphone and camera to receive the raw data collected in real time.

[0093] Visualization of test results: The host computer displays the health status data sent back by the processing end through a "graphical user interface (GUI)". The interface includes image recognition results, sound recognition results, vital sign data and health status information.

[0094] Early warning and feedback: If the system detects health problems or abnormal behavior in pigs, the upper computer will use sound and light alarms and pop-up early warning windows to remind breeders to take timely measures.

[0095] Analysis and diagnosis: The upper computer combines the experience of breeding experts and big data analysis technology to provide breeding technicians with timely and accurate information on pig abnormalities, disease warnings and health diagnosis results, so that scientific measures can be taken in time, loss costs can be reduced, and breeding efficiency can be improved.

[0096] The present invention uses a Sony IMX415 high-definition camera, which is installed above or on the side of the pig house to provide high-resolution video data for real-time capture of pig activities. The present invention uses a Shure SM7B high-sensitivity microphone, which is installed in the pig house and can efficiently capture various sound signals of pigs, including breathing, calling and other sound features, to ensure the quality of sound acquisition. The collected raw data is transmitted to the remote processing end through an Ethernet module, and posture processing, sound processing, feature fusion and classifier are all completed at the processing end. The processing end is composed of a high-performance computing device in the same local area network as the host computer, and receives information such as physical signs, sound and video streams from the host computer in real time through the Ethernet module, realizes online prediction of pig abnormalities, disease warnings and health diagnosis models, and is responsible for storing real-time and historical data, so as to conduct subsequent timely analysis of the health status trend of pigs and historical health status backtracking.

[0097] The sound collection in the sound sample dataset is implemented by installing multiple high-quality microphones in the pig house, arranged at different positions and heights, to capture the sounds of pigs in their natural environment. Ensure that the microphones can record the sounds of pigs at different time periods (such as day and night) and in different environmental scenes (such as eating areas and resting areas). For data collection, care should be taken to avoid excessive background noise interference and ensure that the recorded sound samples are of high quality and representativeness.

[0098] The collected sound data sets are classified into Crushing, Enriched, Run, Surprise, Longcough, and Shortcough. The classified data are downsampled, segmented, and filtered, and then input into the Mel filter bank to obtain the Mel spectrum graph. The sound sample data set to be trained is formed according to the Mel spectrum graph. The specific process includes:

[0099] (1) Downsampling: Use audio processing technology to reduce the sampling rate of the sound sample dataset from the original frequency to a sampling rate of 16000Hz. Effects: 1) Reduce the number of audio data points per second, reduce hardware storage requirements and the complexity of model calculations. 2) Reduce the number of features for each audio sample, and improve the speed of feature extraction, model training and inference processing. 3) Reduce the impact of high-frequency noise, improve signal clarity and model performance. 4) Improve the adaptability of the deep learning model LSTM to processing downsampled audio data.

[0100] (2) Segmentation: The sound data in the downsampled pig sound sample dataset is segmented into smaller samples, and the segmentation time interval is set to 1 second, that is, the audio data is divided into one-second segments. Effect: 1) It helps to enhance the generalization ability of the model and its adaptability to different situations, especially when the abnormal feature dataset is small. This segmentation can expand the training set and make the model more robust. 2) By segmenting the sound data into 1-second segments, the dynamic changes of short-term abnormal behaviors can be captured more finely, and the temporal resolution of the model can be improved. 3) Reduce the context confusion caused by long audio and avoid the model being distracted by irrelevant background information or noise when processing long audio.

[0101] (3) Filtering: Filter the segmented sound data, filter out signals with a threshold value lower than 20, and retain only signals with an amplitude greater than or equal to 20. For digital audio (the amplitude range of 16-bit audio in a WAV file is -32768 to 32767), the amplitude is usually expressed as an integer or floating point number, and the range depends on the bit depth of the sample. Effect:

[0102] 1) Remove background noise or weak signals from pig sounds and improve the clarity of abnormal pig sound signals.

[0103] 2) Optimize the quality of abnormal audio fragment data through filtering to improve the accuracy and performance of the model.

[0104] 3) Reduce irrelevant signals in the data set, further reducing the computational burden.

[0105] (4) Conversion: The filtered sound signal is converted into a Mel-spectrogram feature. The specific process includes:

[0106] 1) Framing: Framing the filtered pig sound signal into 25ms frames, so the frame length of 16kHz is 400 (25ms×16kHz) sampling points. At a sampling rate of 16kHz, the number of sampling points corresponding to a step length of 10ms is 160 (10ms×16kHz) sampling points. Because the frame length is 25ms and the step length is 10ms, it means that every two adjacent frames will have an overlap of 15ms (240 sampling points). If the pig voice file is not divided into an even number of frames, its total length may not be an integer multiple of the frame length, that is, the last frame may be less than 400 sampling points, then it is padded with zeros to make its length reach 400 sampling points to ensure the consistency of processing of each frame.

[0107] 2) The framing operation will truncate the pig sound signal, resulting in leakage of the signal spectrum relative to the original signal spectrum. The windowing operation is performed on the pig sound signal after framing, which can smooth the discontinuously changing part of the pig sound signal, thereby correcting the non-periodicity of the pig sound signal and reducing spectrum leakage.

[0108] 3) Perform short-time Fourier transform (STFT) on each frame signal after frame division and windowing. The formula is:

[0109]

[0110] Here N is 512, k is the frequency index, which represents the different frequency components obtained after Fourier transform. K is the total number of frequency components, the total number of discrete frequencies of the pig sound signal. i (n) is the signal value of the nth sampling point in the i-th frame, S i (k) is the Fourier transform result of the i-th frame at the k-th frequency, that is, the amplitude or energy of the frequency component. n is the time index within the frame, indicating the n-th sampling point in a frame, and N is the length of the frame, that is, the total number of sampling points in a frame.

[0111] 4) Calculate the power spectrum of the frame, the formula is:

[0112]

[0113] Among them, x i is the i-th frame of the signal. i (k) represents the spectrum FFT (x i ) at a specific frequency k, FFT(x i ) is to perform fast Fourier transform (FFT) on the time domain signal of the i-th frame to obtain the i-th frame spectrum, and then take the modulus square of the spectrum to obtain the spectral line energy of the frame, and the power spectrum P is obtained by the above formula.

[0114] 5) The power spectrum P is filtered through a triangular filter with 26 Mel scales. The frequency response of the triangular filter is defined as:

[0115]

[0116] Where k is the frequency index, indicating the frequency position in the frequency domain, and f(m-1), f(m), and f(m+1) are the frequency left boundary, center frequency, and right boundary of the Mel filter bank. Each filter H in the filter bank m (k) all correspond to a specific frequency range on the Mel scale.

[0117] The formula for converting speech frequency to Mel frequency is:

[0118]

[0119] Where f is the speech frequency in Hz, which is the actual frequency obtained directly from the signal. m is the Mel frequency, which represents the frequency scale after Mel conversion. M(f) is the Mel frequency, which represents the value of the actual frequency on the Mel scale after nonlinear conversion.

[0120] In order to eliminate the amplitude differences between different samples of pig sound signals and improve the accuracy of training machine learning models, the Mel spectrum is mean normalized to effectively improve the convergence speed of the model and the robustness of the system.

[0121] like Figure 5 As shown in the figure, the flowchart of building the LSTM neural network model includes:

[0122] 1) The sound sample data input constructed by the Mel spectrum is flattened according to each time step and feature dimension, and arranged together to form a long one-dimensional sequence, which improves the processing speed of the LSTM neural network model and the accuracy of capturing the sequence.

[0123] 2) In order to solve the nonlinear problem of pig coughing sound recognition, which cannot be directly segmented using a linear model, the system uses the tanh function to introduce nonlinear factors, activate the generated output sequence S, and improve the model's segmentation ability for nonlinear problems.

[0124] 3) In order to better capture the contextual information in the pig sound sequence and obtain richer abnormal features, the system uses a bidirectional LSTM layer with 32 units to perform bidirectional LSTM processing on the output sequence S of the previous step, generate forward and reverse hidden states, and obtain the complete output sequence X.

[0125] 4) The output sequence S of the original time-distributed fully connected layer and the output sequence X of the bidirectional LSTM layer are concatenated along the third axis (i.e., the feature axis) to form a skip connection result, which improves the effectiveness of obtaining pig sound information transmission, avoids information loss, and enhances the model's ability to represent input signals.

[0126] 5) Then use two fully connected layers and insert a maximum pooling layer in the middle to obtain multi-dimensional data.

[0127] The first fully connected layer uses the relu activation function, has 64 units, and generates a 64-dimensional output vector.

[0128] The input of the first fully connected layer is the sequence data or feature vector after passing through the bidirectional LSTM layer and the time-distributed fully connected layer. The fully connected layer linearly transforms the input data into a new feature representation by weighted summation. The ReLU activation function is applied to preserve nonlinear information. The ReLU activation function sets negative values ​​to 0 and keeps positive values ​​unchanged.

[0129] In order to improve computational efficiency and reduce model complexity, a maximum pooling layer is added between the two fully connected layers so that the relu activation function of the second fully connected layer outputs a 32-dimensional output vector, which helps to retain the key features of the input data while suppressing useless noise and improving the robustness of the model.

[0130] 6) Flatten the multidimensional data into a one-dimensional vector so that it can be connected to the final output layer. Setting a dropout rate of 0.2 means that 20% of the neurons will be randomly ignored during training, reducing the model's dependence on specific nodes and improving the model's generalization ability.

[0131] 7) The last fully connected layer has 32 units and uses the relu activation function. With the help of L2 regularization (weight is 0.001), a penalty term is added to the weight of the network to avoid overfitting caused by excessive weights. The most important pig behavior feature information extracted by the model is converted into a 32-dimensional vector and passed to the output layer for the final prediction or classification task.

[0132] 8) The output layer is a fully connected layer with 10 units and uses a softmax activation function for multi-class classification.

[0133] Import the LSTM neural network model into the configured Python environment, such as Figure 4 As shown, the constructed sound sample data set (processed Mel spectrum) is input into the constructed LSTM neural network model, the LSTM neural network model is used to train the sound sample data set, and the model training process is accelerated by the parallel processing of a large number of matrix operations and data calculations by the GPU, and the trained model file is applied to the detection of sound data collected by the microphone, and sound detection is performed through real-time sound input.

[0134] Step 2: Collect pig posture images at different angles and different time periods, perform data enhancement and label annotation on the pig posture images to obtain a posture sample data set, input the pig posture sample data set into the improved YOLOv5 model for training, and obtain a posture detection model with optimized parameters.

[0135] like Figure 2 As shown, the image processing unit includes constructing a pig posture sample data set, optimizing and improving the YOLOv5 model, and training the model.

[0136] Video and image acquisition: Install multiple high-definition cameras in the pig house to continuously capture the postures of pigs from different angles and at different time periods. Ensure that behavioral images of pigs can be collected under various lighting conditions (such as day and night) and in different environmental scenes (such as eating areas and rest areas). In order to ensure the representativeness of the dataset, the collected data should cover pigs of different ages, sizes and breeds. The posture sample dataset should include the postures of pigs in different seasons and climatic conditions to improve the generalization ability of the model.

[0137] The video captured by the high-definition camera is framed regularly or dynamically based on motion detection to extract image frames to obtain representative posture images. The posture images are enhanced using mosaic-9, which stitches 9 images together to form a new training image to increase the diversity of the data set and improve the robustness of the model in different scenarios.

[0138] Posture label definition: Based on the typical behavior of pigs, the classification labels are defined for posture images. Common posture labels include: Feeding: Pigs lower their heads to eat, usually with their heads down, standing or half squatting; Lyingprone: Pigs lie with their abdomens on the ground, their limbs folded, and their heads facing forward or to the side; Sitting: Pigs place their hind legs on the ground and their front legs raised, similar to the sitting posture of dogs; Lying sideways: Pigs lie sideways with their limbs on one side of the ground; Standing: Pigs stand upright with their limbs supporting their bodies.

[0139] Use target detection and annotation tools such as LabelImg to annotate each pose image. The annotation content includes defining an anchor box for each pig pose and assigning it a corresponding classification label. Accurately calibrate the position and range of the pig in the pose image; the anchor box should be close to the edge of the pig so that the target detection model can learn the precise characteristics of the pose. If multiple pig poses appear in the same pose image, each pose should be annotated with an anchor box and assigned a label separately.

[0140] The pose sample dataset is divided into training set, validation set and test set. 70% is used for training, 15% for validation and 15% for testing to ensure that the training and evaluation of the model are performed on different data. The format of the pose sample dataset: According to the deep learning framework used, a standardized dataset format txt format is constructed, including image files, annotation files and corresponding classification labels.

[0141] like Figure 3 As shown in the figure, the overall block diagram of the improved YOLOv5s model is shown. The specific improvement process includes the following steps:

[0142] (1) A new attention mechanism CBAM is introduced after the seventh layer of the backbone network, that is, a new attention mechanism CBAM is added after the third feature extraction layer C3. The channel attention mechanism of the new attention mechanism CBAM performs global average pooling (AvgPool) and global maximum pooling (MaxPool) on the input feature map. Average pooling retains the average features of the entire collected image, and maximum pooling emphasizes the areas with larger feature values ​​in the collected image. The organic fusion of the two can better capture the important abnormal behavior feature areas of pigs and form a more representative global perception domain.

[0143] (2) Use a shared fully connected layer to build the correlation between channels. In CBAM, the two results after the pooling operation will pass through a shared fully connected layer (MLP) to build the correlation between the feature channels. The fully connected layer here is a multi-layer perceptron (MLP) containing two fully connected layers. The first fully connected layer reduces the dimension of the pooling result to 1 / 16 of the original number of feature channels, and the second fully connected layer restores it to the original number of channel features.

[0144] (3) Add the output of the second fully connected layer to the output of the first fully connected layer to obtain a feature map. The feature map is passed to the Sigmoid activation function for activation to obtain normalized weights. These weights are the same as the number of input feature channels. The calculation formula is:

[0145]

[0146] In the formula, W0 and W1 are the weights of the first and second fully connected layers, respectively. σ(*) is the Sigmoid activation function, MLP (Multi-Layer Perceptron) represents two fully connected shared layers, and the features of the global maximum pooling MaxPool and the global average pooling AvgPool are respectively obtained by the feature map obtained by the global average pooling (AvgPool) And the feature map obtained by global maximum pooling (MaxPool) Representation. Feature map It performs average pooling on each channel of the input feature map F to generate the average value of each channel to form a new feature vector. It performs maximum pooling on each channel of the input feature map F to generate the maximum value of each channel to form another feature vector. F represents the input feature map, which is the input data when processing the feature map output by the convolution layer in the CBAM module. It comes from the feature map of the previous layer (such as the C3 layer) and represents the local and global features extracted by the convolution layer.

[0147] (4) The spatial attention mechanism is introduced, and the feature map weighted by the channel attention mechanism is passed as input to the spatial attention mechanism. In the spatial attention mechanism, the feature map is subjected to global average pooling and global maximum pooling to obtain the pooling result. The purpose of the two pooling operations is to capture spatial information and extract the salient areas in the feature map. The pooling result is convolved with a 7×7 convolution kernel, and the two pooling results are adjusted to one feature layer; the size of the 7×7 convolution kernel means that the operation not only considers the local information of the pixel point, but also can capture a larger range of contextual information. Compared with small convolution kernels, the 7×7 convolution kernel can more effectively integrate spatial information and highlight the salient features of a specific area. The 7×7 convolution operation in the spatial attention mechanism extracts and fuses features for the entire space, rather than allocating weights in the channel dimension through a fully connected layer. The channel attention mechanism uses a fully connected layer to process the relationship between each channel, emphasizing the importance of different channels. The spatial attention mechanism processes information in the spatial dimension through convolution operations, emphasizing the importance of different spatial positions.

[0148] (5) The feature map after the 7×7 convolution kernel operation is activated by the Sigmoid function to obtain the normalized weight. The calculation formula is:

[0149]

[0150] f in the formula 7×7 Indicates that the size of the convolution kernel is 7 × 7. F′ represents the feature map processed by the channel attention module, which is passed to the spatial attention module as input: first, the input feature map F is globally averaged and max pooled to extract the global information of each channel respectively; secondly, a shared fully connected layer (MLP) is used to calculate the correlation between channels to generate channel weights; finally, the feature map F is weighted using the channel weights to obtain the channel-adjusted feature map F′.

[0151] The outputs of the spatial attention module and the channel attention module are added together to form a feature map adjusted by the new attention mechanism CBAM, and then passed to the next layer. The purpose is: by considering the attention of both channel and spatial dimensions at the same time, important features can be better highlighted, so that the model can better understand the key information in the image; the CBAM module enhances the focus on meaningful information during the feature extraction process through the attention mechanism, thereby improving the detection ability and recognition accuracy of the model; the spatial and channel attention mechanisms can help the model ignore some unnecessary features, reduce noise interference, and improve the performance of the model.

[0152] The present invention optimizes the original YOLOv5s model, and also includes using a weighted bidirectional feature pyramid network (BiFPN) to improve the bidirectional cross-scale connection of the target detection algorithm YOLOv5, and performing weighted feature fusion. BiFPN is performed on the basis of the feature pyramid network (FPN) of the neck part of the YOLOv5s model. The improvement is specifically to modify the FPN module in the neck part of the YOLOv5s model into a BiFPN module. The reference schematic diagram is as shown in FIG. Figure 7 As shown, specifically including:

[0153] (1) Delete the nodes that do not meet the contribution standards in the two fusion feature networks at different levels of the non-adjacent feature pyramids P7 and P5, that is, the nodes with only one input edge but no feature fusion; by deleting these unnecessary nodes, the amount of calculation can be reduced while retaining the key feature fusion nodes, thereby improving the calculation efficiency.

[0154] (2) Additional edges are then added from the original input to the output node in P6 to fuse more features without increasing the cost too much.

[0155] (3) Finally, a pair of paths is regarded as a feature layer, and then repeated multiple times to obtain more high-level feature fusion, which is fused using the weighted fast normalization fusion method. The calculation formula is:

[0156]

[0157] Among them, I i is the input feature, O is the output feature, ω i and ω j is a learnable weight, the ReLU activation function is used to scale the learnable weight between [0,1], and ε = 0.0001 is a small value used to ensure stable output.

[0158] Bidirectional cross-scale connection: The BiFPN module allows the features from the upper and lower layers to be fused simultaneously by introducing a learnable weight ω i ,ω j The BiFPN module can automatically adjust the contribution of different feature layers to the output, avoiding simple average fusion and learning the optimal weight for each feature map. By repeatedly stacking BiFPN layers, high-level and low-level feature maps can be fused multiple times, enhancing the model's ability to perceive pig behaviors at different scales.

[0159] In the YOLOv5s model, the Neck part adopts the FPN+PAN structure, which uses a pyramid form to connect feature maps of different scales and fuse high-level features with low-level features. Although the combination of FPN+PAN effectively improves the feature fusion capability of the network, there is also a problem, that is, the input of the PAN structure all comes from the feature information processed by the FPN structure, and part of the original feature information of the backbone feature extraction network is lost. The lack of participation of original information in learning can easily lead to deviations in the training process, thereby affecting the accuracy of detection. In order to solve this problem, a path aggregation network (PAN) is used to add a bottom-up aggregation path, which can achieve higher accuracy, but it will also increase parameters and computational complexity. Therefore, the present invention selects a weighted bidirectional (top-down + bottom-up) BiFPN to replace the feature fusion (Concat) block in the neck network.

[0160] The idea of ​​the BiFPN module is to achieve path enhancement through efficient bidirectional cross-scale connections and weighted feature fusion. First, the top-down feature fusion is performed, and then the bottom-up feature fusion is performed. The BiFPN module is based on PAN, which removes nodes with only one input and adds a new aggregation path between the original input node and the output node. Through multiple superpositions, more feature information is fused.

[0161] Taking node P6 as an example, the following two fusion feature processes are formed:

[0162]

[0163] In formulas (8) and (9), is the top-down intermediate feature, is the output feature from bottom to top, Conv represents the convolution operation, ω i ≥0 is a learnable weight, DBS represents the upsampling or downsampling operation of the input, and the rest of the features are constructed in a similar way.

[0164] The EfficientDet network structure after the introduction of BiFPN uses EfficientNet as the backbone network, BiFPN as the feature network, and shares the category / bounding box prediction network. Depending on different resource constraints, the BiFPN layer and the category / bounding box network layer will be repeated multiple times.

[0165] Import the optimized YOLOv5s model into the configured Python environment, such as Figure 2As shown in the figure, the optimized YOLOv5s model is used to train the posture sample data set, and the model is trained by GPU acceleration to obtain the trained best.pt weight file. The trained weight file is applied to camera detection, and posture detection is performed through real-time image input.

[0166] Step three: input the real-time collected pig sound data into the sound detection model for preprocessing to obtain sound features; input the real-time collected pig posture images into the posture detection model with optimized parameters to obtain posture features; use an infrared temperature sensor to measure the pig's body temperature characteristics, use a weight sensor to measure the pig's weight characteristics, and fuse the sound features, posture features, body temperature features and weight features to obtain a feature vector.

[0167] The infrared temperature sensor of the present invention uses a FLIR A320 infrared temperature sensor to measure the body temperature of live pigs in real time, and adopts high-precision infrared technology to accurately detect the body surface temperature of live pigs without contacting them. The steps of using the FLIR A320 infrared temperature sensor to measure the body temperature of live pigs involve equipment installation, sensor calibration, data acquisition, processing and analysis.

[0168] Install the FLIR A320 infrared temperature sensor in the pig house, 1.2m above the pig trough, without any obstacles in the field of view that block the transmission of infrared rays. Ensure that the sensor can cover the main activity area of ​​the target pigs and the body surface area of ​​the pigs, especially the ears, abdomen and other key parts. Connect the infrared temperature sensor to the data acquisition system or computer, and ensure the stability of signal transmission through the interface. Debug the infrared temperature sensor to ensure that the image is clear and the focus and temperature sensing settings are correct.

[0169] When using for the first time or when the environmental conditions change, calibrate the infrared temperature sensor to the ambient temperature. Record the ambient temperature T amb The infrared temperature sensor uses an internal algorithm to compensate for the influence of external temperature on the measurement accuracy. measured Based on the ambient temperature, the temperature after ambient temperature compensation is obtained, that is, the corrected temperature T obtained after considering the influence of ambient temperature on the measurement result comp =T measured +α·(T amb -T ref ), where α is the temperature compensation coefficient of the infrared temperature sensor, T ref is the reference temperature. The compensation coefficient α is usually determined by the manufacturer of the infrared temperature sensor through calibration experiments and is generally between 0.1 and 0.5. ref It is a temperature reset according to the actual use environment. When the ambient temperature is relatively stable, it is close to the ambient temperature. For example, if the ambient temperature is often at 26℃, set it to 26℃.

[0170] According to the emissivity ε of pig skin, the emissivity value 0.95 is entered in the setting of the infrared temperature sensor. This step is to ensure the accurate conversion between infrared radiation energy and actual temperature. The actual temperature obtained is:

[0171]

[0172] Where E is the infrared radiation energy and σ is the Stefan-Boltzmann constant, which is a physical constant used to describe the total radiation energy emitted by an object according to its temperature. The fixed value of σ is: σ = 5.670 × 10 -8 W / m 2 k 4 .

[0173] The infrared temperature sensor collects infrared images of pigs in real time, captures infrared radiation from the pig's body surface, converts infrared radiation energy into temperature data, and generates a temperature distribution image. The temperature distribution image (also called a thermal map or thermal imaging image) is an image generated after processing the infrared radiation data collected by the infrared temperature sensor. The infrared detector in the sensor detects the amount of infrared radiation in each pixel and converts these signals into corresponding temperature data. Infrared thermal image I thermal =f(T body ), where T body is the body surface temperature matrix, which represents the original temperature data collected by the infrared sensor, T real is the true temperature after taking into account the emissivity of the object, and T comp Through T real or T measured The result after environmental compensation. Function f() represents the conversion function from pig surface temperature to infrared thermal image, which is a nonlinear mapping function.

[0174] Infrared thermal image I thermal Automatically identify the highest temperature point T on the pig surface max , usually at the base of the ear, abdomen, etc. The infrared temperature sensor uses pixel-by-pixel scanning and temperature compensation algorithms to scan the entire image and extract the highest temperature T max The pixel point is used as the representative of the core body temperature. The pixel-by-pixel scanning and temperature compensation algorithm gradually scans each pixel in the image, records the infrared radiation intensity of each pixel, and then converts it into a temperature value. And T max = max{T(i,j)|(i,j)∈image}, where T(i,j) is the temperature value of the (i,j)th pixel in the infrared image image. The infrared image records the temperature information, and the infrared heat map is the result of converting this temperature information into a visual display.

[0175] The temperature data of the identified highest temperature point is recorded and transmitted to the data storage system through the interface of the infrared temperature sensor. The real-time monitoring system can continuously collect temperature data and record changes in the body temperature of live pigs.

[0176] The data acquisition system obtains real-time data from the infrared temperature sensor and transmits this data to the storage and monitoring system. The data acquisition system runs the FLIR Tools software provided by FLIR to process and read the data from the sensor and convert it into a temperature matrix or image data. The data storage system is a computer hard disk used to save historical temperature data and images. The real-time monitoring system displays the temperature data and heat map obtained from the infrared temperature sensor through a graphical user interface (GUI). The interface can display the current highest temperature point and temperature distribution map. When the body temperature of the pig is abnormal (for example, the temperature exceeds the threshold), the system can trigger an automatic alarm. Temperature trend analysis: The system will record and display the temperature change trend over a period of time to monitor the health of the pigs.

[0177] Connection method of the overall architecture: The FLIR A320 infrared temperature sensor is connected to the data acquisition system via Ethernet. The data acquisition system transmits the collected temperature data to the data storage system through the network interface, and the data acquisition system transmits the real-time temperature data to the real-time monitoring system through the network for users to conduct visual monitoring and analysis. When the real-time monitoring system detects abnormal body temperature, it will issue an alarm to relevant personnel.

[0178] The real-time monitoring system uses a filtering algorithm to process the temperature data of the highest temperature point to eliminate abnormal temperature readings that may be caused by environmental changes or sensor noise. Sliding average or other filtering algorithms can be used to smooth the temperature data to ensure data stability.

[0179]

[0180] Among them, T filtered [n] is the temperature data after filtering. N represents the window size of the sliding average when processing temperature data, that is, the number of consecutive temperature data to be calculated. For example, when N = 10, it means that the temperature data of 10 consecutive samples are averaged. n represents the current time point or sampling time. As time goes by, n gradually increases, which means that the temperature data is recorded in chronological order. T max [ni] represents the temperature of the highest temperature point measured at the nith sampling time. max Weighted averaging or filtering is performed to smooth the data and reduce the effect of noise on temperature.

[0181] The real-time monitoring system analyzes temperature data over a period of time and identifies possible abnormal changes in body temperature. If the pig's body temperature continues to be higher than the normal range, the real-time monitoring system will sound an alarm to alert staff to possible health problems.

[0182] Temperature difference ΔT = T filtered [n]-T baseline , if the temperature difference ΔT>T threshold , then the alarm is triggered. baseline Indicates the normal body temperature baseline value of pigs. This value can be determined by statistical analysis of temperature data over a period of time or by referring to the normal body temperature range of pigs. Generally speaking, the normal body temperature range of healthy pigs is between 38℃ and 39.5℃. baseline An average value can be set within this range, depending on the actual situation and data. threshold The threshold for abnormal body temperature is used to determine whether the pig has a fever or health problems. The threshold can be set to a critical value that exceeds the normal body temperature. For example, if Tbaseline is set to 39.5℃, then T threshold It can be set to around 0.5℃, that is, when the pig's body temperature exceeds 40℃ (Tbaseline+Tthreshold), the system will trigger an alarm.

[0183] T baseline The value is usually between 38℃ and 39.5℃, depending on the normal physiological body temperature of the pig. threshold : Usually set to 0.5℃ to ℃, which means that once the body temperature exceeds this range of the baseline, an abnormal alarm is triggered. These values ​​are adjusted according to actual health monitoring needs.

[0184] The processed body temperature data is presented to the user in the form of a heat map to monitor the health status of the pigs in real time. The real-time monitoring system can also generate trend analysis reports based on the historical data of the pigs' body temperature. The GUI interface uses the NumPy data analysis library to calculate the average body temperature and temperature fluctuation range based on the received body temperature data, and identifies abnormal data (such as body temperature changes above the threshold) to generate a PDF report containing key information such as body temperature trends and abnormal points.

[0185] Use ZEMICHM9B double shear beam load cell to measure pig weight. Install the ZEMICHM9B sensor directly below the weighing platform in the pig feeding aisle of the pig house and firmly fix it on the supporting structure of the weighing platform to ensure that no additional vibration or displacement is generated when the pig stands on it, thereby improving the weighing accuracy of the pig. Use a connector to connect the output end of the weighing sensor to the data acquisition system to ensure the stability of signal transmission.

[0186] In the absence of live pigs, record the output signal V of the weighing sensoroutput (no load), as the no-load reference point V empty =V output (No load), this step is to eliminate the bias signal of the sensor itself.

[0187] Use a standard weight of known weight to calibrate the weight sensor. Place the standard weight on the weighing platform, record the output signal of the weight sensor, and calculate the calibration factor based on the known weight. Among them, W known is the known weight of the standard weight, V output (Standard Weight) is the weight of the standard weight to be measured. output (Standard weight) = The value of the electrical signal output by the sensor when the standard weight is placed on the weighing platform. known The calibration factor C is calculated by comparing the difference between the load cell output signal and the sensor output signal (compared with the no-load signal).

[0188] Hang a pig trough above the weighing platform and place pig feed to attract pigs to enter the weighing area naturally, guide the pigs to the weighing platform, and ensure that they stand stably in the center of the platform. The weight sensor will sense the weight of the pig and output a corresponding electrical signal. The data acquisition system receives the output signal of the weight sensor in real time and converts it into the actual weight W of the pig through the calibration factor C. pig =C·(V output (Pig)-V empty ), where V output (Pig) is the weight of the pig measured by the weight sensor.

[0189] When the pig is moving or unstable, the weight sensor's weighing data may fluctuate. The moving average filter is used to smooth the weighing data and obtain a stable weight reading: Among them, W filtered [n] is the weight data after filtering, and N is the size of the filter window. pig [n] represents the pig weight collected at the current moment. pig [ni] represents the weight data collected at the previous i time points (i.e., the past i seconds or sampling periods).

[0190] Taking into account the influence of environmental factors (such as temperature, humidity, and slight tilt of the platform) on the measurement results, the error correction algorithm can be applied for further adjustment to obtain the corrected weight data W final =W filtered +δ(T,H,θ); where δ(T,H,θ) is the error correction term that takes into account temperature T, humidity H, and tilt angle θ.

[0191] The corrected weight data W finalThe data is recorded in the data storage system for further analysis and management. If the real-time monitoring system detects abnormal weight data of pigs, an alarm can be issued to remind staff to check.

[0192] The feature vector construction and annotation process is as follows:

[0193] Posture characteristics: P i =[P i1 ,P i2 ,···,P im ] T ; where P i1 ,P i2 ,···,P im is the confidence of each posture. The model will output the confidence value of each possible posture. For example, standing, lying, side-lying and other postures, each posture will have a corresponding confidence score, and m represents the total number of postures. For example: 'dog sitting': P1 = 0.8 (80% confidence); 'standing': P2 = 0.15 (15% confidence).

[0194] Labeling rules: When the posture is recognized as "dog sitting" and the confidence level is > 0.7, it is labeled as "sub-healthy". The rest of the postures are labeled as "healthy".

[0195] Sound characteristics: S i =[S i1 ,S i2 ,···,S in ] T ; Among them, S i1 ,S i2 ,···,S in Represents the confidence of different sound features, and n represents the total number of sound features. For example: 'Long cough': S1 = 0.7 (70% confidence); 'Short cough': S2 = 0.2 (20% confidence).

[0196] Labeling rules: If 'cough' is recognized and the confidence S1>0.8 or S2>0.8, it is labeled as 'sub-healthy'. Other sound features are labeled as 'healthy'.

[0197] Physical characteristics: C i =[t i ,w i ] T Among them, C i Represents the body temperature and weight of the i-th sample, which is a scalar (i.e., both the body temperature feature and the weight feature have only one value).

[0198] Labeling rules: If the body temperature exceeds the normal range (such as >39.5℃), it is directly labeled as 'unhealthy'. Samples within the normal temperature range are labeled as 'healthy'. If the weight changes significantly or the weight is significantly lower than the normal range, it is labeled as 'unhealthy'. Samples with stable or slightly fluctuating weight are labeled as 'healthy' or 'sub-healthy'.

[0199] Fusion vector F i It is obtained by concatenating these three eigenvectors by column: i =[P i1 ,P i2 ,···,P im ,S i1 ,S i2 ,···,S in ,t i ,w i ] T Each pig has a unique RFID tag, and each data is attached with the ID of this pig every time it is collected. In this way, the system can identify data from different pigs and prevent data confusion.

[0200] Step 4: Input the feature vector into the trained random forest classifier to make a decision and determine the health status of the pig.

[0201] like Figure 6 The figure shows a schematic diagram of random forest decision making. The decision part uses the trained random forest classifier for real-time data processing and prediction. The overall performance is improved by combining multiple models to reduce the risk of overfitting of a single model. In the training phase, the collected historical data of pigs is used to construct a training set. Each training sample includes the fused vector F i , the corresponding label yi, that is, the health status of the pig, such as "healthy", "sub-healthy", "unhealthy"; in the real-time prediction stage, new pig data will be continuously collected, and the fusion vector F of each pig at the current moment will be i Input to the trained random forest classifier. Receive the output of each sensor in real time, generate the corresponding feature vector, and ensure that the timing of the data corresponds to the ID. Once the fusion vector F i The generated data will be fed into the random forest classifier to predict the health status.

[0202] Specific judgment methods include:

[0203] Priority judgment: Abnormal body temperature (highest priority): If the body temperature is T body Abnormal (such as greater than 39.5℃ or less than 37℃), directly judged as "unhealthy". Abnormal weight: If the weight is abnormal and accompanied by abnormal posture or voice, it is judged as "unhealthy"; if the body temperature is normal and the weight is slightly abnormal, it is judged as "sub-healthy".

[0204] Comprehensive judgment of posture and sound: Single abnormality: If the posture is "dog sitting" or the sound is "coughing", and the confidence is high (such as greater than 0.7), they are marked as "sub-healthy". Multiple abnormalities: If the posture and sound are marked as "sub-healthy" at the same time, and the confidence is high > 50%, the comprehensive judgment is "unhealthy".

[0205] The random forest classifier dynamically adjusts the weight of each feature based on confidence i , if a certain feature is more reliable in predicting the health status, a higher weight is given to the feature so that the impact of the feature is more important in the comprehensive judgment. Finally, the health status of the pig is comprehensively judged through the weighted and nonlinear decision function f(X):

[0206] f(X)=ω1·T body +ω2·S+ω3·P+ω4·ΔW (12)

[0207] Among them, f(X) is the prediction result output by the model, which indicates the confidence of the model in the health of pigs. Its value is between 0 and 1. The higher the value, the better the health status, and the lower the value, the worse the health status. Its value is between 0 and 1. ω1, ω2, ω3, and ω4 are the weights of weight change ΔW, posture P, body temperature T, and sound feature S, respectively. The weights are dynamically adjusted according to the confidence of the features, indicating the importance attached to different features. ΔW is the weight change of pigs, P is the posture feature value, T body is the body surface temperature, and S is the sound feature value. Each feature is assigned a different weight in the final classification decision.

[0208] The classification threshold of pig health status is set to θ. If the predicted results of the health status f(X) are mapped to the classification threshold values ​​of 0.7<θ≤1, 0.4≤θ<0.7, and 0≤θ<0.4, respectively, the comprehensive evaluation results of the corresponding pig health diagnosis model are "healthy", "sub-healthy", and "unhealthy".

[0209] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A pig health monitoring method based on multi-source information fusion, characterized in that: The steps are as follows: Step 1: Collect the sound data of pigs in the natural environment, classify and preprocess the sound data to form a sound sample data set, input the sound sample data set into the LSTM neural network model for training, and obtain the sound detection model; Step 2: Collect pig posture images at different angles and different time periods, perform data enhancement and posture labeling on the pig posture images to obtain a posture sample data set, input the pig posture sample data set into the YOLOv5s model for training, and obtain a posture detection model with optimized parameters; Step 3: pre-process the real-time collected pig sound data and input it into the sound detection model to obtain the sound features; Inputting the real-time collected pig posture images into the posture detection model with optimized parameters to obtain posture features; The temperature characteristics of the pigs are measured using an infrared temperature sensor, and the weight characteristics of the pigs are measured using a weight sensor. The sound characteristics, posture characteristics, temperature characteristics, and weight characteristics are fused to obtain a fusion vector. Step 4: Input the fusion vector into the trained random forest classifier to make a decision and determine the health status of the pig.

2. The pig health monitoring method based on multi-source information fusion according to claim 1 is characterized in that: The pig health monitoring system used includes a vital sign monitoring component, a sound monitoring component and a video monitoring component, which are all connected to a data acquisition system, which is connected to a host computer, which is provided with a data storage system and an alarm system, which is connected to a processing end via Ethernet, which is provided with a real-time monitoring system, which is respectively connected to the data storage system and the alarm system, and which processes and analyzes the data collected by the data acquisition system to realize online prediction of pig abnormalities, disease warnings and health diagnosis models, and transmits the obtained health status data of the pigs to the host computer, which displays the health status data of the pigs through a graphical user interface, and issues an alarm through the alarm system when abnormal pig data is detected; The real-time monitoring system uses an improved YOLOv5s model to process pig status images for posture recognition and analysis; the real-time monitoring system uses an LSTM neural network model to process sound signals to achieve sound recognition and analysis.

3. The pig health monitoring method based on multi-source information fusion according to claim 2 is characterized in that: The vital sign monitoring component includes an infrared temperature sensor and a weight sensor. The infrared temperature sensor is used to measure the body temperature of the pigs in real time, and the weight sensor is used to monitor the weight of the pigs. The sound monitoring component includes a microphone installed in the pig house to capture the sound signals of the pigs including breathing, calling and other sound characteristics. The video monitoring component includes a camera installed above or on the side of the pig house to capture the activities of the pigs in real time and provide high-resolution posture images of the pigs. The host computer is connected to the infrared temperature sensor, the weight sensor, the microphone and the camera in the same local area network to receive the raw data collected in real time; The health status data includes image recognition results, sound recognition results, vital sign data and health status information.

4. The pig health monitoring method based on multi-source information fusion according to any one of claims 1 to 3, characterized in that: Classify the sound data into: destruction, eating, running, surprise, long cough, short cough; The preprocessing is to sequentially downsample, segment and filter the sound data, input the filtered sound signal into a Mel filter bank to convert the filtered sound signal into a Mel spectrum graph, and form a sound sample data set to be trained according to the Mel spectrum graph; The posture labels include: eating, crawling, dog sitting, side lying and standing.

5. The pig health monitoring method based on multi-source information fusion according to claim 4 is characterized in that: The downsampling is to reduce the sampling rate of the sound data from the original frequency to a sampling rate of 16000Hz; the segmentation is to segment the downsampled sound data into smaller samples, and the segmentation time interval is set to 1 second; the filtering is to filter out the sound signals with a threshold value lower than 20 from the segmented sound data; The method for converting the filtered sound signal into a Mel-spectrogram is as follows: dividing the filtered sound signal into frames of 25 ms each; adding a Hamming window to each frame; performing a short-time Fourier transform on each frame of the sound signal after adding the Hamming window to obtain a spectrogram; The spectrum graph of each frame is filtered through a Mel filter bank to obtain a Mel spectrum graph; the Mel spectrum in the Mel spectrum graph is mean normalized to obtain a processed Mel spectrum graph; The Mel filter bank is a triangular filter; Perform timed frame extraction or dynamic frame extraction based on motion detection on the video captured by the camera to extract image frames and obtain representative pig posture images; The pig posture images were enhanced using mosaic-9; The target detection and annotation tool is used to annotate each pig posture image with posture labels. The annotation content includes defining anchor boxes for each posture and assigning corresponding classification labels.

6. The pig health monitoring method based on multi-source information fusion according to claim 4 is characterized in that: The method for the LSTM neural network model to process the Mel-spectrogram in the sound sample data set is: 1) Flatten and arrange the Mel-spectrograms according to each time step and feature dimension to form a one-dimensional sequence; 2) Input the one-dimensional sequence into the time-distributed fully connected layer, activate it using the tanh function, and generate the output sequence S; 3) Use a bidirectional LSTM layer to perform bidirectional LSTM processing on the output sequence S to generate forward and reverse hidden states to obtain the complete output sequence X; 4) Concatenate the output sequence S and the bidirectional output sequence X along the feature axis to obtain a feature vector; 5) The first fully connected layer uses the ReLU activation function to perform weighted summation on the feature vectors, and the maximum pooling layer downsamples the data output by the first fully connected layer; the second fully connected layer applies the ReLU activation function to the downsampled data for weighted summation to obtain multi-dimensional data; 6) Set the discard rate to 0.2 and flatten the multidimensional data into a one-dimensional vector; 7) The last fully connected layer uses the ReLUrelu activation function to perform weighted summation on the one-dimensional vector and uses L2 regularization to add a penalty term to the network weights; 8) The output layer is a fully connected layer with 10 units, using the softmax activation function for multi-class classification.

7. The pig health monitoring method based on multi-source information fusion according to claim 5 or 6, characterized in that: The YOLOv5s model is an improved YOLOv5s model, and the improved YOLOv5s model is: A new attention mechanism CBAM is introduced after the seventh layer of the backbone network. The new attention mechanism CBAM includes a spatial attention module and a channel attention module. The channel attention module processes the input feature map F to obtain the channel weight. The channel weight is multiplied by the input feature map F to obtain the feature map F′. The feature map F′ is passed as input to the spatial attention mechanism. The spatial attention module processes the feature map F′ to obtain the spatial weight. The spatial weight is multiplied by the feature map F′ to obtain the output feature map. The FPN module in the neck part of the YOLOv5s model is modified to a BiFPN module, and the BiFPN module is used to improve the bidirectional cross-scale connection and perform weighted feature fusion.

8. The pig health monitoring method based on multi-source information fusion according to claim 7 is characterized in that: Add a new attention mechanism CBAM after the third feature extraction layer C3; The channel attention module performs global average pooling and global maximum pooling on the input feature map, generating two pooling results representing the average value and maximum value of the feature map respectively; uses a shared fully connected layer to build the correlation between channels, adds the outputs of the two fully connected layers to obtain a feature map, and activates the feature map through the Sigmoid function to obtain the channel weight M c ; The spatial attention mechanism performs global average pooling and global maximum pooling on the feature map F′ to obtain a pooling result, uses a 7×7 convolution kernel to perform a convolution operation on the pooling result, and activates the feature map after the 7×7 convolution kernel operation through a Sigmoid function to obtain a spatial weight; The shared fully connected layer is a multi-layer perceptron, which includes two fully connected layers. The first fully connected layer reduces the dimension of the pooling result to 1 / 16 of the original number of feature channels, and the second fully connected layer restores it to the original number of channel features. The output of the second fully connected layer is added to the output of the first fully connected layer to obtain a feature map, which is passed to the Sigmoid activation function for activation to obtain the normalized channel weight: Among them, W0 and W1 are the weights of the first and second fully connected layers respectively, σ() is the Sigmoid activation function, MLP() represents the multi-layer perceptron, AvgPool and MaxPool represent the global average pooling function and the global maximum pooling function respectively. represents the feature map obtained by global average pooling, Represents the feature map obtained by global maximum pooling; The global average pooling of the spatial attention mechanism performs global average on each spatial position to generate a two-dimensional feature map, which represents the average value on all channels; the global maximum pooling of the spatial attention mechanism takes the maximum value on all channels for each pixel position to generate another two-dimensional feature map, which represents the strongest response in the spatial dimension; the 7×7 convolution operation in the spatial attention mechanism performs feature extraction and fusion on the entire space; The feature map after the 7×7 convolution kernel operation is activated by the Sigmoid function to obtain the normalized spatial weight: M s =σ(f 7×7 (AvgPool(F′))+f 7×7 (MaxPool(F′))) Among them, f 7×7 Represents the 7×7 convolution kernel operation function; The BiFPN module is as follows: deleting nodes that do not contribute to the standard in the two fusion feature networks of the 7th and 5th layers in the non-adjacent feature pyramid; adding additional edges from the original input to the output node in the 6th layer of the feature pyramid to fuse more features without increasing too much cost; treating a pair of paths as a feature layer, and then repeating multiple times to obtain more high-level feature fusion, and using a fast normalized fusion method with weights for fusion, the calculation formula is: Among them, I i is the input feature, O is the output feature, ω i and ω j is a learnable weight, ε = 0.0001 is a small value; The BiFPN module first performs top-down feature fusion and then bottom-up feature fusion.

9. The pig health monitoring method based on multi-source information fusion according to claim 7 is characterized in that: The infrared temperature sensor is installed on the side of the pig. When it is used for the first time or when the environmental conditions change, the infrared temperature sensor is calibrated for the ambient temperature, and the influence of the external temperature on the measurement accuracy is compensated by the internal algorithm; the conversion between the infrared radiation energy and the actual temperature is ensured to be accurate according to the emissivity ε of the pig's skin, and the real temperature is obtained; The infrared temperature sensor collects infrared images of pigs in real time, captures infrared radiation from the pigs’ body surface, converts infrared radiation energy into temperature data, and generates infrared thermal maps. It uses pixel-by-pixel scanning and temperature compensation algorithms to scan the entire infrared thermal map to identify the highest temperature point on the pigs’ body surface, and records and transmits the temperature data of the identified highest temperature point to the data storage system, continuously collecting temperature data and recording changes in pigs’ body temperature. Use a filtering algorithm to filter the temperature data of the highest temperature point to eliminate abnormal temperature readings that may be caused by environmental changes or sensor noise; The temperature difference ΔT is obtained based on the pig's body temperature after filtering and the normal body temperature. If the temperature difference ΔT>T threshold , then trigger the alarm; The method for calibrating the weight sensor is: in the absence of live pigs, record the output signal V output (no load), as the no-load reference point V empty =V output (no load); place a standard weight of known weight on the weight sensor, record the output signal of the weight sensor, and calculate the calibration factor based on the known weight Among them, W known is the known weight of the standard weight, V output (Standard Weight) is the weight of the standard weight measured; Converted to actual pig weight W by calibration factor C pig =C·(V output (Pig)-V empty ), where V output (Pig) is the weight of the pig measured by the weight sensor; The weighing data is smoothed by moving average filtering to obtain a stable weight reading: Among them, W filtered [n] is the weight data after filtering, N is the size of the filter window; W pig [ni] represents the weight data collected at i time points before the current moment; Apply the error correction algorithm for further adjustment to obtain the corrected weight data W final =W filtered +δ(T,H,θ); where δ(T,H,θ) is the error correction term that takes into account body temperature T, humidity H, and tilt angle θ.

10. The pig health monitoring method based on multi-source information fusion according to claim 9 is characterized in that: The sound feature includes the confidence of each type of sound, and the posture feature includes the confidence of each posture; The judgment method of the random forest classifier for decision making is: Priority judgment: If the body temperature is abnormal, it is directly judged as 'unhealthy'; if the weight is abnormal and accompanied by abnormal posture or voice, it is judged as 'unhealthy'; if the body temperature is normal and the weight is slightly abnormal, it is judged as 'sub-healthy'; If the posture is "dog sitting" or the sound is "coughing", and the confidence level is higher than 0.7, they are marked as "sub-healthy" respectively. If the posture and sound are marked as "sub-healthy" at the same time, and the confidence level is higher than 50%, the comprehensive judgment is "unhealthy"; The random forest classifier dynamically adjusts the weight of each feature based on the confidence level, and comprehensively judges the health status of pigs through the weighted and nonlinear decision function f(X): f(X) = ω1·T body +ω2·S+ω3·P+ω4·ΔW Among them, f(X) is the prediction result output by the model, which indicates the confidence of the model in the health of pigs; ω1, ω2, ω3, and ω4 are the weights of weight change ΔW, posture P, body temperature T, and sound feature S, respectively; The classification threshold of pig health status is set to θ. If the predicted results of the health status f(X) are mapped to the classification threshold values ​​of 0.7<θ≤1, 0.4≤θ<0.7, and 0≤θ<0.4, respectively, the comprehensive evaluation results of the corresponding pig health diagnosis model are "healthy", "sub-healthy", and "unhealthy".

Images