Pig respiratory rate monitoring method based on target detection and optical flow analysis

Abstract

The application provides a pig respiratory frequency monitoring method based on target detection and optical flow analysis, and relates to the field of intelligent livestock breeding, and comprises the following steps: S1, data acquisition; S2, construction of a respiratory significant region of interest; S3, extraction of a one-dimensional respiratory time sequence signal; S4, signal preprocessing; S5, robust frequency preliminary estimation; S6, feature construction and neural network regression prediction. The method positions a pig body region through target detection, constructs a chest and abdomen respiratory significant region of interest (ROI) in the target region, constructs a one-dimensional respiratory time sequence signal by using dense optical flow amplitude statistics of all pixels in the ROI, and outputs a respiratory frequency prediction value by combining robust frequency estimation and a neural network regression model. The method can effectively solve the problems in the prior art, such as high stress of contact type monitoring, difficulty in deployment, poor anti-interference of traditional visual monitoring, low precision, high labeling cost and the like, and can realize non-contact and high-precision monitoring of pig respiratory frequency in a complex delivery room environment, effectively reduces artificial cost of breeding, and improves the efficiency of pig health management.

Description

Technical Field

[0001] This invention relates to the field of intelligent livestock farming technology, specifically to a method for monitoring the respiratory rate of pigs based on target detection and optical flow analysis. Background Technology

[0002] Respiratory rate is one of the core physiological indicators for assessing the health status of pigs. Abnormal breathing (such as rapid, shallow, or deep breathing) is usually an early sign of pneumonia, heat stress, cardiopulmonary disease, or parturition stress. Timely and accurate monitoring of respiratory rate can enable early detection and intervention of diseases, thereby reducing breeding losses.

[0003] Currently, swine respiratory rate monitoring mainly includes two types: contact monitoring and non-contact visual monitoring. Among them, contact monitoring methods directly collect respiratory signals through chest strap respiratory sensors, implanted physiological monitoring devices, etc. This method has high measurement accuracy, but has obvious drawbacks: 1) The sensors need to be strapped to or implanted in the pig's body, which can cause stress to the animal and affect its normal physiological state; 2) The equipment is easily damaged by pigs chewing or bumping, resulting in high maintenance costs; 3) In large-scale farming, deploying and replacing equipment one by one is time-consuming and labor-intensive, making it difficult to achieve large-scale real-time monitoring.

[0004] Non-contact visual monitoring methods employ computer vision technology, using frame difference, conventional optical flow, or thermal imaging for respiratory monitoring. However, existing visual methods suffer from several problems in complex farrowing environments: 1) Poor anti-interference capability: Changes in lighting, barring, head movements, and turning over in the farrowing room can easily mask the weak abdominal respiratory fluctuations, resulting in a very low signal-to-noise ratio; 2) Insufficient feature extraction accuracy: Existing optical flow-based visual respiratory monitoring methods mainly utilize local motion information to estimate respiratory frequency. However, since respiration is a periodic movement with small amplitude and continuous distribution, local feature points are easily affected by factors such as changes in lighting, occlusion, and posture changes, thus reducing the stability of the respiratory signal; 3) Difficulty in selecting ROI (Region of Interest): Existing methods typically use fixed ROIs or directly detect the abdominal region, lacking methods to automatically construct significant respiratory regions by combining pig anatomy, thus easily introducing a large amount of non-respiratory region motion interference; 4) High annotation cost: Deep learning model training requires a large amount of high-quality labeled data, while manual annotation of pig respiratory frequency is time-consuming, labor-intensive, and subject to subjective errors. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to propose a method for monitoring the respiratory frequency of pigs in complex pigsty environments, which is susceptible to the influence of background movement, local occlusion, posture changes, sudden body movements, and harmonic interference. This method combines target region localization, construction of regions of significant interest in respiration, regional optical flow statistics, robust frequency estimation, and neural network regression prediction.

[0006] This method first acquires a video sequence of the pigs to be detected and identifies a target region containing the pig's body. Then, a Region of Significant Interest (ROI) for respiration is constructed within the target region, preferably in the central half to reduce interference from pen edges, background changes, head and tail movements, and limb movements on respiratory signal extraction. Dense optical flow analysis is then performed on consecutive video frames within the ROI, and a one-dimensional respiratory time-series signal is constructed by statistically analyzing the overall motion response of the region. This one-dimensional respiratory time-series signal is further bandpass filtered to obtain the respiratory signal within the respiratory frequency band. A robust frequency estimation module integrating Fast Fourier Transform, peak detection, and autocorrelation analysis is then used to obtain an initial estimate of the respiratory frequency. Features are extracted based on the respiratory signal and the initial respiratory frequency estimate. Finally, the features are input into a neural network regression model to output a predicted value of the pig's respiratory frequency, thereby improving the stability and accuracy of respiratory frequency monitoring in complex farming environments.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for monitoring respiratory rate in pigs based on target detection and optical flow analysis, characterized by comprising the following steps:

[0009] Step S1: Collect the video sequence of the pig to be tested, and extract consecutive video frames in chronological order.

[0010] Step S2: Determine the target region of pigs in the video frame, and construct a region of significant interest (ROI) for respiration within the target region of pigs.

[0011] Preferably, video frames are input into a target detection model to obtain a bounding box for the entire pig, and the central half of the bounding box is selected as a region of significant interest (ROI) for respiratory function. This central half is used to reduce the impact of pen edges, background changes, head and tail movements, limb movements, and noise at the edge of the bounding box on respiratory signal extraction.

[0012] Step S3: For the region of significant interest (ROI) corresponding to breathing in consecutive video frames, calculate the dense optical flow field of all pixels in the region, and construct the region motion response value based on the optical flow amplitude of all pixels. Connect the region motion response values ​​at consecutive time moments in chronological order to form a one-dimensional breathing time sequence signal.

[0013] Step S4: Perform bandpass filtering on the one-dimensional respiratory timing signal to obtain the respiratory signal within the respiratory frequency band.

[0014] Preferably, the passband range of the bandpass filter is 0.15Hz to 1.1Hz, corresponding to a respiratory frequency range of approximately 9BPM to 66BPM.

[0015] Step S5: Use the robust frequency estimation module to perform a preliminary estimation of the respiratory frequency of the respiratory signal to obtain a robust preliminary estimate of the respiratory frequency.

[0016] The robust frequency estimation module includes frequency domain estimation based on Fast Fourier Transform, time domain estimation based on peak detection, and period estimation based on autocorrelation analysis, and performs fundamental and harmonic candidate analysis on the frequency domain estimation results based on Fast Fourier Transform.

[0017] Step S6: Extract features based on the respiratory signal and the robust respiratory rate initial estimate, input the features into a pre-trained neural network regression model, and output the corresponding video respiratory rate prediction value.

[0018] Preferably, the features include a 32-dimensional handcrafted feature vector, which includes time-domain statistical features, frequency-domain energy features, spectral peak features, autocorrelation features, and respiratory dominant frequency features; wherein, the respiratory dominant frequency features are provided by the robust respiratory frequency initial estimate output in step S5.

[0019] Preferably, the neural network regression model is a multilayer perceptron regression model, which takes the 32-dimensional handcrafted feature vector as input and outputs continuous respiratory rate prediction values.

[0020] Based on further optimization of the above scheme, the specific calculation method for the one-dimensional motion response value in step S3 is as follows:

[0021] First, calculate the dense optical flow field within the ROI between two consecutive frames:

[0022]

[0023] In the formula: This represents the optical flow vector at pixel (x,y) of the dense optical flow field. and represents the horizontal and vertical displacement components of the pixel (x, y), respectively; t represents the t-th sampling time.

[0024] Calculate the optical flow amplitude of each pixel based on the optical flow vector:

[0025]

[0026] Then, the optical flow amplitude of all pixels within the ROI is averaged to obtain the one-dimensional motion response value at time t:

[0027]

[0028] In the formula: R represents the region of interest, |R| represents the number of pixels within the region of interest; s(t) represents the one-dimensional motion response value at time t.

[0029] This invention uses the average optical flow amplitude of all pixels within a Region of Interest (ROI) between adjacent video frames as the overall regional motion response at the current sampling time, which is used to characterize the intensity of respiratory motion in the chest and abdominal region. Arranging the overall regional motion response values ​​from consecutive time points in chronological order forms a one-dimensional respiratory temporal signal.

[0030] Based on further optimization of the above scheme, in step S5, the robust frequency estimation module specifically includes the following processing steps:

[0031] First, perform a fast Fourier transform on the respiratory signal obtained in step S4, and select several candidate peaks with high energy ranking within the preset respiratory frequency band. For each candidate peak, construct a candidate frequency set including the current frequency, half frequency, one-third frequency, and double frequency. Calculate the candidate frequency score based on the spectral amplitude and frequency penalty term corresponding to the candidate frequency, and select the candidate frequency with the highest score that is within the preset respiratory frequency band as the frequency domain estimation result after fundamental and harmonic correction.

[0032] Secondly, peak detection is performed on the respiratory signal obtained in step S4 to obtain the positions of multiple respiratory peaks, and the respiratory rate estimation result based on peak detection is obtained by converting the average interval between adjacent peaks.

[0033] Next, the respiratory signal obtained in step S4 is processed to remove the mean and the autocorrelation function is calculated. The position of the maximum correlation peak is found within the physiologically reasonable lag search range. The respiratory rate estimation result based on autocorrelation analysis is obtained by converting the period corresponding to the position of the maximum correlation peak.

[0034] Finally, the frequency domain estimation results, peak detection estimation results, and autocorrelation estimation results are fused for consistency: when the difference between any two estimation results does not exceed the preset consistency threshold, the average of the two consistent estimation results is taken as the initial estimate of robust respiratory rate; when the three estimation results are inconsistent, the respiratory rate estimation result based on autocorrelation analysis is preferred as the initial estimate of robust respiratory rate.

[0035] Preferably, the preset consistency threshold is 3 BPM to 5 BPM, and more preferably 5 BPM.

[0036] Based on further optimization of the above scheme, in the frequency domain estimation based on Fast Fourier Transform, the frequency of any candidate peak... The corresponding candidate frequency set is:

[0037]

[0038] Candidate frequency score:

[0039]

[0040] In the formula: Indicates candidate frequency; This represents the spectral amplitude corresponding to the candidate frequency; This represents the frequency penalty term. The candidate frequency with the highest score is selected as the frequency domain estimation result and converted into respiratory rate.

[0041] Based on further optimization of the above scheme, in step S6, the multilayer perceptron regression model includes an input layer, a first fully connected layer, a second fully connected layer, and an output layer. The input layer is used to input a 32-dimensional handcrafted feature vector; the first fully connected layer contains 256 neurons, the second fully connected layer contains 128 neurons, and the output layer contains 1 neuron, used to output continuous respiratory rate prediction values.

[0042] The training process of the multilayer perceptron regression model uses the mean squared error loss function:

[0043]

[0044] In the formula: B represents the batch sample size; This represents the true respiratory rate label of the i-th sample; This represents the model's predicted respiratory rate for the i-th sample.

[0045] Based on further optimization of the above scheme, in step S6, the neural network regression model can also be a one-dimensional convolutional neural network regression model with a lightweight temporal attention module. Correspondingly, a four-channel temporal feature is constructed based on the breathing signal, and the four-channel temporal feature is input into the one-dimensional convolutional neural network regression model with a lightweight temporal attention module to output the corresponding video breathing frequency prediction value.

[0046] The four-channel time-series features include a normalized raw respiratory signal channel, a first-order differential signal channel, a sliding mean square envelope signal channel, and a differential signal energy envelope signal channel. The normalized raw respiratory signal channel is used to characterize the overall amplitude change of the respiratory time-series signal, the first-order differential signal channel is used to characterize the local change trend between adjacent sampling points, the sliding mean square envelope signal channel is used to characterize the local energy envelope of the respiratory signal, and the differential signal energy envelope signal channel is used to characterize sudden body movement or local abnormal changes.

[0047] The one-dimensional convolutional neural network regression model with a lightweight temporal attention module includes an input layer, multiple one-dimensional convolutional modules, a lightweight temporal attention module, and a fully connected regression module. The multiple one-dimensional convolutional modules are used to extract local rhythmic features and deep temporal representations of the respiratory signal; the lightweight temporal attention module is used to generate normalized weights in the time dimension and perform weighted aggregation of time-dimensional features to replace global average pooling; the fully connected regression module is used to output continuous respiratory rate prediction values.

[0048] Based on further optimization of the above scheme, the training samples of the neural network regression model include real labeled samples and synthetic samples. The real labeled samples are obtained by manually counting breaths and converting them into respiratory frequency labels; the synthetic samples are controlled-difficulty synthetic respiratory signal samples, including one or more of the following: dominant frequency component, adjacent frequency confusion component, harmonic component, occlusion disturbance component, and power frequency interference component.

[0049] Preferably, the real labeled samples are oversampled, and the oversampled real labeled samples are mixed with the synthetic samples for training, so as to improve the adaptability of the neural network regression model to real aquaculture scenarios.

[0050] Technical effect

[0051] The present invention provides a method for monitoring the respiratory rate of pigs based on target detection and optical flow analysis, which has the following beneficial effects:

[0052] (1) In this invention, the central half of the target area of ​​the pig is preferably selected as the region of significant interest in respiration. This can reduce the influence of pen edges, background changes, head and tail swinging, limb movement and target frame edge noise on the extraction of respiratory signals and improve the quality of one-dimensional respiratory time sequence signals.

[0053] (2) The present invention utilizes the overall motion response of all pixels in the region of significant interest in breathing to construct a one-dimensional respiratory time-series signal. Compared with methods that only utilize local motion points or single local regions, it can more fully characterize the weak respiratory motion continuously distributed in the chest and abdomen region and improve the stability of the respiratory signal.

[0054] (3) The present invention employs a robust frequency estimation module that combines fast Fourier transform, peak detection and autocorrelation analysis, and performs fundamental and harmonic candidate correction in frequency domain estimation, which can reduce the impact of harmonic false detection, adjacent frequency confusion, power frequency interference and non-respiratory body movement on the initial estimation of respiratory frequency.

[0055] (4) The present invention uses the robust initial estimate of respiratory frequency as the respiratory dominant frequency feature in the 32-dimensional manual features and inputs it into the neural network regression model, so that the frequency estimation result is combined with the regression feature expression, thereby improving the accuracy and stability of respiratory frequency prediction.

[0056] (5) The present invention can use a one-dimensional convolutional neural network regression model with a lightweight temporal attention module, and construct four-channel temporal features of the original signal, differential signal, sliding mean square envelope and differential energy envelope, so that the model can simultaneously utilize respiratory amplitude, slope change, envelope modulation and sudden body movement information, and adaptively focus on the effective respiratory phase in the time dimension.

[0057] (6) The present invention uses real labeled samples and controlled difficulty synthetic samples for mixed training, and oversamples the real labeled samples, which can reduce the pressure of manual labeling while enhancing the model's adaptability to real pig house scenarios, and realize non-contact automatic monitoring of pig respiratory frequency. Attached Figure Description

[0058] Figure 1 This is a schematic diagram comparing the original video frame and the cropped target video frame of the pig in an embodiment of the present invention.

[0059] Figure 2 This is an image of the abdominal region identified by the YOLO v11 target detection algorithm in an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram comparing the original optical flow signal with the respiration signal after Butterworth bandpass filtering in an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the architecture and data flow of a one-dimensional convolutional neural network regression model containing a lightweight temporal attention module in an embodiment of the present invention.

[0062] Figure 5 This is a graph showing the relationship between model tolerance accuracy and error tolerance in an embodiment of the present invention.

[0063] Figure 6 This is a comparison chart of the tolerance accuracy and MAE of the model on the training set, validation set, and test set in the embodiments of the present invention.

[0064] Figure 7 This is a quantile-quantile distribution of the model prediction error in an embodiment of the present invention. Detailed Implementation

[0065] The following detailed description provides further details through specific embodiments. However, it should be noted that the embodiments described below are merely for illustrating the content of the invention and do not represent that the invention is limited to the described embodiments. Therefore, non-essential improvements and adjustments made to the implementation schemes by those skilled in the art based on the above-described invention still fall within the protection scope of the invention, and the scope of protection of the appended claims shall prevail.

[0066] Example:

[0067] This invention provides a method for monitoring the respiratory rate of pigs based on target detection and optical flow analysis. The overall process includes six stages: video acquisition, automatic construction of significant respiratory regions, one-dimensional respiratory signal extraction, respiratory signal preprocessing, robust initial frequency estimation, and neural network regression prediction. Each stage is executed sequentially according to the data flow order.

[0068] The dataset in this embodiment comes from videos captured by overhead cameras in the farrowing pens of pig farms. It contains breathing videos of 32 sows, which were cropped to obtain 2424 video sequences. Each video is 15-30 seconds long, with a frame rate of at least 15 fps and a resolution of 1920×1080 pixels. All videos are divided into training, validation, and test sets in an 8:1:1 ratio. The training set contains approximately 1939 videos, the validation set approximately 242 videos, and the test set approximately 243 videos. The training set is used for model training, the validation set for model parameter optimization, and the test set for model performance evaluation.

[0069] To ensure the accuracy of subsequent respiratory frequency labeling, three experienced aquaculture personnel were employed to count the number of breaths in each video segment, and the average of the three counts was taken as the manual labeling result to reduce human statistical error. Furthermore, to improve the diversity and robustness of the model training samples, this embodiment can also construct synthetic respiratory signal samples and combine real labeled samples with synthetic samples for model training, thereby balancing label quality and data scale. The invention will now be described in detail with reference to each step.

[0070] Step S1, Data Acquisition: Video sequences of pigs are acquired using a fixed camera positioned above the pigpen. The camera's installation height is preferably 2-3 meters to ensure complete coverage of each pig's body. Continuous video frames are extracted in chronological order and used as input data for subsequent respiratory region extraction and respiratory rate calculation.

[0071] Preferably, the pig is in a side-lying position in the acquired video, allowing the chest and abdomen area to be fully exposed within the camera's field of view, which facilitates the subsequent detection of respiratory micro-movements. Videos that have already undergone target cropping can also be directly used as input data for this invention without needing to perform the target cropping operation again.

[0072] Step S2, Automatic Construction of Significant Respiratory Regions: (e.g.) Figure 2 As shown, this step is used to automatically extract the Region of Significant Interest (ROI) from the video frame that can adequately characterize the respiratory fluctuations of pigs, so as to reduce the influence of non-respiratory movements on subsequent respiratory signal extraction.

[0073] First, video frames are input into a pre-trained object detection model to detect individual pigs in the video, obtaining overall target bounding boxes for the pigs. Since the overall bounding box includes the head, tail, limbs, and part of the background area, and head swaying, tail swaying, and environmental changes can easily generate movements unrelated to breathing, the overall bounding box is not directly used as the breathing analysis area.

[0074] Furthermore, based on the structural characteristics of pigs lying on their side, with the chest and abdomen located near the middle of the trunk and respiratory fluctuations mainly concentrated in the trunk area, the central half of the overall detection frame is selected to automatically construct a significant respiratory region.

[0075] In this invention, within the target detection box, using the center point of the target detection box as a reference, the central region of the target detection box is selected in both the horizontal and vertical directions. Preferably, the central half of the target detection box area is selected as the significant respiratory ROI. This central half-region can effectively reduce interference from the head, tail, limbs, and background areas at the edge of the target box, while preserving the main respiratory motion information corresponding to the mid-thigh and chest / abdomen undulations.

[0076] The aforementioned central half-region is not a unique limiting value, but rather a preferred region used for the sow side-lying video in the dataset of this invention. Its determination is based on two principles: firstly, it effectively removes non-respiratory motion interference from the head, tail, and background areas; secondly, it preserves as much continuous respiratory motion information as possible corresponding to chest and abdominal undulations, thereby improving the signal-to-noise ratio and stability of subsequent respiratory signals. For pigs of different breeds, growth stages, or shooting angles, those skilled in the art can adaptively adjust the above-mentioned ROI range based on the relative position of the chest and abdominal region within the overall target. Any region of interest where chest and abdominal respiratory undulations are the primary motion information falls within the scope of protection of this invention.

[0077] The significant respiratory region obtained using the above method provides a unified input region for the subsequent construction of the overall motion response of the region, which helps to reduce the impact of local posture changes and background motion on the respiratory rate estimation results.

[0078] Step S3, One-dimensional respiratory signal extraction: This step is used to convert the weak chest and abdominal respiratory movements in continuous video frames into a one-dimensional respiratory time sequence signal that can characterize changes in respiratory rhythm, providing a stable input for subsequent respiratory rate prediction.

[0079] Since swine respiration is a continuous, low-amplitude, periodic movement distributed across the chest and abdomen, its motion information is typically distributed throughout the entire salient respiratory region, rather than concentrated in a few local feature points. When using local motion features for respiration estimation, it is easily affected by factors such as missing feature points, local occlusion, and posture changes, leading to decreased stability of the respiratory signal. Therefore, this invention does not use local feature points as a representation of respiration, but instead utilizes the overall motion information of all pixels within the salient respiratory region to construct a regional motion response.

[0080] Specifically, for two adjacent frames, it is preferable to calculate the dense optical flow field within the central half-region (ROI) obtained in step S2, rather than directly calculating the optical flow across the entire video field. This reduces interference from background, head and tail movements, and limb motions on the extraction of respiratory signals. The formula for calculating the dense optical flow field is as follows:

[0081]

[0082] In the formula: This represents the optical flow vector at pixel (x,y) of the dense optical flow field; and These represent the horizontal and vertical displacement components of the pixel (x, y), respectively; t represents the current frame corresponding to the t-th sampling time, and t+1 represents the next frame.

[0083] Simultaneously, the optical flow amplitude of each pixel (x, y) at sampling time t is calculated using the aforementioned displacement components, with the specific formula as follows:

[0084]

[0085] After obtaining the optical flow amplitude of all pixels within the ROI, regional statistics are performed on all pixel optical flow amplitudes to characterize the overall motion intensity of the chest and abdomen region at the current sampling time. The average of the optical flow amplitudes of all pixels within the ROI is then used to obtain the one-dimensional motion response value s(t) at the corresponding time, i.e., the sampling time t. Specifically:

[0086]

[0087] In the formula: R represents the region of interest, |R| represents the total number of pixels within the region of interest; s(t) represents the one-dimensional motion response value at sampling time t. The overall motion response value of the region does not represent the motion of a single pixel or local feature point, but rather reflects the overall respiratory activity intensity of the entire salient respiratory region at the current moment. When the pig inhales or exhales, causing the chest and abdomen region to fluctuate, most pixels within the ROI produce motions with consistent direction and small amplitude, thus increasing the overall motion response value of the region; when there is a respiratory pause or the respiratory motion weakens, the overall motion response value of the region decreases accordingly.

[0088] Finally, the one-dimensional motion response values ​​s(t) at all times are concatenated to form the original one-dimensional respiratory time sequence signal s as follows:

[0089]

[0090] Where T represents the total number of sampling points in the target video.

[0091] Compared to directly using local motion features for respiratory estimation, this invention constructs respiratory timing by using the overall motion response of the region within the central half of the ROI. This can make full use of the weak motion information continuously distributed in the chest and abdomen region, reduce random errors caused by local occlusion, changes in illumination, head and tail swinging, limb movements and background changes, and improve the stability of respiratory signals and the accuracy of subsequent respiratory rate prediction.

[0092] Step S4, Signal Preprocessing: The one-dimensional respiratory time-series signal obtained in Step S3, in addition to containing the thoracic and abdominal respiratory movements of the pig, inevitably includes low-frequency drift and high-frequency random noise caused by factors such as camera noise, lighting changes, slight adjustments in the pig's posture, and environmental interference. Directly inputting the raw time-series signal into the respiratory rate prediction model can easily reduce the ability to identify respiratory cycle features. Therefore, this invention preprocesses the one-dimensional respiratory time-series signal before respiratory rate prediction to improve the signal-to-noise ratio of the respiratory signal. For example... Figure 3 As shown, Butterworth bandpass filtering is used to extract the respiratory frequency band components, resulting in the respiratory signal x(t) after filtering and preprocessing s(t):

[0093]

[0094] In the formula: , These represent the lower and upper limits of the respiratory frequency band, respectively. For pig respiration, the preferred values ​​are... = 0.15 Hz =1.10Hz, corresponding to a respiratory rate range of [0.15×60,1.10×60]=[9,66] BPM; Indicates the filter order. This indicates a bandpass filtering operation.

[0095] Based on the normal respiratory frequency distribution range of pigs, the preferred setting for the filter passband is a respiratory frequency range of 9–66 BPM, i.e., the lower limit frequency of the filter. Preferably 0.15 Hz, upper limit frequency The preferred Hz is 1.10 Hz. The above parameters can cover the normal respiratory rate range of sows in a resting state and under general breeding conditions, while suppressing slow posture changes below the respiratory rate range and random noise above the respiratory rate range.

[0096] It should be noted that the above-mentioned filtering parameters are preferred parameters for this embodiment and are not intended to limit the present invention. Those skilled in the art can adaptively adjust the filter passband and filter order according to different varieties, growth stages, sampling frame rates, or monitoring environments. As long as the respiratory cycle components are preserved and non-respiratory noise is suppressed, they all fall within the protection scope of the present invention.

[0097] After bandpass filtering, the periodic respiratory components in the one-dimensional respiratory time series signal are enhanced, and environmental noise and non-respiratory motions are effectively suppressed, resulting in a smoother and more stable respiratory signal. This provides high-quality input for subsequent robust frequency estimation, time series feature construction, and the learning of respiratory rhythm features by a one-dimensional convolutional neural network regression model with a lightweight time series attention module, thereby improving the accuracy and robustness of respiratory frequency prediction.

[0098] Step S5, Robust Frequency Estimation: Input the filtered and preprocessed respiratory signal x(t) into the robust frequency estimation module. Obtain the respiratory frequency estimation results through fast Fourier transform, peak detection and autocorrelation analysis, respectively. Then, perform fundamental frequency / octave correction and consistency fusion on each estimation result to obtain the robust respiratory frequency initial estimate.

[0099] Step S6, Neural Network Regression Prediction: Construct model input based on the filtered and preprocessed respiratory signal x(t), the robust initial estimate of respiratory frequency, and the temporal variation characteristics. Then, feed the model input into a pre-trained one-dimensional convolutional neural network regression model with a lightweight temporal attention module to output the predicted value of pig respiratory frequency for the corresponding video sequence.

[0100] The one-dimensional convolutional neural network regression model containing a lightweight temporal attention module was trained offline before actual deployment. The model training process included steps such as training dataset construction, artificial label construction, synthetic sample construction, and network parameter optimization. After training, the obtained one-dimensional convolutional neural network regression model was used for respiratory rate prediction in this step.

[0101] Specifically, three respiratory rate estimates are obtained from the filtered and preprocessed respiratory signal x(t) using the FFT main frequency estimation method, the peak detection method, and the autocorrelation method, respectively. , , The three estimation methods described above independently estimate respiratory rate from three perspectives: frequency domain dominant frequency characteristics, time domain peak interval characteristics, and signal periodic correlation. Since different estimation methods have varying sensitivities to noise, local abnormal fluctuations, harmonic interference, and peak omissions, this invention integrates FFT, peak detection, and autocorrelation analysis results through a robust frequency estimation module and performs fundamental / octave correction to reduce frequency estimation errors caused by misjudgments from a single estimation method and improve the reliability of the initial respiratory rate estimate.

[0102] Specifically, the FFT main frequency estimation method involves: firstly, performing a Fourier transform on the filtered and preprocessed respiratory signal x(t) to obtain the frequency domain signal X(f):

[0103]

[0104] In the formula: N represents the total signal length; j represents the imaginary unit; e represents exponential operation with the natural constant e as the base; f represents the frequency domain index variable, X(f) represents the frequency domain signal; t represents the time index variable. Only the positive frequency portion of the frequency domain signal is retained, and it is limited to the breathing frequency bandwidth. Since X(f) is a complex number, its amplitude |X(f)| is used to characterize the intensity of the frequency components in practical analysis.

[0105] Furthermore, within the respiratory frequency band... Within the range, several candidate peaks with the highest energy ranking are selected, and for each candidate peak, candidate sets for fundamental frequency, sub-frequency, and harmonic frequencies are constructed. For any candidate peak frequency... Construct a candidate frequency set:

[0106]

[0107] Then, the candidate frequency score is calculated based on the spectral amplitude and frequency penalty term corresponding to the candidate frequency:

[0108]

[0109] In the formula: Indicates candidate frequency; This represents the spectral amplitude corresponding to the candidate frequency; This represents the frequency penalty term. The candidate frequency with the highest score and falling within a preset respiratory frequency band is selected as the corrected frequency domain estimation result, and then converted to obtain the respiratory frequency using the FFT dominant frequency estimation method:

[0110]

[0111] FFT fundamental frequency estimation can reflect the main frequency components of the entire respiratory signal. At the same time, through the above-mentioned fundamental frequency, sub-frequency and harmonic frequency candidate analysis, it can reduce frequency misjudgment caused by the presence of spectral leakage, local noise or harmonic interference in the signal.

[0112] Peak detection is used to estimate the average time interval between adjacent respiratory cycles from a time-domain perspective. Specifically, it involves first detecting the positions of all peaks in the filtered and preprocessed respiratory signal x(t). And calculate the average interval between adjacent peaks:

[0113]

[0114]

[0115] In the formula: This represents the video frame rate, i.e., frames per second (fps); K represents the total number of peaks, and i represents the i-th peak.

[0116] Finally, the respiratory rate obtained by peak detection method is calculated:

[0117]

[0118] Peak detection can intuitively reflect the periodic changes of respiratory signals in the time domain, but the estimation results may be biased when the signal has local fluctuations, peak distortion, or missing peaks. Therefore, this embodiment integrates the peak detection results with the FFT main frequency estimation results and autocorrelation analysis results for judgment.

[0119] The autocorrelation method specifically includes the following steps:

[0120] First, the mean of the pre-filtered respiratory signal x(t) is removed:

[0121]

[0122] In the formula: x'(t) represents the output value of the respiratory signal after removing the mean, and x(t) represents the original input value of the respiratory signal after filtering and preprocessing. This represents the average value of the entire respiratory signal, where t represents the t-th sampling time.

[0123] Then, calculate the autocorrelation function using the following formula:

[0124]

[0125] In the formula: N represents the total length of the signal; m represents the number of sampling points shifted backward, i.e., the hysteresis; x'() represents the respiratory signal after removing the mean, where t represents the t-th sampling time, and t+m represents the number of sampling points shifted by m at the t-th sampling time.

[0126] Third, based on the respiratory frequency band range Obtain the lag search range ,in:

[0127]

[0128] In the lag search range Find the location of the maximum correlation peak:

[0129]

[0130] In the formula, argmax represents the position function of the independent variable, thus obtaining the optimal lag value of m corresponding to the maximum peak in the autocorrelation analysis. .

[0131] Fourth, the respiratory rate is finally obtained based on autocorrelation analysis. :

[0132]

[0133] In the formula: This indicates the video frame rate, or frames per second (fps). This represents the optimal lag for the value of m corresponding to the maximum peak in autocorrelation analysis.

[0134] Autocorrelation analysis can characterize the periodic similarity between a respiratory signal and its delayed version, and it is more robust to local peak loss than simple peak detection. However, when there is significant low-frequency drift or non-respiratory cycle motion in the signal, the autocorrelation peak may be affected. Therefore, in this embodiment, the autocorrelation estimation results are used together with the FFT main frequency estimation and peak detection results for robust estimation of respiratory rate.

[0135] In this embodiment, if the difference between any two of the FFT main frequency estimate, peak detection estimate, and autocorrelation estimate does not exceed [the specified value], then [the following condition is met]. If they are consistent, then they are considered to be the same. The value of is in the range of 3–5 BPM, preferably 5 BPM. If the FFT and peak detection are consistent, the average of the two is taken as the initial robust respiratory rate estimate; if the FFT and autocorrelation are consistent, the average of the two is taken as the initial robust respiratory rate estimate; if the peak detection and autocorrelation are consistent, the average of the two is taken as the initial robust respiratory rate estimate; if none of the three are consistent, the autocorrelation estimate is preferred as the initial robust respiratory rate estimate. The resulting initial robust respiratory rate estimate is used as one of the input features of the subsequent model to assist the neural network regression model in learning the mapping relationship between respiratory rate and time-series waveform.

[0136] After obtaining the initial robust respiratory frequency estimate, a 32-dimensional feature vector is extracted based on the filtered and preprocessed respiratory signal x(t), signal statistical characteristics, frequency domain energy characteristics, spectral peak characteristics, autocorrelation characteristics, and the initial robust respiratory frequency estimate. The dominant frequency feature in the 32-dimensional feature vector is given by the robust frequency estimation module, rather than directly using the maximum amplitude frequency of a single FFT as the dominant frequency, thereby avoiding harmonic peaks or noise peaks being mistaken for the true respiratory dominant frequency.

[0137] Furthermore, a four-channel temporal feature is constructed based on the filtered and preprocessed respiratory signal x(t), and the 32-dimensional feature vector and the four-channel temporal feature are used together as input information for a one-dimensional convolutional neural network regression model containing a lightweight temporal attention module. The four-channel temporal feature includes a normalized original respiratory signal channel, a first-order differential signal channel, a sliding mean square envelope signal channel, and a differential signal energy envelope signal channel.

[0138] Among them, the normalized raw respiratory signal channel is used to characterize the overall amplitude change of the respiratory time-series signal, the first-order differential signal channel is used to characterize the local change trend between adjacent sampling points, the sliding mean square envelope signal channel is used to characterize the local energy envelope of the respiratory signal, and the differential signal energy envelope signal channel is used to characterize sudden body movement or local abnormal changes. Through the above four-channel time-series features, the model can simultaneously obtain multiple time-series information such as respiratory waveform, local changes, energy distribution, and abnormal perturbations.

[0139] like Figure 4 As shown, the one-dimensional convolutional neural network regression model with a lightweight temporal attention module includes an input layer, multiple one-dimensional convolutional feature extraction modules, a lightweight temporal attention module, a feature fusion module, and a fully connected regression module. The input layer receives the four-channel temporal features, with an input dimension of (B, 4, T), where B represents the number of batch samples and T represents the total number of sampling points within a fixed sampling time window. Simultaneously, the 32-dimensional feature vector is input to the feature fusion module and concatenated with the temporal aggregated features output by the lightweight temporal attention module.

[0140] Multiple one-dimensional convolutional feature extraction modules extract respiratory rhythm features stepwise along the time dimension. The first convolutional module extracts local respiratory waveform features, the second learns mesoscale temporal variation features, and the third refines deep respiratory rhythm representations. A lightweight temporal attention module generates normalized weights in the time dimension and aggregates the temporal features based on these weights, adaptively focusing on effective respiratory phases and reducing the impact of occlusion, sudden body movements, and local abnormal segments on the prediction results. A feature fusion module fuses the temporal aggregated features output by the lightweight temporal attention module with a 32-dimensional feature vector, and a fully connected regression module maps the fused features to continuous respiratory rate prediction values.

[0141] Specifically, let the temporal features output by the one-dimensional convolutional feature extraction module be:

[0142]

[0143] in, Let represent the feature vector at time position t. The lightweight temporal attention module first calculates the attention score based on the features at each time position:

[0144]

[0145] Then, the time-dimensional attention weights are obtained through a normalization function:

[0146]

[0147] Finally, the time-dimensional features are weighted and aggregated:

[0148]

[0149] In the formula: and These are the learnable parameters for the attention module; Let represent the attention weight corresponding to the t-th time position; z represents the feature vector aggregated by the lightweight temporal attention module. Compared with global average pooling, this module can adaptively allocate weights according to the effectiveness of different time segments of the respiratory signal, improving the model's robustness to occlusion and non-respiratory perturbations.

[0150] Furthermore, the feature vector z output by the lightweight temporal attention module is concatenated with the 32-dimensional feature vector q to obtain the fused feature vector. :

[0151]

[0152] In the formula: This represents the 32-dimensional feature vector obtained from the filtered preprocessed respiratory signal and the robust frequency estimation module; [z;q] represents the vector concatenation operation. Then, the predicted respiratory rate is output through the fully connected regression module.

[0153]

[0154] In the formula: This represents the respiratory rate predicted by the model; This represents the nonlinear mapping function corresponding to the fully connected regression module.

[0155] In this embodiment, the training process of the one-dimensional convolutional neural network regression model containing a lightweight temporal attention module uses an error loss function L, preferably calculated using the following mean squared error loss function formula:

[0156]

[0157] In the formula: B represents the batch sample size; This represents the respiratory rate corresponding to the real or synthetic label of the i-th sample; The model predicts the respiratory rate; the mean squared error (MSE) is calculated to obtain the error loss function L, which measures the deviation between the predicted value and the true value, and is used to train the neural network to improve the prediction accuracy.

[0158] By minimizing the aforementioned loss function, the breathing frequency output by the model gradually approaches the breathing frequency corresponding to the real or synthetic label. After training, the model parameters are saved, and the trained one-dimensional convolutional neural network regression model with a lightweight temporal attention module is used as the pre-trained model in step S6 for predicting the breathing frequency of actual video sequences.

[0159] To improve the training performance of the respiratory rate prediction model, this embodiment employs a dataset construction method that combines real-labeled samples with synthetic samples. The dataset construction process includes two stages: real-labeled sample construction and synthetic sample construction.

[0160] First, a real-world labeled dataset was constructed. Video data from 32 sows in the farrowing house environment was collected, resulting in 2424 video sequences, each 15–30 seconds in length. The data was divided into training, validation, and test sets in an 8:1:1 ratio, with approximately 1939 videos in the training set, approximately 242 videos in the validation set, and approximately 243 videos in the test set.

[0161] To train the object detection model, bounding boxes were used to label the pigs in the video, forming object detection training data. Simultaneously, the actual number of breaths was counted for each video as the respiratory frequency label. Considering that manual counting is susceptible to observational errors and subjective factors, this embodiment uses three labelers to count the number of breaths, and the average of the three counts is taken as the true label for the video, thus improving the reliability of the manual labeling.

[0162] Since the distribution of samples with different breathing frequencies, noise intensities, and interference conditions in real video samples may be uneven, this embodiment further constructs synthetic breathing signal samples to expand the frequency range and interference types of the training samples. The synthetic breathing signal samples include one or more of the following: dominant frequency components, adjacent frequency confusion components, harmonic components, occlusion disturbance components, amplitude modulation components, and power frequency interference components.

[0163] By adjusting the dominant breathing frequency, harmonic intensity, noise level, occlusion disturbance ratio, and amplitude variation of the synthesized signal, breathing signals under different difficulty conditions can be simulated. During training, real labeled samples can be oversampled, and the oversampled real labeled samples can be mixed with synthetic samples for training to take into account both the distribution of real video data and the distribution of controllable interference samples, thereby improving the model's generalization ability.

[0164] By combining real labeled samples with synthetic samples, we can make full use of existing manually labeled data to ensure that the model training closely matches the actual application scenario. On the other hand, we can use synthetic samples to supplement training samples under different frequencies, noise levels, and interference conditions, thereby improving the model's adaptability to complex pigsty environments.

[0165] As an optional implementation, the neural network regression model can also employ a multilayer perceptron regression model. Correspondingly, the 32-dimensional feature vector can be used as the input to the multilayer perceptron regression model, outputting continuous respiratory rate prediction values. Preferably, the multilayer perceptron regression model includes an input layer, a first fully connected layer, a second fully connected layer, and an output layer; the input layer is used to input the 32-dimensional feature vector, the first fully connected layer contains 256 neurons, the second fully connected layer contains 128 neurons, and the output layer contains 1 neuron, used to output continuous respiratory rate prediction values. It should be noted that the multilayer perceptron regression model is an optional implementation of the present invention and is not the only limitation on the neural network regression model of the present invention.

[0166] The one-dimensional convolutional neural network regression model with a lightweight temporal attention module obtained through the above training process does not require the synthetic sample construction process during the actual deployment phase. It only needs to execute steps S1 to S6 sequentially to output the predicted respiratory rate value based on the video to be detected. It should be noted that the FFT main frequency estimation method, peak detection method, and autocorrelation analysis method in this embodiment can be used not only for feature construction during the training phase but also for robust initial frequency estimation during the actual deployment phase. The final respiratory rate prediction process is completed by the pre-trained one-dimensional convolutional neural network regression model with a lightweight temporal attention module.

[0167] Using MSE to obtain the error loss function L during training helps guide the network to continuously optimize prediction accuracy. For overall model evaluation, mean absolute error (MAE) and root mean square error (RMSE) can also be used to assess model accuracy; these error calculation methods are commonly used average metrics in this field.

[0168] Results based on the test set show that the proposed non-contact respiratory rate detection method for pigs achieves high accuracy on the test set, with an accuracy of 93.3% under a tolerance of ±3 BPM and 96.25% under a tolerance of ±4 BPM. The MAE (mean absolute error) is 1.20 BPM, the RMSE (root mean square error) is 2.03 BPM, the R² (coefficient of determination) is 0.9724, and the 95% consistency bound is [-4.16, 3.77] BPM, indicating good consistency between the model's predictions and the actual values. Overall, this method can accurately complete the task of estimating the respiratory rate of pigs, verifying the effectiveness of the one-dimensional convolutional neural network regression modeling scheme based on optical flow signal extraction within the central half-region (ROI), robust frequency estimation, and a lightweight temporal attention module.

[0169] This invention aims to achieve a complete detection process from video input and respiratory signal extraction to respiratory rate output in complex delivery room environments, and to comprehensively evaluate model performance using indicators such as MAE, RMSE, and tolerance accuracy. Figure 6 (As shown). Current experimental results indicate that the invention's objectives have been largely achieved: on the one hand, as... Figure 5 As shown, the model achieves an accuracy exceeding 90% within a tolerance of ±3 BPM, indicating its high practical applicability within an acceptable error range for real-world applications; on the other hand, as... Figure 7 As shown, the high correlation coefficient and low error index indicate that the model can not only distinguish whether it is close to the true value, but also make continuous numerical estimations more accurately.

[0170] The above descriptions are merely embodiments of the present invention. Commonly known technical knowledge in the solutions is not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the filing date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical well-known technologies should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several adjustments and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for monitoring respiratory rate in pigs based on target detection and optical flow analysis, characterized in that, The process includes the following steps: Step S1: Acquire video sequences of the pigs to be detected and extract consecutive video frames in chronological order; Step S2: Determine the target region of the pig in the video frames and construct a Region of Significant Interest (ROI) for breathing, including the chest and abdominal respiratory fluctuations, within the target region; Step S3: For the corresponding ROI in consecutive video frames, calculate the dense optical flow field of all pixels within the region and construct the region motion response value based on the optical flow amplitude of all pixels. Connect the region motion response values ​​at consecutive time points in chronological order to form a one-dimensional respiratory time-series signal; Step S4: Perform bandpass filtering on the one-dimensional respiratory time-series signal to obtain the respiratory signal within the respiratory frequency band; Step S5: Use a robust frequency estimation module to perform a preliminary estimation of the respiratory frequency of the respiratory signal to obtain a robust preliminary respiratory frequency estimate; Step S6: Extract features based on the respiratory signal and the robust preliminary respiratory frequency estimate, input the features into a pre-trained neural network regression model, and output the corresponding predicted video respiratory frequency value.

2. The method according to claim 1, characterized in that: In step S2, the video frame is input into the target detection model to obtain the overall target detection box of the pig. Based on the center point of the target detection box, the central area of ​​the target detection box is selected in the horizontal and vertical directions respectively. Preferably, the central half of the area of ​​the target detection box is selected as the region of significant interest (ROI) for breathing. The ROI for breathing is used to reduce the interference of pen edges, background changes, head swinging, tail swinging, limb movements and target box edge noise on the extraction of breathing signals, and to retain the main respiratory movement information corresponding to the mid-thigh and chest and abdomen undulations.

3. The method according to claim 1, characterized in that: In step S3, the dense optical flow vector between two consecutive frames is calculated for all pixels within the ROI. The corresponding optical flow amplitude is calculated based on the optical flow vector of each pixel, and the average value of the optical flow amplitude of all pixels within the ROI is used as the regional motion response value at the current time. The regional motion response values ​​at all times constitute a one-dimensional breathing time sequence signal in chronological order.

4. The method according to claim 1, characterized in that: The robust frequency estimation module in step S5 includes frequency domain estimation based on fast Fourier transform, time domain estimation based on peak detection, and period estimation based on autocorrelation analysis; fundamental and harmonic candidate analysis is performed on the frequency domain estimation results based on fast Fourier transform to obtain the frequency domain estimation results. The frequency domain estimation results, peak detection estimation results, and autocorrelation estimation results are fused for consistency. When the difference between any two estimation results does not exceed a preset consistency threshold, the average of the two consistent estimation results is taken as the initial estimate of the robust respiratory rate. When the three estimation results are inconsistent, the respiratory rate estimation result based on autocorrelation analysis shall be given priority as the robust initial estimate of respiratory rate; the preset consistency threshold is preferably 3 BPM to 5 BPM, and more preferably 5 BPM.

5. The method according to claim 1, characterized in that: In step S6, a 32-dimensional handcrafted feature vector is extracted based on the filtered respiratory signal and the robust respiratory frequency initial estimate. The 32-dimensional handcrafted feature vector includes time-domain statistical features, frequency-domain energy features, spectral peak features, autocorrelation features, and respiratory dominant frequency features. The respiratory dominant frequency features are provided by the robust respiratory frequency initial estimate output in step S5.

6. The method according to claim 5, characterized in that: The neural network regression model is a one-dimensional convolutional neural network regression model with a lightweight temporal attention module. A four-channel temporal feature is constructed based on the respiratory signal, including a normalized original respiratory signal channel, a first-order differential signal channel, a sliding mean square envelope signal channel, and a differential signal energy envelope signal channel. The one-dimensional convolutional neural network regression model with the lightweight temporal attention module includes an input layer, multiple one-dimensional convolutional feature extraction modules, a lightweight temporal attention module, a feature fusion module, and a fully connected regression module. The multiple one-dimensional convolutional feature extraction modules are used to extract local rhythmic features and deep temporal representations of the respiratory signal. The lightweight temporal attention module is used to generate normalized weights in the time dimension and perform weighted aggregation of time-dimensional features. The feature fusion module is used to fuse the aggregated temporal features with a 32-dimensional handcrafted feature vector. The fully connected regression module is used to output a continuous respiratory rate prediction value.

7. The method for monitoring pig respiratory rate based on target detection and optical flow analysis according to claim 6, characterized in that: The training samples of the neural network regression model include real labeled samples and synthetic samples. The real labeled samples are obtained by manually counting the number of breaths and converting them into respiratory frequency labels. The synthetic samples are controlled-difficulty synthetic respiratory signal samples, including one or more of the following: main frequency component, adjacent frequency confusion component, harmonic component, occlusion disturbance component, amplitude modulation component and power frequency interference component. The training process of the neural network regression model uses the mean squared error loss function L, with the following formula: In the formula, B represents the batch sample size; This represents the respiratory rate corresponding to the real or synthetic label of the i-th sample; This represents the respiratory rate predicted by the model.

8. The method according to claim 5, characterized in that: The neural network regression model can also employ a multilayer perceptron regression model, which takes the 32-dimensional handcrafted feature vector as input and outputs continuous respiratory rate predictions. Preferably, the multilayer perceptron regression model includes an input layer, a first fully connected layer, a second fully connected layer, and an output layer. The input layer is used to input the 32-dimensional handcrafted feature vector, the first fully connected layer contains 256 neurons, the second fully connected layer contains 128 neurons, and the output layer contains 1 neuron.

Images