Video pig respiration monitoring method and device based on time-frequency domain fusion

Abstract

The embodiment of the invention provides a video pig respiration monitoring method and device based on time-frequency domain fusion, and the method comprises the steps: locking a respiration-related region at the chest and abdomen of a pig through a target detection model, building a respiration time sequence signal based on an enhanced respiration movement signal after the weak movement signal related to the respiration in a detection frame is enhanced, and carrying out the detection of the respiration time sequence signal. Secondly, analyzing the respiratory signals in a time domain, obtaining respiratory frequency through peak detection, analyzing the respiratory signals in a frequency domain, and extracting respiratory dominant frequency through Fourier transform; and performing fusion analysis on time domain and frequency domain results to obtain an accurate respiratory rate, further judging whether the respiration of the pig is abnormal according to the respiratory rate, and triggering an alarm under an abnormal condition. According to the method, the problems that single time domain analysis is easily interfered by burrs and single frequency domain analysis is insensitive to local changes can be solved, the accuracy and robustness of pig respiration monitoring are remarkably improved, and the method is suitable for non-contact health monitoring and management in large-scale farms.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a video-based method and device for monitoring pig respiration based on time-frequency domain fusion. Background Technology

[0002] In the livestock industry, pig health management is crucial for improving farming efficiency and reducing disease incidence. Traditional pig health monitoring mainly relies on manual observation or physiological testing equipment. These methods are not only labor-intensive but may also suffer from problems such as untimely or unsustainable monitoring. Real-time, efficient, and accurate health monitoring is particularly important in large-scale farms or highly automated environments.

[0003] With the rapid development of artificial intelligence and computer vision technologies, video-based pig health monitoring has become an emerging technological approach. Acquiring dynamic information about pigs through video surveillance equipment and analyzing it using image processing and deep learning algorithms not only enables continuous monitoring of pig behavior but also reduces the burden of manual monitoring to some extent. However, video-based monitoring methods also face some technical challenges. For example, when monitoring pig respiration, overcoming the susceptibility of single-time-domain analysis to glitches and the insensitivity of single-frequency-domain analysis to local changes are technical difficulties faced by researchers.

[0004] Therefore, due to the accuracy limitations of video monitoring, existing technologies still largely rely on traditional sensors or manual observation to obtain respiratory information. However, these methods are limited by monitoring range and real-time requirements, making it difficult to meet the needs of large-scale farms for real-time monitoring and accurate diagnosis. Therefore, developing a video monitoring method that can automatically stabilize video, accurately detect the chest and abdomen of pigs, and extract respiratory signals has become a current research and application hotspot. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides a video-based method and device for monitoring pig respiration based on time-frequency domain fusion, so as to improve the accuracy and robustness of pig respiration monitoring and provide a technical solution for non-contact health monitoring and management in large-scale farms.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides a video-based method for monitoring pig respiration based on time-frequency domain fusion, comprising:

[0008] The video of pigs was captured using camera equipment and the images were deduplicated. The coordinates of the detection box of the chest and abdomen of the pigs in the video sequence were obtained to locate the respiratory-related region. The respiratory motion signal of the respiratory-related region in the video sequence was enhanced, and a respiratory time sequence signal was constructed based on the enhanced respiratory motion signal.

[0009] Peak values ​​are monitored in the respiratory time sequence signal, and the height difference between each peak and its adjacent left and right valleys is calculated. Invalid peak values ​​with height differences less than a preset threshold are removed, and the time-domain respiratory frequency is calculated based on the remaining valid peak values. The respiratory time sequence signal is subjected to Fourier transform to obtain the frequency domain signal, and the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum is taken as the frequency domain respiratory frequency.

[0010] The final respiratory rate of the pig is obtained by weighted fusion of the time-domain respiratory rate and the frequency-domain respiratory rate. When the final respiratory rate of the pig exceeds the preset range, the pig's breathing is judged to be abnormal, and an alarm is triggered or a manual review is prompted.

[0011] Furthermore, the de-jitter processing steps include:

[0012] Keypoint detection algorithms were used to identify keypoints in each frame of the pig video.

[0013] Motion estimation is performed on each frame of the pig video, and the motion vector of each pixel is calculated.

[0014] The position of key points is compensated based on motion vectors, thereby eliminating video jitter.

[0015] Furthermore, the step of obtaining the coordinates of the detection box of the pig's chest and abdomen in the video sequence and locking the respiratory-related region includes:

[0016] The YOLOX object detection model was used to identify pig targets in the video;

[0017] Next, based on the pig's body structure, a deep learning model was used to identify the chest and abdominal region of the pig; and a motion tracking algorithm was used to track the positional changes of the target region in consecutive frames.

[0018] Furthermore, the step of enhancing the respiratory motion signal in the respiratory-related region of the video sequence includes:

[0019] The Euler amplification algorithm is used to process the breathing motion signal in the video sequence;

[0020] The video after Euler amplification is converted into a grayscale frame sequence. The change in each frame is obtained by frame-by-frame difference, and these difference results are accumulated to form a cumulative difference map.

[0021] The cumulative difference image is normalized by mapping the pixel values ​​to a preset range to obtain a normalized image.

[0022] Thresholding is performed on the normalized image to obtain a binarized breathing region mask. In this mask, regions with a value of 1 represent regions with significant breathing movements, and regions with a value of 0 represent background regions.

[0023] Furthermore, the step of constructing a respiratory timing signal based on the enhanced respiratory motion signal includes:

[0024] Based on the binary respiratory motion salient region mask, the set of pixels within the salient region and the set of pixels outside the salient region are extracted in each frame of grayscale image;

[0025] Calculate the mean values ​​of pixels within and outside the salient region, respectively;

[0026] A respiratory time-series signal that varies over time is constructed by calculating the difference between the mean pixel value outside the salient region and the mean pixel value within the salient region.

[0027] The obtained respiratory timing signal is filtered by moving average to finally output a smooth respiratory timing signal.

[0028] Furthermore, the step of monitoring the peak values ​​of the respiratory timing signal, calculating the height difference between each peak value and its adjacent left and right trough values, removing invalid peak values ​​with a height difference less than a preset threshold, and calculating the time-domain respiratory rate based on the remaining valid peak values ​​includes:

[0029] Peak and valley identification is performed on the respiratory time sequence signal to obtain the set of peak values ​​and the set of valley values;

[0030] For each peak, find its nearest neighboring valleys to the left and right, and calculate the height difference between the peak and its neighboring valleys.

[0031] Calculate the average height difference of all peaks and remove false peaks whose height difference is less than a preset threshold to obtain the set of valid peaks;

[0032] The number of effective peak values ​​was counted, and the temporal breathing frequency was calculated based on the video duration.

[0033] The step of performing a Fourier transform on the respiratory time sequence signal to obtain a frequency domain signal, and using the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency, includes:

[0034] The frequency domain representation of the respiratory timing signal is obtained by performing a Fourier transform on the signal.

[0035] Calculate the amplitude spectrum in the frequency domain and find the frequency corresponding to the maximum amplitude within the positive frequency range;

[0036] The frequency with the largest amplitude in the spectrum is used as the breathing frequency for frequency domain estimation.

[0037] Furthermore, the step of determining that the pig's breathing is abnormal and triggering an alarm or prompting manual review when the final respiratory rate of the pig exceeds the preset range includes:

[0038] If the difference between the time-domain respiratory rate and the frequency-domain respiratory rate exceeds a preset threshold, a manual review will be prompted.

[0039] The final respiratory rate is compared with the preset physiological range for pigs. If the respiratory rate exceeds the range, it is judged as abnormal breathing and an alarm is triggered.

[0040] Secondly, this application provides a video-based pig respiratory monitoring device based on time-frequency domain fusion, comprising:

[0041] The data processing module is used to acquire videos of pigs using camera equipment and perform shading processing, obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence, and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence, and construct a respiratory time sequence signal based on the enhanced respiratory motion signal;

[0042] The data conversion module is used to monitor the peak values ​​of the respiratory time sequence signal, calculate the height difference between each peak value and its adjacent left and right valley values, remove invalid peak values ​​with a height difference less than a preset threshold, calculate the time-domain respiratory frequency based on the remaining valid peak values, perform Fourier transform on the respiratory time sequence signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0043] The result output module is used to perform weighted fusion of the time-domain respiratory frequency and the frequency-domain respiratory frequency to obtain the final respiratory frequency of the pig. When the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is judged to be abnormal, and an alarm is triggered or a manual review is prompted.

[0044] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the video pig respiration monitoring method based on time-frequency domain fusion.

[0045] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video pig respiration monitoring method based on time-frequency domain fusion.

[0046] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the video pig respiration monitoring method based on time-frequency domain fusion.

[0047] As can be seen from the above technical solution, this application provides a video-based method and device for monitoring pig respiration based on time-frequency domain fusion. It utilizes a target detection model to locate the respiration-related region on the pig's chest and abdomen. After enhancing weak respiration-related motion signals within the detection frame, a respiration time-series signal is constructed based on the enhanced respiration motion signal. The respiration signal is then analyzed in the time domain, obtaining the respiration frequency through peak detection. In the frequency domain, the respiration signal is analyzed, extracting the dominant respiration frequency through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain a precise respiration frequency. Further, the respiration frequency is used to determine whether the pig's respiration is abnormal, triggering an alarm in case of abnormality. This invention overcomes the problems of single time-domain analysis being susceptible to glitches and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiration monitoring. It is suitable for non-contact health monitoring and management in large-scale farms. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the video pig respiration monitoring method based on time-frequency domain fusion in the embodiments of this application;

[0050] Figure 2 This is a structural diagram of the video pig respiratory monitoring device based on time-frequency domain fusion in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0052] Figure label:

[0053] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0056] In view of the problems existing in the prior art, this application provides a video pig respiratory monitoring method and device based on time-frequency domain fusion. The method uses a target detection model to locate the respiratory-related region in the chest and abdomen of the pig. After enhancing the weak respiratory-related motion signals within the detection frame, a respiratory time-series signal is constructed based on the enhanced respiratory motion signals. The respiratory signal is then analyzed in the time domain, and the respiratory frequency is obtained through peak detection. In the frequency domain, the respiratory signal is analyzed, and the dominant respiratory frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain a precise respiratory frequency. Further, the respiratory frequency is used to determine whether the pig's breathing is abnormal, and an alarm is triggered in case of abnormality. This invention overcomes the problems of single time-domain analysis being susceptible to glitches and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiratory monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0057] To improve the accuracy and robustness of pig respiratory monitoring and provide a technical solution for non-contact health monitoring and management in large-scale farms, this application provides an embodiment of a video-based pig respiratory monitoring method based on time-frequency domain fusion. See [link to relevant documentation]. Figure 1 The video-based pig respiration monitoring method based on time-frequency domain fusion specifically includes the following:

[0058] Step S101: Use a camera device to capture videos of pigs and perform shading processing to obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence, and construct a respiratory time sequence signal based on the enhanced respiratory motion signal.

[0059] Optionally, in this embodiment, the open-source video stabilization algorithm DUT-RAFT can be used to perform shake reduction processing on the video. This algorithm includes key point detection, motion estimation and smoothing modules, which can process the input shaking video into a stable video to eliminate the interference of camera shake on subsequent breathing signal analysis.

[0060] Optionally, the shading process includes the following steps:

[0061] Deep learning models, especially keypoint detection algorithms, are used to detect keypoints in each frame of the image to identify stable and moving regions in the video, providing basic data for subsequent motion estimation.

[0062] Motion estimation is performed on each frame of the video, and the motion vector of each pixel is calculated. These vectors represent the direction and magnitude of pixel motion between adjacent frames, providing motion information for the shading process.

[0063] After motion estimation, motion vectors are applied to keypoints in the video sequence. By performing displacement compensation on each frame and aligning each frame with the previous frame, moving parts in the video are restored to their true motion trajectories, eliminating short-term motion fluctuations caused by device jitter and resulting in more stable video output. Displacement compensation can be achieved through various methods, such as time-domain weighted averaging or Kalman filtering.

[0064] Ultimately, the video processed by deep learning algorithms will remove the original jitter, presenting a stable image sequence. This shatter-free video data will be used for subsequent analysis of pig respiratory signals, preventing jitter from affecting the extraction and analysis results.

[0065] Optionally, in this embodiment, the target region detection of the abdomen can use the target detection model YOLOX to obtain the coordinates of the detection box of the pig's chest and abdomen, so as to lock the breathing-related area and avoid interference from the movement of unrelated parts on the extraction of respiratory signals.

[0066] Optionally, the method for locking the respiratory-related area includes:

[0067] Step 1: Extract each frame of the video sequence to provide data input for subsequent object detection.

[0068] Step 2: Preprocess each frame of the image. This includes: converting it to grayscale to reduce the complexity of color information and improve processing speed; applying Gaussian filtering or other denoising methods to reduce noise in the video, especially when the background is complex, and removing interference.

[0069] Step 3: The YOLOX object detection model is used to detect pigs in the video frames. YOLOX is a high-efficiency object detection model capable of detecting multiple targets in real-time processing. The YOLOX model can identify all possible pig targets in the video frames. This model uses a convolutional neural network to extract features and can regress the bounding box and class probability of the target in each frame.

[0070] Step 4: Based on the target areas of pigs detected by the YOLOX model, further filter out the abdominal region.

[0071] Within the bounding box of object detection, the overall location of the pig can usually be obtained. Then, based on the pig's body structure, a deep learning model is used to further identify the abdominal region. Region regression can be used to train the model to more accurately identify the chest and abdominal regions, especially when the pig is at different angles and distances in the video.

[0072] Step 5: Track the positional changes of the target region across consecutive frames through time-series tracking to avoid detection errors caused by background changes or rapid motion. Motion tracking algorithms such as Kalman filtering or optical flow can be used to correlate the target detection result with the detection box of the previous frame, improving detection stability across consecutive frames.

[0073] Step Six: When locating the abdominal region, the model threshold can be adjusted according to the actual situation to cope with the impact of changes in pig posture, lighting conditions and video quality on the model detection accuracy.

[0074] Step 7: The detected abdominal target region will be labeled and output as input data for subsequent respiratory signal extraction and analysis. The accuracy of this region is particularly important for subsequent analysis. Ensuring that the model can accurately locate the abdominal region helps reduce errors and improve the accuracy of subsequent respiratory signal analysis.

[0075] Optionally, in this embodiment, the respiratory motion signal enhancement uses the Euler amplification algorithm to amplify the weak respiratory motion signal within the chest and abdomen detection box, thereby highlighting respiratory-related dynamic features and improving the detectability of the respiratory signal. The acquisition of the significant respiratory motion region includes the following steps:

[0076] Step 1: Convert the video after Euler upscaling into a grayscale frame sequence, denoted as F. t (x,y), t=1,2,…,T, where (x,y) represents pixel coordinates, obtained by frame-by-frame differencing.

[0077] D t (x,y)=|F t (x,y)-F t-1 (x,y)|

[0078] The difference results from each frame are then summed to form a cumulative difference map.

[0079]

[0080] Step 2: Normalize the cumulative difference image S(x,y) by mapping the pixel values ​​to the interval [0,255] to obtain the normalized image.

[0081]

[0082] Step 3: Perform thresholding on the normalized image to obtain a binary breathing region mask M(x,y), which is defined as:

[0083]

[0084] The regions with a value of 1 correspond to regions of significant respiratory movement, while the regions with a value of 0 correspond to background regions.

[0085] Optionally, in this embodiment, a respiratory time-series signal is constructed based on the enhanced regional changes, and the time-series signal can be filtered and smoothed to obtain a stable respiratory signal sequence. The extraction of this respiratory time-series signal includes the following steps:

[0086] Step 1: Based on the binary respiratory motion salient region mask M(x,y) obtained in the previous step, in each frame of grayscale image F t Extract the pixel set Ω from the salient region in (x,y) respectively. in and the set of pixels outside the salient region Ω out ,in:

[0087] Ω_in={(x,y)|M(x,y)=1}

[0088] Ω_out = {(x,y)|M(x,y) = 0}

[0089] Step 2: Calculate the pixel mean within and outside the region respectively:

[0090]

[0091] Step 3: Construct a differential signal by subtracting the mean pixel value within the region from the mean pixel value outside the region to obtain the respiratory time-series signal that varies over time.

[0092] S(t)=μ out (t)-μ in (t), t=1,2,…,T

[0093] Step 4: Apply a moving average filter to the respiratory timing signal S(t) to achieve smoothing, and obtain the final smoothed respiratory signal for subsequent time and frequency domain analysis.

[0094] Step S102: Perform peak monitoring on the respiratory timing signal and calculate the height difference between each peak and its adjacent left and right valleys. Remove invalid peaks with height differences less than a preset threshold. Calculate the time-domain respiratory frequency based on the remaining valid peaks. Perform Fourier transform on the respiratory timing signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0095] Optionally, in this embodiment, the extraction of time-domain respiratory frequency includes the following steps:

[0096] Step 1: Perform peak and trough identification on the respiratory time series signal S(t) obtained in step S101, t=1,2,…,T, to obtain the set of peak values ​​denoted as P and the set of trough values ​​denoted as V:

[0097] P={t|S(t)>S(t-1),S(t)>S(t+1)}

[0098] V={t|S(t)<S(t-1),S(t)<S(t+1)}

[0099] Step 2: For each peak P i ∈P, find its left and right nearest neighboring valley values ​​v i-1 ,v i+1 ∈V, calculate the height difference between the peak value and the left and right valley values:

[0100]

[0101] The minimum value is taken as the effective height difference of the peak:

[0102]

[0103] Step 3: Calculate the average height difference β of all peaks:

[0104]

[0105] Then, spurious peaks with height differences less than β / 2 are removed to obtain the effective peak set P*.

[0106] Step 4: Count the number of valid peak values ​​|P*|, then the number of breaths corresponding to the entire video duration is:

[0107] N resp =|P * |

[0108] Combined with video length Calculate the time-domain respiratory rate:

[0109]

[0110] Optionally, in this embodiment, the frequency domain respiratory frequency extraction includes the following steps:

[0111] Step 1: Perform a Fourier transform on the respiratory time-series signal S(t) obtained in step S101, t = 1, 2, ..., T, to obtain its frequency domain representation:

[0112]

[0113] Where T is the length of the timing signal. This represents the sampling time interval.

[0114] Step 2: Calculate the frequency domain amplitude spectrum:

[0115] A(f)=|S(f)|

[0116] and in the positive frequency range Find the frequency corresponding to the maximum amplitude within the range:

[0117] f peak =argmaxA(f)

[0118] Step 3: Set the peak frequency f peak The respiratory rate as a frequency domain estimate is:

[0119] f freq =f peak

[0120] Step S103: The time-domain respiratory frequency and the frequency-domain respiratory frequency are weighted and fused to obtain the final respiratory frequency of the pig; when the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is determined to be abnormal, and an alarm is triggered or a manual review is prompted.

[0121] Optionally, in this embodiment, respiratory rate fusion and abnormality detection includes the following steps:

[0122] Step 1: Calculate the time-domain respiratory rate f obtained in step S102. time and frequency domain respiratory rate f peak Weighted fusion is performed to obtain the final respiratory rate:

[0123] f final =α*f time +(1-α)*f freq

[0124] Where α∈[0,1] are weighting coefficients used to balance the time-domain and frequency-domain estimation results.

[0125] Step 2: To avoid frequency distortion due to noise affecting a single method, if the difference between the breathing frequency in the time domain and the frequency domain exceeds the preset threshold ε, a manual review will be prompted.

[0126] Step 3: Set the final respiratory rate f final [f] min ,f max Compare:

[0127] If f min <f final <f max If so, then breathing is considered normal;

[0128] If f final <fmin or f fina >f max If this occurs, it is determined to be abnormal breathing, triggering an alarm.

[0129] As described above, the video pig respiration monitoring method based on time-frequency domain fusion provided in this application can use a target detection model to locate the respiration-related area in the chest and abdomen of the pig. After enhancing the weak motion signals related to respiration within the detection frame, a respiration time-series signal is constructed based on the enhanced respiration motion signal. Then, the respiration signal is analyzed in the time domain, and the respiration frequency is obtained through peak detection. The respiration signal is also analyzed in the frequency domain, and the respiration frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain an accurate respiration frequency. Furthermore, the respiration frequency is used to determine whether the pig's respiration is abnormal, and an alarm is triggered in case of abnormality. The method of this invention can overcome the problems of single time-domain analysis being susceptible to glitch interference and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiration monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0130] To improve the accuracy and robustness of pig respiratory monitoring and provide a technical solution for non-contact health monitoring and management in large-scale farms, this application provides an embodiment of a time-frequency domain fusion-based video pig respiratory monitoring device for implementing all or part of the aforementioned time-frequency domain fusion-based video pig respiratory monitoring method. See [link to embodiment]. Figure 2 The video-based pig respiration monitoring device based on time-frequency domain fusion specifically includes the following components:

[0131] Data processing module 10 is used to acquire pig videos using camera equipment and perform shading processing, obtain the detection box coordinates of the pig's chest and abdomen in the video sequence, lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence, and construct a respiratory time sequence signal based on the enhanced respiratory motion signal;

[0132] The data conversion module 20 is used to monitor the peak values ​​of the respiratory time sequence signal, calculate the height difference between each peak value and its adjacent left and right valley values, remove invalid peak values ​​with a height difference less than a preset threshold, calculate the time-domain respiratory frequency based on the remaining valid peak values, perform Fourier transform on the respiratory time sequence signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0133] The result output module 30 is used to perform weighted fusion of the time-domain respiratory frequency and the frequency-domain respiratory frequency to obtain the final respiratory frequency of the pig; when the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is determined to be abnormal, and an alarm is triggered or a manual review is prompted.

[0134] As described above, the video pig respiration monitoring device based on time-frequency domain fusion provided in this application can use a target detection model to locate the respiration-related area in the chest and abdomen of the pig. After enhancing the weak motion signals related to respiration within the detection frame, a respiration time-series signal is constructed based on the enhanced respiration motion signal. Then, the respiration signal is analyzed in the time domain, and the respiration frequency is obtained through peak detection. The respiration signal is also analyzed in the frequency domain, and the respiration frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain an accurate respiration frequency. Furthermore, the respiration frequency is used to determine whether the pig's respiration is abnormal, and an alarm is triggered in case of abnormality. The method of this invention can overcome the problems of single time-domain analysis being susceptible to glitch interference and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiration monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0135] From a hardware perspective, in order to improve the accuracy and robustness of pig respiratory monitoring and provide a technical solution for non-contact health monitoring and management in large-scale farms, this application provides an embodiment of an electronic device for implementing all or part of the video pig respiratory monitoring method based on time-frequency domain fusion. The electronic device specifically includes the following components:

[0136] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the time-frequency domain fusion-based video pig respiration monitoring device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the time-frequency domain fusion-based video pig respiration monitoring method and the embodiment of the time-frequency domain fusion-based video pig respiration monitoring device, the content of which is incorporated herein, and repeated details will not be described again.

[0137] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0138] In practical applications, parts of the video-based pig respiration monitoring method based on time-frequency domain fusion can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0139] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0140] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0141] In one embodiment, the video-based pig respiration monitoring method based on time-frequency domain fusion can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0142] Step S101: Use a camera device to capture videos of pigs and perform shading processing to obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence and construct a respiratory time sequence signal based on the enhanced respiratory motion signal;

[0143] Step S102: Perform peak monitoring on the respiratory timing signal and calculate the height difference between each peak and its adjacent left and right valleys. Remove invalid peaks with height differences less than a preset threshold. Calculate the time-domain respiratory frequency based on the remaining valid peaks. Perform Fourier transform on the respiratory timing signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0144] Step S103: The time-domain respiratory frequency and the frequency-domain respiratory frequency are weighted and fused to obtain the final respiratory frequency of the pig; when the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is determined to be abnormal, and an alarm is triggered or a manual review is prompted.

[0145] As described above, the electronic device provided in this application uses a target detection model to locate the respiratory-related region on the chest and abdomen of a pig. After enhancing the weak respiratory-related motion signals within the detection frame, a respiratory time-series signal is constructed based on the enhanced respiratory motion signals. The respiratory signal is then analyzed in the time domain, and the respiratory frequency is obtained through peak detection. In the frequency domain, the respiratory signal is analyzed, and the dominant respiratory frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain a precise respiratory frequency. Furthermore, the respiratory frequency is used to determine whether the pig's breathing is abnormal, and an alarm is triggered in case of abnormality. The method of this invention can overcome the problems of single time-domain analysis being susceptible to glitch interference and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiratory monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0146] In another embodiment, the video pig respiration monitoring device based on time-frequency domain fusion can be configured separately from the central processing unit 9100. For example, the video pig respiration monitoring device based on time-frequency domain fusion can be configured as a chip connected to the central processing unit 9100, and the function of the video pig respiration monitoring method based on time-frequency domain fusion can be realized through the control of the central processing unit.

[0147] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0148] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0149] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0150] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0151] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0152] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0153] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0154] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0155] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the time-frequency domain fusion-based video pig respiration monitoring method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the time-frequency domain fusion-based video pig respiration monitoring method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0156] Step S101: Use a camera device to capture videos of pigs and perform shading processing to obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence and construct a respiratory time sequence signal based on the enhanced respiratory motion signal;

[0157] Step S102: Perform peak monitoring on the respiratory timing signal and calculate the height difference between each peak and its adjacent left and right valleys. Remove invalid peaks with height differences less than a preset threshold. Calculate the time-domain respiratory frequency based on the remaining valid peaks. Perform Fourier transform on the respiratory timing signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0158] Step S103: The time-domain respiratory frequency and the frequency-domain respiratory frequency are weighted and fused to obtain the final respiratory frequency of the pig; when the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is determined to be abnormal, and an alarm is triggered or a manual review is prompted.

[0159] As described above, the computer-readable storage medium provided in this application uses a target detection model to locate the respiratory-related region in the chest and abdomen of a pig. After enhancing the weak respiratory-related motion signals within the detection frame, a respiratory time-series signal is constructed based on the enhanced respiratory motion signals. The respiratory signal is then analyzed in the time domain, and the respiratory frequency is obtained through peak detection. In the frequency domain, the respiratory signal is analyzed, and the dominant respiratory frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain a precise respiratory frequency. Further, the respiratory frequency is used to determine whether the pig's breathing is abnormal, and an alarm is triggered in case of abnormality. This invention overcomes the problems of single-time-domain analysis being susceptible to glitches and single-frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiratory monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0160] Embodiments of this application also provide a computer program product capable of implementing all steps of the time-frequency domain fusion-based video pig respiration monitoring method described above, where the execution subject is a server or client. When executed by a processor, this computer program / instruction implements the steps of the time-frequency domain fusion-based video pig respiration monitoring method. For example, the computer program / instruction implements the following steps:

[0161] Step S101: Use a camera device to capture videos of pigs and perform shading processing to obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence and construct a respiratory time sequence signal based on the enhanced respiratory motion signal;

[0162] Step S102: Perform peak monitoring on the respiratory timing signal and calculate the height difference between each peak and its adjacent left and right valleys. Remove invalid peaks with height differences less than a preset threshold. Calculate the time-domain respiratory frequency based on the remaining valid peaks. Perform Fourier transform on the respiratory timing signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency.

[0163] Step S103: The time-domain respiratory frequency and the frequency-domain respiratory frequency are weighted and fused to obtain the final respiratory frequency of the pig; when the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is determined to be abnormal, and an alarm is triggered or a manual review is prompted.

[0164] As described above, the computer program product provided in this application uses a target detection model to locate the respiratory-related area on the chest and abdomen of a pig. After enhancing the weak respiratory-related motion signals within the detection frame, a respiratory time-series signal is constructed based on the enhanced respiratory motion signals. Then, the respiratory signal is analyzed in the time domain, and the respiratory frequency is obtained through peak detection. In the frequency domain, the respiratory signal is analyzed, and the respiratory dominant frequency is extracted through Fourier transform. The time-domain and frequency-domain results are fused and analyzed to obtain a precise respiratory frequency. Furthermore, the respiratory frequency is used to determine whether the pig's breathing is abnormal, and an alarm is triggered in case of abnormality. The method of this invention can overcome the problems of single time-domain analysis being susceptible to glitch interference and single frequency-domain analysis being insensitive to local changes, significantly improving the accuracy and robustness of pig respiratory monitoring. It is suitable for non-contact health monitoring and management in large-scale farms.

[0165] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A video-based method for monitoring pig respiration based on time-frequency domain fusion, characterized in that, The method includes: The video of pigs was captured using camera equipment and the images were deduplicated. The coordinates of the detection box of the chest and abdomen of the pigs in the video sequence were obtained to locate the respiratory-related region. The respiratory motion signal of the respiratory-related region in the video sequence was enhanced, and a respiratory time sequence signal was constructed based on the enhanced respiratory motion signal. Peak values ​​are monitored in the respiratory time sequence signal, and the height difference between each peak and its adjacent left and right valleys is calculated. Invalid peak values ​​with height differences less than a preset threshold are removed, and the time-domain respiratory frequency is calculated based on the remaining valid peak values. The respiratory time sequence signal is subjected to Fourier transform to obtain the frequency domain signal, and the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum is taken as the frequency domain respiratory frequency. The final respiratory rate of the pig is obtained by weighted fusion of the time-domain respiratory rate and the frequency-domain respiratory rate. When the final respiratory rate of the pig exceeds the preset range, the pig's breathing is judged to be abnormal, and an alarm is triggered or a manual review is prompted.

2. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 1, characterized in that, The de-jitter processing steps include: Keypoint detection algorithms were used to identify keypoints in each frame of the pig video. Motion estimation is performed on each frame of the pig video, and the motion vector of each pixel is calculated. The position of key points is compensated based on motion vectors, thereby eliminating video jitter.

3. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 1, characterized in that, The step of obtaining the coordinates of the detection box of the pig's chest and abdomen in the video sequence and locking the respiratory-related region includes: The YOLOX object detection model was used to identify pig targets in the video; Next, based on the pig's body structure, a deep learning model was used to identify the chest and abdominal region of the pig; and a motion tracking algorithm was used to track the positional changes of the target region in consecutive frames.

4. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 1, characterized in that, The step of enhancing the respiratory motion signal in the respiratory-related region of the video sequence includes: The Euler amplification algorithm is used to process the breathing motion signal in the video sequence; The video after Euler amplification is converted into a grayscale frame sequence. The change in each frame is obtained by frame-by-frame difference, and these difference results are accumulated to form a cumulative difference map. The cumulative difference image is normalized by mapping the pixel values ​​to a preset range to obtain a normalized image. Thresholding is performed on the normalized image to obtain a binarized breathing region mask. In this mask, regions with a value of 1 represent regions with significant breathing movements, and regions with a value of 0 represent background regions.

5. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 4, characterized in that, The step of constructing a respiratory time-series signal based on the enhanced respiratory motion signal includes: Based on the binary respiratory motion salient region mask, the set of pixels within the salient region and the set of pixels outside the salient region are extracted in each frame of grayscale image; Calculate the mean values ​​of pixels within and outside the salient region, respectively; A respiratory time-series signal that varies over time is constructed by calculating the difference between the mean pixel value outside the salient region and the mean pixel value within the salient region. The obtained respiratory timing signal is filtered by moving average to finally output a smooth respiratory timing signal.

6. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 1, characterized in that, The steps of monitoring peak values ​​in the respiratory time-series signal, calculating the height difference between each peak and its adjacent left and right troughs, removing invalid peak values ​​with height differences less than a preset threshold, and calculating the time-domain respiratory rate based on the remaining valid peak values ​​include: Peak and valley identification is performed on the respiratory time sequence signal to obtain the set of peak values ​​and the set of valley values; For each peak, find its nearest neighboring valleys to the left and right, and calculate the height difference between the peak and its neighboring valleys. Calculate the average height difference of all peaks and remove false peaks whose height difference is less than a preset threshold to obtain the set of valid peaks; The number of effective peak values ​​was counted, and the temporal breathing frequency was calculated based on the video duration. The step of performing a Fourier transform on the respiratory time sequence signal to obtain a frequency domain signal, and using the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency, includes: The frequency domain representation of the respiratory timing signal is obtained by performing a Fourier transform on the signal. Calculate the amplitude spectrum in the frequency domain and find the frequency corresponding to the maximum amplitude within the positive frequency range; The frequency with the largest amplitude in the spectrum is used as the breathing frequency for frequency domain estimation.

7. The video-based pig respiration monitoring method based on time-frequency domain fusion according to claim 1, characterized in that, The step of determining that the pig's breathing is abnormal and triggering an alarm or prompting manual review when the final respiratory rate of the pig exceeds the preset range includes: If the difference between the time-domain respiratory rate and the frequency-domain respiratory rate exceeds a preset threshold, a manual review will be prompted. The final respiratory rate is compared with the preset physiological range for pigs. If the respiratory rate exceeds the range, it is judged as abnormal breathing and an alarm is triggered.

8. A video-based pig respiration monitoring device based on time-frequency domain fusion, characterized in that, The device includes: The data processing module is used to acquire videos of pigs using camera equipment and perform shading processing, obtain the coordinates of the detection box of the pig's chest and abdomen in the video sequence, and lock the breathing-related area; enhance the respiratory motion signal of the breathing-related area in the video sequence, and construct a respiratory time sequence signal based on the enhanced respiratory motion signal; The data conversion module is used to monitor the peak values ​​of the respiratory time sequence signal, calculate the height difference between each peak value and its adjacent left and right valley values, remove invalid peak values ​​with a height difference less than a preset threshold, calculate the time-domain respiratory frequency based on the remaining valid peak values, perform Fourier transform on the respiratory time sequence signal to obtain the frequency domain signal, and take the frequency corresponding to the maximum value of the frequency domain signal amplitude spectrum as the frequency domain respiratory frequency. The result output module is used to perform weighted fusion of the time-domain respiratory frequency and the frequency-domain respiratory frequency to obtain the final respiratory frequency of the pig. When the final respiratory frequency of the pig exceeds the preset range, the pig's breathing is judged to be abnormal, and an alarm is triggered or a manual review is prompted.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the video pig respiration monitoring method based on time-frequency domain fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the video pig respiration monitoring method based on time-frequency domain fusion as described in any one of claims 1 to 7.

Images