A multi-camera based animal vital sign monitoring method, medium, device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF GEOSCIENCES (WUHAN)
- Filing Date
- 2025-02-12
- Publication Date
- 2026-07-24
AI Technical Summary
Existing methods for monitoring animal vital signs suffer from problems such as excessive human intervention, highly invasive equipment, low accuracy, and unsuitability for nighttime detection. In particular, methods based on infrared thermal imaging, radar, photoplethysmography, and computer vision are deficient in terms of accuracy and reliability.
A multi-camera system was used in conjunction with the DeepLabCut pose estimation network to acquire key point information of animals through multi-view videos, analyze the center of mass motion, detect corner points and perform correlation analysis, and use power spectrum analysis to obtain respiratory rate and heart rate.
It reduces human intervention, improves monitoring accuracy and reliability, and can accurately acquire animal vital signs signals in various environments, reducing the impact of the equipment on animals.
Smart Images
Figure CN120078379B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of animal vital sign monitoring technology, and in particular to a method, medium, and device for monitoring animal vital signs based on multiple cameras. Background Technology
[0002] Animal experimental research has always been key to scientific breakthroughs in the fields of medicine and biology. Heart rate and respiratory rate are the most important and fundamental physiological parameters characterizing the metabolic level of animals, and the data obtained can detect and highly predict dynamic clinical trial outcomes. Medically, it has been found that respiratory rate is closely related to changes in some blood biochemical parameters, potentially affecting blood oxygen saturation, carbon dioxide partial pressure, and blood oxygen content.
[0003] Previously, respiratory rate was typically measured manually in many experimental settings. This method is subjective and not a reliable measurement method. With the advent of automation technology, various long-distance telemetry technologies have been applied to animal vital sign monitoring, such as invasive and non-invasive contact devices. Implanting expensive devices or performing potentially risky physical interventions can cause stress and long-term trauma to animals, severely affecting their physiological state during both immediate and long-term monitoring. The installation of such monitoring devices may also increase the risk of infectious disease transmission between humans and animals or between laboratory animals. Contact wearable devices can also cause discomfort, skin damage, and infection risks in laboratory animals. Therefore, non-contact systems for tracking and acquiring animal vital sign data have attracted significant attention.
[0004] Non-contact methods, such as infrared thermal imaging, suffer from insufficient image detail and poor resolution, resulting in relatively low accuracy of acquired vital sign data. While radar-based methods offer strong penetration and accurate measurements, long-term radiation exposure may be harmful to animals, and they are susceptible to electromagnetic interference. Imaging photoplethysmography (IPPG) is affected by lighting conditions and cannot detect at night. Computer vision-based monitoring methods require extracting specific information from images for subsequent monitoring and diagnosis; however, current information extraction accuracy still lags behind manual calibration. Deep learning-based animal behavior recognition methods require substantial data accumulation and model training to improve accuracy and reliability. Summary of the Invention
[0005] The purpose of this invention is to address the problem of excessive human intervention in existing animal vital sign monitoring methods by proposing a multi-camera-based animal vital sign monitoring method, comprising the following steps: S1. Select the main viewpoint and other viewpoints of the multi-camera monitoring of animal vital signs to obtain multi-view videos of the animals; S2. Use the DeepLabCut pose estimation network to train on videos taken from various perspectives to obtain key point information of the animals; S3. Analyze the animal's center of mass movement based on the animal's key point information from the main perspective and compare it with the movement threshold to determine the animal's state. In a continuous image sequence during a relatively static period, obtain the region of interest based on the key point information. S4. Detect corner points within the region of interest and track the corner points to obtain corresponding motion signals; S5. Perform correlation analysis on motion signals from multiple perspectives, rank the importance of relevant components based on information gain, extract motion signals above a set threshold, and fuse them to obtain vital sign signals. S6. Preprocess the vital signs signals, use power spectrum analysis to analyze the preprocessed vital signs signals, obtain the energy distribution of the vital signs signals, obtain continuous frequency data of energy intensity distribution based on the animal's breathing and heartbeat frequency range, and calculate the values of respiratory rate and heart rate.
[0006] Furthermore, the specific state of the animal is determined as follows: Select a set of animal keypoints from the main viewpoint, calculate the average value of this set of keypoints to obtain the centroid position, obtain the animal motion amplitude of the current frame by comparing the displacement of the centroid of the current frame with that of the previous frame, compare the motion amplitude with the set motion threshold, when the motion amplitude is greater than or equal to the motion threshold, the animal is in motion, when the motion amplitude is less than the motion threshold, the animal is in a relatively static state.
[0007] Furthermore, the motion threshold is calculated as follows:
[0008]
[0009] in, Indicates the motion threshold. This represents the initially set motion threshold. Indicates the adjustment factor. This represents the average range of motion of the animal over the past n time steps. This represents the basic proportionality coefficient, and P represents the animal's body length.
[0010] Furthermore, the period of relative stillness refers to remaining still for longer than the length of the vital signs calculation window.
[0011] Furthermore, corner points are detected using the Shi-Tomasi algorithm.
[0012] Furthermore, the KLT algorithm is used to track corner points, tracking their displacement between different frames, integrating multi-dimensional information, and obtaining multi-channel vital sign signals.
[0013] Furthermore, canonical correlation analysis was used to perform correlation analysis on motion signals from multiple perspectives.
[0014] Furthermore, the formula for calculating information gain is:
[0015]
[0016] in, This represents the information entropy of category C, where C is the category of different respiratory or heart rate states of the animal. Indicates relevant components The set of possible values, This represents the subset of categories corresponding to the value v of A. and They are set C and The number of elements, Indicates category The probability of occurrence.
[0017] For each relevant component Let k be the total number of relevant components. Calculate the information gain for each relevant component to form an information gain set. For the information gain set The information gain order corresponds to the importance order of the relevant components when sorted in descending order. Components with higher information gain are listed first, indicating that they contribute more to distinguishing animal vital signs.
[0018] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-camera-based animal vital sign monitoring method.
[0019] The present invention also proposes an electronic device, including a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described multi-camera-based animal vital sign monitoring method.
[0020] The beneficial effects of the technical solution provided by this invention are: This invention acquires multi-view animal videos. The video streaming method accelerates data collection and reduces the impact of human intervention in experiments. A DeepLabCut pose estimation network is used to train the videos captured from various perspectives to obtain keypoint information of the animal. Based on the keypoint information from the main viewpoint, the animal's center of mass movement is analyzed and compared with a motion threshold to determine the animal's state. Regions of interest (ROIs) are identified from a continuous image sequence within a relatively static time period based on the keypoint information. Corner points are detected within the ROIs, and their corresponding motion signals are acquired by tracking them. Correlation analysis is performed on the motion signals from multiple perspectives to obtain vital signs signals. These vital signs signals are preprocessed, and power spectrum analysis is used to obtain their energy distribution. Continuous frequency data of energy intensity distribution is obtained based on the animal's respiratory and heart rate frequency ranges, and the respiratory rate and heart rate are calculated. By recording animal vital signs from multiple angles and comprehensively, and using machine learning methods to acquire motion signals and statistically analyze the correlations of various data, this invention effectively reduces human intervention and promotes innovation in animal experimental methods and data analysis. Attached Figure Description
[0021] Figure 1 This is a flowchart of an animal vital signs monitoring method based on multiple cameras according to an embodiment of the present invention; Figure 2 This is a schematic diagram of multi-view camera shooting according to an embodiment of the present invention; Figure 3 This is a graph showing the relationship between the animal's center of mass displacement and the threshold in an embodiment of the present invention; Figure 4 This is a diagram showing the ROI selection in an embodiment of the present invention, wherein... Figure 4 (a) is the feature point distribution map under the field of view of the main camera, (b) is the right ventral region under the field of view of the main camera, and (c) is the right ventral region under the field of view of camera ②. Figure 5 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0023] The flowchart of the multi-camera animal vital sign monitoring method of this invention is as follows: Figure 1 Specifically, it includes the following steps: S1. Select the main viewpoint and other viewpoints of the animal's vital signs multi-camera monitoring to obtain multi-view videos of the animal.
[0024] When selecting an animal monitoring perspective, the first step is to determine the optimal primary monitoring perspective. This perspective should provide a comprehensive view of most of the animal's limb features and approximate movement trajectory. From this perspective, the animal's limbs, torso, and other major body parts should be clearly visible, and the direction and main movement path of the animal within its activity area should be roughly traced. For example, for animals moving freely in the experimental area, the primary perspective should cover all key movements within its main activity area, including walking, jumping, and other common behaviors, all of which should be presented relatively completely from this viewpoint.
[0025] After determining the main perspective, other shooting angles need to be selected at certain tilt angles. On one hand, the size of the tilt angle should be moderate; too small and it may not be able to acquire enough information to fully observe the animal's overall condition; too large and it may cause some details to be lost, or make the animal appear too small in the frame, affecting the monitoring effect. On the other hand, the selected other shooting angles should complement the main perspective. These other angles can observe the animal from different directions and heights. For example, shooting from a slightly higher position at a certain tilt angle can capture the features of the animal's back and head, while also observing the animal's vertical movements, such as jumping up and down, lying down and getting up, thus complementing the information from the main perspective and forming a more comprehensive and three-dimensional picture of animal activity monitoring.
[0026] In actual monitoring, the top view is generally selected as the main viewpoint to determine the animal's movement status and as a basis for selecting the ROI for other views. A schematic diagram of multi-view camera shooting in this embodiment of the invention is shown below. Figure 2 As shown.
[0027] S2. Use the DeepLabCut pose estimation network to train on videos taken from various perspectives to obtain key point information of the animals.
[0028] DeepLabCut is an open-source deep learning tool for pose estimation and keypoint tracking. Its core idea is to use convolutional neural networks (CNNs) to identify specific keypoints of objects in an image.
[0029] First, image and video data containing animals are collected, covering various poses and angles to ensure the model's generalization ability. Next, using DeepLabCut's graphical user interface, images or videos are manually annotated, primarily including the animal's nose, ears, shoulders, limbs, and tail. Only visible keypoints from the current viewpoint are annotated; occluded or invisible parts are left unannotated. After annotation, a deep learning model is trained on the data. DeepLabCut primarily uses convolutional neural network-based deep learning models, typically pre-trained CNN architectures such as ResNet or VGG. During training, the model learns how to accurately identify keypoints from images. After training, the model needs to be evaluated to ensure its generalization ability on new data. This usually involves using a hold-set test or cross-validation to assess the model's performance. Once the model is trained and validated, it can be applied to new image and video data to track selected keypoints.
[0030] S3. Analyze the animal's center of mass movement based on the animal's key point information from the main perspective and compare it with the movement threshold to determine the animal's state. In a continuous image sequence of a relatively static state time period, obtain the region of interest based on the key point information.
[0031] Select a set of animal keypoints from the main viewpoint, calculate the average value of this set of keypoints to obtain the centroid position, obtain the animal motion amplitude of the current frame by comparing the displacement of the centroid of the current frame with that of the previous frame, compare the motion amplitude with the set motion threshold, when the motion amplitude is greater than or equal to the motion threshold, the animal is in motion, when the motion amplitude is less than the motion threshold, the animal is in a relatively static state.
[0032] Key features of a group of animals from a first-person perspective: centroid position The calculation formula is:
[0033]
[0034] By comparing the centroid of the current frame centroid of the previous frame The displacement is used to obtain the animal's movement amplitude d, and the calculation formula is:
[0035] Compare the motion amplitude d with the set motion threshold T, when... At that time, the animal's state is movement; when At that time, the animal is in a relatively static state.
[0036] Considering that differences in animal size can affect their range of motion, we will use the animal's body length as a reference. Assuming the animal's body length is P, we will establish a basic proportionality coefficient. (This can be determined based on a large amount of experimental data), so the initial motion threshold is set accordingly. It can be represented as: .
[0037] To more accurately reflect changes in animal movement, the movement threshold is adjusted by considering the animal's recent movement trends. Let the current time be... t ,in the past n The average range of motion of the animal within each time step is It is possible to calculate the centroid of the animal's key points from the main viewpoint. n The average displacement over a time step is obtained. An adjustment factor is introduced. (Based on experimental determination, the degree of influence of motion tendency on the threshold is used), then the formula for calculating the motion threshold is:
[0038] in This is the initial set movement threshold. If the animal's recent average movement amplitude is greater than the baseline threshold, it indicates that the animal is in a relatively active state. The amplitude will increase accordingly; conversely, if the recent average amplitude of movement is less than the baseline threshold, This reduces the threshold, thus enabling adaptive adjustment of the motion threshold based on the motion trend.
[0039] In this embodiment of the invention, six key points on the animal's body outline are selected, and the average value of these points is calculated to obtain the centroid position. The overall movement amplitude of the animal in this sequence is compared with the displacement of the centroid of the current sequence to that of the previous centroid, and then compared with a preset appropriate movement threshold. When the movement threshold is reached, the animal is considered to be moving; otherwise, it is considered stationary. The movement threshold is determined by calculating the pixel size occupied by the main body parts of the mouse and selecting the threshold for animal movement judgment at a ratio of 1:0.05. Since the centroid displacement occasionally exhibits outliers (far exceeding the values of nearby points), and the centroid displacement changes continuously over a short period, outliers can be removed from the displacement values. Finally, continuous image sequences that remain stationary for more than the length of the vital signs calculation window (corresponding to a time of 10 seconds) are used as valid segments for vital signs monitoring. (See the diagram showing the relationship between animal centroid displacement and threshold in this embodiment of the invention.) Figure 3 .
[0040] Within each static time period, the average distances of the left and right ventral sides from other viewpoints are calculated based on the main viewpoint. The side with the smaller average distance is selected as the ROI. Figure 4 , Figure 4This is a diagram showing the ROI selection in an embodiment of the present invention, wherein... Figure 4 (a) is the feature point distribution map under the field of view of the main camera, (b) is the right ventral region under the field of view of the main camera, and (c) is the right ventral region under the field of view of camera ②. Figure 4 In (a), camera ② should select the right abdomen of the animal as the ROI selection side, and then select the ROI region based on key point information, such as... Figure 4 (b) and (c).
[0041] S4. Detect corner points within the region of interest and track the corner points to obtain the corresponding motion signals.
[0042] Corner points are points in an image where grayscale changes drastically. They typically correspond to feature points of objects, such as edges and textures. Due to their uniqueness and stability within an image, corner points are widely used in image matching and motion detection. Compared to manual identification and labeling, automated algorithms can efficiently identify feature corner points on the skin surface. This reduces external intervention compared to traditional methods of manual labeling, improving data processing efficiency and accuracy. Therefore, corner points are suitable for identifying natural features of the left and right abdomens of animals, serving as signal anchor points for acquiring vital signs.
[0043] Corner points were detected using the Shi-Tomasi algorithm within the selected Region of Interest (ROI), and the 30 strongest points were selected as feature points based on their intensity. The Shi-Tomasi algorithm, also known as the minimum eigenvalue algorithm, is a corner detection method based on an improved Harris algorithm. It possesses good robustness and a corner quality evaluation index, which enable it to effectively identify corner features at various locations in animals, making it particularly advantageous for detecting significant features in areas of cardiopulmonary function in animals.
[0044] The KLT algorithm is an improved version of the Lucas-Kanade optical flow algorithm, capable of accurately tracking the displacement of detected keypoints between different frames. By recording and analyzing the inter-frame displacements of these keypoints in detail, multi-dimensional information is integrated to obtain initial multi-channel vital sign signals. These multi-channel signals will reflect the cardiopulmonary function-related physiological characteristics of animals in a resting state from multiple perspectives.
[0045] S5. Perform correlation analysis on motion signals from multiple perspectives, rank the importance of relevant components based on information gain, and extract motion signals above a set threshold for fusion to obtain vital sign signals.
[0046] Canonical Correlation Analysis (CCA) was used to conduct in-depth analysis of multi-channel signals obtained from the primary perspective and other different perspectives. CCA effectively uncovers the intrinsic correlation between two sets of variables. By constructing a suitable mathematical model, highly correlated components among multi-channel signals from multiple perspectives are identified. Based on these highly correlated components, data fusion is performed to obtain the fused vital sign signals. The fused signals integrate key information about animal vital signs from multiple perspectives, providing a more comprehensive and accurate reflection of the animal's true physiological state.
[0047] The formula for calculating information gain is as follows:
[0048]
[0049] in, This represents the information entropy of category C, where C is the category of different respiratory or heart rate states of the animal. Indicates relevant components The set of possible values, This represents the subset of categories corresponding to the value v of A. and They are set C and The number of elements, Indicates category The probability of occurrence.
[0050] S6. Preprocess the vital signs signals, use power spectrum analysis to analyze the preprocessed vital signs signals, obtain the energy distribution of the vital signs signals, obtain continuous frequency data of energy intensity distribution based on the animal's breathing and heartbeat frequency range, and calculate the values of respiratory rate and heart rate.
[0051] The fused vital sign signals were detrended and bandpass filtered to remove most of the noise interference. Power spectrum analysis was then used for in-depth analysis of the vital sign signals. Power spectrum analysis is a powerful tool for converting time-domain signals into frequency-domain representations, revealing the energy distribution of the signal at different frequencies. For vital sign signals, physiological activities such as respiration and heartbeat have specific frequency ranges, and power spectrum analysis clearly shows the contribution of these frequency components to the overall signal energy. The Fast Fourier Transform (FFT) algorithm was used to calculate the power spectrum of the vital sign signals. The FFT algorithm can efficiently decompose time-domain signals into combinations of sine and cosine waves of different frequencies, thus obtaining the amplitude and phase information of the signal at each frequency. By calculating the power spectrum, a spectrum diagram with frequency as the horizontal axis and power (energy) as the vertical axis can be obtained. In this spectrum diagram, the peak values and amplitudes at different frequency positions reflect the energy intensity of the corresponding frequency components in the vital sign signals.
[0052] Based on the animal's respiratory and heart rate ranges, frequency data with continuous energy intensity distribution are obtained from the power spectrum. Different species of experimental animals have different respiratory and heart rate ranges; for example, the respiratory rate of adult rats is typically between 80-160 breaths per minute (bpm), and the heart rate is between 250-600 bpm. Based on these known frequency ranges, corresponding frequency intervals are carefully screened from the power spectrum. Within these intervals, frequency components with relatively high energy intensity and continuous distribution are observed. These frequency components are often closely related to the animal's respiratory and heart rate activities because respiration and heart rate are periodic and regular physiological activities in animal vital signs, and they will appear as relatively stable and continuous energy peaks in the power spectrum. Algorithms are used to extract and label these frequency data to ensure accurate capture of signal features related to respiration and heart rate.
[0053] Finally, the respiratory rate and heart rate are calculated based on the continuous frequency data of the acquired energy intensity distribution. For respiratory rate calculation, the frequency corresponding to the energy peak within the selected respiratory frequency range is found; this frequency is the dominant frequency of the respiratory signal. Based on the relationship between respiratory frequency and respiratory rate (respiratory rate equals respiratory frequency multiplied by 60 seconds), the animal's respiratory rate can be calculated. For example, if the dominant frequency within the respiratory frequency range in the power spectrum is 1.5 Hz (i.e., 90 breaths per minute), then the animal's respiratory rate is 90 breaths per minute. Similarly, for heart rate calculation, the frequency corresponding to the energy peak within the heart rate range is found, and then the heart rate value is calculated according to the conversion relationship between heart rate and heart rate. During the calculation process, signal stability and reliability are also considered. Frequency data from multiple consecutive time periods are averaged to reduce the influence of random factors on the calculation results, thus obtaining more accurate respiratory rate and heart rate values.
[0054] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the above-described multi-camera-based animal vital sign monitoring method.
[0055] Please see Figure 5 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.
[0056] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the above-mentioned multi-camera-based animal vital sign monitoring method.
[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for monitoring animal vital signs based on multiple cameras, characterized in that, Includes the following steps: S1. Select the main viewpoint and other viewpoints of the multi-camera monitoring of animal vital signs to obtain multi-view videos of the animals; S2. Use the DeepLabCut pose estimation network to train on videos taken from various perspectives to obtain key point information of the animals; S3. Analyze the animal's center of mass movement based on the animal's key point information from the main perspective and compare it with the movement threshold to determine the animal's state. In a continuous image sequence during a relatively static period, obtain the region of interest based on the key point information. S4. Detect corner points within the region of interest and track the corner points to obtain corresponding motion signals; S5. Perform correlation analysis on motion signals from multiple perspectives, rank the importance of relevant components based on information gain, extract motion signals above a set threshold, and fuse them to obtain vital sign signals. S6. Preprocess the vital signs signals, use power spectrum analysis to analyze the preprocessed vital signs signals, obtain the energy distribution of the vital signs signals, obtain continuous frequency data of energy intensity distribution based on the animal's breathing and heartbeat frequency range, and calculate the values of respiratory rate and heart rate. Among them, the top view is selected as the main view, which covers all key point information. Other shooting views complement the main view camera in terms of the direction of movement and activity path captured.
2. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, The specific steps to determine the animal's condition are: Select a set of animal keypoints from the main viewpoint, calculate the average value of this set of keypoints to obtain the centroid position, obtain the animal motion amplitude of the current frame by comparing the displacement of the centroid of the current frame with that of the previous frame, compare the motion amplitude with the set motion threshold, when the motion amplitude is greater than or equal to the motion threshold, the animal is in motion, when the motion amplitude is less than the motion threshold, the animal is in a relatively static state.
3. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, The motion threshold is calculated as follows: in, Indicates the motion threshold. This represents the initially set motion threshold. Indicates the adjustment factor. This represents the average range of motion of the animal over the past n time steps. This represents the basic proportionality coefficient, and P represents the animal's body length.
4. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, The period of relative stillness refers to the time during which the person remains still for more than the length of the vital signs calculation window.
5. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, Corner points are detected using the Shi-Tomasi algorithm.
6. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, The KLT algorithm is used to track corner points, monitor their displacement between different frames, integrate multi-dimensional information, and obtain multi-channel vital signs signals.
7. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, Canonical correlation analysis was used to perform correlation analysis on motion signals from multiple perspectives.
8. The method for monitoring animal vital signs based on multiple cameras according to claim 1, characterized in that, The formula for calculating information gain is: in, This represents the information entropy of category C, where C is the category of different respiratory or heart rate states of the animal. Indicates relevant components The set of possible values, express The subset of categories corresponding to the value v. and They are set C and The number of elements, Indicate category The probability of occurrence.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.
10. An electronic device, characterized in that, The device includes a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1-8.