Method for detecting heart rate, respiration and oxyhemoglobin saturation based on micro-vibration image

Abstract

The invention discloses a method for detecting heart rate, respiration and oxyhemoglobin saturation based on micro-vibration images, and belongs to the technical field of physiological detection and image processing. A camera collects facial micro-vibration videos of a testee in real time, uses an ROI algorithm to lock a facial key area, extracts RGB micro-vibration change signals to replace red light and near-red light signals, and determines whether the micro-vibration videos are abnormal or not; performing wavelet transform five-layer decomposition, three rounds of peak valley screening and abnormal value elimination on the RGB microvibration time sequence signal, performing fine tuning training by using a yov11 model, and calculating to obtain heart rate, respiration and blood oxygen physiological indexes. According to the invention, non-contact detection is realized through video acquisition and signal processing of micro-vibration of a face area, and the comfort level and the use convenience of a testee are improved; interference caused by motion artifacts, baseline drift and expression changes is eliminated through multi-scale wavelet denoising; the three-wheel peak-valley screening algorithm deeply excavates the features of the face micro-vibration signals, and the recognition accuracy of the signal peak-valley pairs is improved; and an abnormity elimination and self-feedback parameter adjustment mechanism improves the calculation precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of physiological detection and image processing, and particularly relates to a method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images. BACKGROUND

[0002] Heart rate, respiration and blood oxygen saturation (SpO2) are three key physiological indicators reflecting the state of human vital signs, and are widely used in intensive care, clinical assistance, chronic respiratory disease management, sleep respiration monitoring and many other scenarios. In today's society with rapid technological development, non-contact physiological health detection is of great importance. The current commonly used methods for detecting heart rate, respiration and blood oxygen saturation include finger clip type photoplethysmography (PPG) sensors, chest strap type respiratory bands or wrist strap type heart rate meters, which can quickly and accurately detect heart rate, respiration and blood oxygen saturation. However, such contact methods are subject to discomfort and motion artifacts in actual application, and the sensor needs to be closely attached to the skin for a long time, which can easily cause local compression and discomfort. Moreover, when the patient has limb movement or body position change, and the environmental light changes, the signal is often distorted, causing the measurement result to deviate from the true value. Non-contact visual detection, such as video magnification methods such as Eulerian Video Magnification (EVM), can enhance the weak blood vessel pulsation to some extent, making heart rate, respiration and blood oxygen saturation detection more simple and convenient, and not causing discomfort to the human body. However, in complex actual scenarios, factors such as loss of key data and reduction of clarity in video magnification can seriously mask the skin's light absorption signal, resulting in low signal-to-noise ratio after magnification and detection distortion. Traditional non-contact detection is difficult to balance the real physiological changes, which hinders its application in non-contact physiological detection and intelligent physiological detection scenarios. SUMMARY

[0003] The purpose of the present application is to provide a method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images, and to provide a non-contact, high-accuracy and strong anti-interference intelligent detection method for heart rate, respiration and blood oxygen saturation. Through multi-level processing and abnormal data rejection of facial muscle micro-vibration signals, and yolov11 model fine-tuning training, the physiological indicator detection accuracy and stability based on facial micro-vibration signals are improved.

[0004] In order to achieve the above object, the present application provides a method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration image, characterized in that a video camera is used to collect micro-vibration video of the face of the person to be detected in real time, a ROI algorithm is used to lock the key area of the face, and the RGB micro-vibration change signal is extracted to replace the red light and near red light signal, the RGB micro-vibration time sequence signal is subjected to five-layer decomposition by wavelet transform, signal reconstruction, three rounds of peak-valley screening and abnormal value elimination, and yolov11 model fine-tuning feedback training is carried out, and the heart rate, respiration and blood oxygen physiological indicators are calculated; specifically comprising the following steps:

[0005] S1, a monitoring device is built in a detection place, the monitoring device comprises a camera unit, a central processor, a display screen, and an operation mode is set;

[0006] S2, the detection device is started, it is identified that the front personnel passes, the camera unit real-time enters 5-10s video, and is transmitted to the central processor and is preprocessed;

[0007] S3, the central processor locks the face region of interest through the ROI algorithm, extracts the time sequence signal of the RGB three-channel micro-vibration change of each pixel point, and selects two groups of stable RGB micro-vibration time sequence signals;

[0008] S4, the selected two groups of RGB micro-vibration time sequence signals are processed;

[0009] S5, the calibration coefficient is determined based on yolov11 model fine-tuning training, and the two groups of RGB micro-vibration time sequence signals processed are substituted into the physiological index calculation formula;

[0010] S6, the detection result is displayed by the display screen and is stored in the cloud, if the detection result continuously is lower than the preset threshold value, the system automatically alarms and prompts, if the detection result value is in the preset threshold value range, the physiological indicators of the person to be detected are normal.

[0011] Preferably, in S1, the camera unit is a video camera cooperating with a ring-shaped light supplement lamp, so as to ensure that stable face area video is obtained under different environmental light conditions; the video camera is connected with the central controller through an industrial-grade USB 3.0 interface, and outputs three-channel original video stream at a frame rate of not less than 30fps in real time; the video acquisition process is synchronously managed by a timing module in the central processor, so as to ensure that the RGB three-channel image frames of each pixel point are one-to-one corresponding, so as to facilitate subsequent accurate calculation.

[0012] Preferably, the video preprocessing comprises the following steps:

[0013] S21, the collected original video enters the key frame extraction and preprocessing submodule, and the clarity evaluation algorithm based on Laplacian operator and the local light uniformity evaluation model are used to score each frame of image without perceptual quality, and the low-quality frames with blur, overexposure and obvious shadow are removed, and the key frames accurately reflecting the skin micro-vibration are reserved;

[0014] S22, the image is grayed and preliminarily filtered, which lays a foundation for subsequent ROI detection and micro-vibration time sequence signal extraction.

[0015] Preferably, in S3, the following steps are specifically included:

[0016] S31, the micro-vibration signal extraction submodule first calls the face feature point detection algorithm to position the ROI, and aligns the pixels in the region;

[0017] S32, for the time sequence change of the pixel brightness in each ROI region, three groups of original micro-vibration time sequence signals are extracted from the RGB channel respectively, two groups of stable micro-vibration time sequence signals are preliminarily selected, and double-channel signals containing heart rate, respiration and blood oxygen information are obtained, but high-frequency noise and low-frequency drift are mixed therein.

[0018] Preferably, in S4, the following steps are specifically included:

[0019] S41, the multi-scale wavelet denoising submodule uses the wavelet db4 as the mother wavelet to decompose the selected two groups of RGB micro-vibration time sequence signals for five layers, removes the baseline drift artifact and high-frequency noise, and the denoised signal clearly retains the periodic micro-vibration characteristics in form;

[0020] S42, the pure signal is reconstructed in the heart rate frequency band and the respiration frequency band to calculate the heart rate and the respiration;

[0021] S43, the three-round peak-valley screening submodule further processes the signal, the first round calculates the mean and standard deviation of the signal values corresponding to all peak points and valley points, and preliminarily finds all possible peak-valley points; the second round compares the signal value and time interval of adjacent peak points and adjacent valley points, screens out the peak-valley value index that does not meet the time interval requirement, and removes the pseudo-peak-valley that is too close or too far apart; the third round finds the index of abnormal points obviously deviating from the mean value, which is used for data cleaning, further optimization and locking of the effective peak-valley pair of each cardiac cycle;

[0022] S44, for the cycles exceeding the parameter range and the morphological abnormalities, the three-round peak-valley screening submodule automatically discards; when the continuous abnormal cycles exceed the preset proportion, the three-round peak-valley screening submodule feeds back to the central processor, restarts the key frame extraction and prompts the operator to adjust the collection conditions.

[0023] Preferably, in S5, the following steps are specifically included:

[0024] S51, calculate the heart rate and respiratory formula:

[0025]

[0026]

[0027] wherein x and y are two groups of RGB micro-vibration time series signals selected, after multi-scale wavelet transform denoising and signal reconstruction, the abnormal frequency is removed through the filter and the time series signals x(heart) and y(heart) with the frequency range of 1.0-2.0Hz and the time series signals x(breath) and y(breath) with the frequency range of 0.2-0.5Hz are screened out;

[0028] S52, calculate the blood oxygen saturation:

[0029] AC x = mean_peak_x - mean_balley_x;

[0030] AC y = mean_peak_y - mean_valley_y;

[0031] DC x = mean_valley_x;

[0032] DC y = mean_valley_y;

[0033]

[0034] wherein AC is the alternating component of the signal, DC is the direct current component of the signal, x and y are two groups of RGB micro-vibration time series signals, mean_peak_x / y is the final peak average value, mean_valley_x / y is the final valley average value; A and B are calibration coefficients, specifically, the RGB micro-vibration time series and the blood oxygen data measured by the finger clip type photoelectric plethysmograph sensor are put into the yolov11 model training, compared with the blood oxygen data measured by the method, the A and B calibration coefficients are self-feedback fine-tuned and finally determined.

[0035] Therefore, the application adopts the above-mentioned method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images, realizes non-contact intelligent detection of heart rate, respiration and blood oxygen saturation through video collection and multi-level signal processing of micro-vibration of the face region, and improves the comfort and use convenience of the measured person. Motion artifacts, baseline drift and interference caused by expression changes are excluded by multi-scale wavelet denoising; the essence characteristics of the face micro-vibration signal are deeply mined by the three-round peak-valley screening algorithm, and the recognition accuracy of the signal peak-valley pair is improved; the calculation accuracy of heart rate, respiration and blood oxygen saturation is improved by the abnormality elimination and self-feedback parameter adjustment mechanism constructed by the method.

[0036] The technical solutions of the application will be further described in detail below with reference to the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a detection flowchart of an embodiment of the method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images of the application;

[0038] Figure 2 is a signal extraction diagram of collected video of an embodiment of the method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images of the application;

[0039] Figure 3 is a wavelet transform algorithm flowchart of an embodiment of the method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images of the application;

[0040] Figure 4 is a three-round peak-valley screening algorithm flowchart of an embodiment of the method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images of the application;

[0041] Figure 5 is a detection result report schematic diagram of an embodiment of the method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images of the application. DETAILED DESCRIPTION

[0042] The technical solutions of the application will be further described in detail below with reference to the drawings and examples.

[0043] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the meanings that can be commonly understood by a person with ordinary skill in the art to which this application belongs. The terms "first", "second", and similar terms used in the present application do not denote any order, quantity, or importance, but are used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0044] Embodiment one

[0045] The present application provides a method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration image, the detection process is as shown in Figure 1 The micro-vibration video of the face of the person being tested is collected in real time using a camera, the ROI algorithm locks the key area of the face, and the RGB micro-vibration change signal is extracted instead of the red and near-red light signals. The RGB micro-vibration time sequence signal is subjected to five-layer decomposition by wavelet transform, signal reconstruction, three rounds of peak and valley screening and abnormal value elimination, and yolov11 model fine-tuning feedback training is performed, and the heart rate, respiration and blood oxygen physiological indicators are calculated; Specifically, the following steps are included:

[0046] S1, a monitoring device is built in the detection place, the monitoring device includes a camera unit, a central processor, a display screen, and a running mode is set, specifically: the camera unit is a camera cooperating with a ring-shaped light supplement lamp to ensure that stable face area video is obtained under different environmental light conditions; the camera is connected to the central controller through an industrial-grade USB 3.0 interface, and outputs three-channel raw video stream in real time at a frame rate of not less than 30fps; the video acquisition process is synchronously managed by a timing module in the central processor, ensuring that the RGB three-channel image frames of each pixel point are one-to-one corresponding, so as to facilitate subsequent accurate calculation.

[0047] S2, start the detection device, identify the person passing in front, the camera unit real-time input 5-10s video, and transmit to the central processor for pretreatment, the video acquisition and signal extraction process is as shown in Figure 2 The video pretreatment includes the following steps:

[0048] S21, the collected original video enters the key frame extraction and preprocessing submodule, and the clarity evaluation algorithm based on Laplacian operator and the local light uniformity evaluation model are used to score each frame of image without perceptual quality, and the low-quality frames with blur, overexposure and obvious shadow are removed, and the key frames which can accurately reflect the skin micro-vibration are reserved;

[0049] S22, the image is grayed and preliminarily filtered, which lays a foundation for subsequent ROI detection and micro-vibration time sequence signal extraction.

[0050] S3, the central processing unit locks the face region of interest through the ROI algorithm, extracts the time sequence signal of the RGB three-channel micro-vibration of each pixel point, and selects two groups of stable RGB micro-vibration time sequence signals; specifically comprising the following steps:

[0051] S31, the micro-vibration signal extraction submodule first calls the face feature point detection algorithm ROI to position the region of interest, and aligns the region at the pixel level;

[0052] S32, the time sequence change of the pixel brightness in each ROI region is extracted from the RGB channel to obtain three groups of original micro-vibration time sequence signals, two groups of stable micro-vibration time sequence signals are preliminarily selected, and double-channel signals containing heart rate, respiration and blood oxygen information are obtained, but high-frequency noise and low-frequency drift are mixed therein.

[0053] S4, the selected two groups of RGB micro-vibration time sequence signals are processed; specifically comprising the following steps:

[0054] S41, the multi-scale wavelet denoising submodule uses the wavelet db4 as the mother wavelet to decompose the selected two groups of RGB micro-vibration time sequence signals for five layers, removes the baseline drift artifact and high-frequency noise, and the denoised signal clearly retains the periodic micro-vibration characteristics in form;

[0055] S42, the pure signal is reconstructed in the heart rate frequency band and the respiration frequency band to calculate the heart rate and the respiration, and the flow chart is as shown in Figure 3 ;

[0056] S43, the three-round peak-valley screening submodule further processes the signal, and the flow chart is as shown in Figure 4 ; the first round calculates the mean and standard deviation of the signal values corresponding to all peak points and valley points, preliminarily finds all possible peak-valley points; the second round compares the signal value and time interval of adjacent peak points and adjacent valley points, screens out the peak-valley value index which does not meet the time interval requirement, and removes the pseudo-peak-valley which is too close or too far apart; the third round finds the index of abnormal points which obviously deviate from the mean value, which is used for data cleaning, further optimization and locking of the effective peak-valley pair of each cardiac cycle;

[0057] S44, for the parameter range and the cycle of abnormal morphology, three-wheel peak valley screening sub-module automatically discarded; when the continuous abnormal cycle exceeds the preset proportion, the three-wheel peak valley screening sub-module feeds back to the central processor, restarts the key frame extraction and prompts the operator to adjust the collection conditions.

[0058] S5, based on the yolov11 model fine-tuning training to determine the calibration coefficient, the processed two groups of RGB micro-vibration time series signals are substituted into the physiological index calculation formula; specifically comprising the following steps:

[0059] S51, the heart rate and breathing formula are calculated as:

[0060]

[0061] Wherein, x and y are two groups of RGB micro-vibration time series signals selected, after multi-scale wavelet transform denoising and signal reconstruction, the abnormal frequency is removed through the filter and the time series signals x(heart) and y(heart) with frequency range of 1.0-2.0Hz and the time series signals x(breath) and y(breath) with frequency range of 0.2-0.5Hz are screened out;

[0062] S52, calculate the oxygen saturation:

[0063] AC x = mean_peak_x-mean_balley_x;

[0064] AC y = mean_peak_y-mean_valley_y;

[0065] DC x = mean_valley_x;

[0066] DC y = mean_valley_y;

[0067]

[0068] Wherein, AC is the alternating current component of the signal, DC is the direct current component of the signal, x and y are two groups of RGB micro-vibration time series signals, mean_peak_x / y is the final peak average value, mean_valley_x / y is the final valley average value; A, B are calibration coefficients, specifically, through the yolov11 model training fine-tuning, the RGB micro-vibration time series and the blood oxygen data measured by the finger clip type photoelectric plethysmograph sensor are put into the yolov11 model training, compared with the blood oxygen data measured by the method, the A, B calibration coefficients are fine-tuned and finally determined.

[0069] S6, the detection result is displayed by the display screen and stored in the cloud, if the detection result continuously is below the preset threshold, the system automatically alarms; if the detection result value is in the preset threshold range, the physiological index of the measured person is normal, and the detection report is as shown in Figure 5

[0070] Therefore, the application adopts the above-mentioned method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images, realizes non-contact intelligent heart rate, respiration and blood oxygen saturation detection through video acquisition and multi-level signal processing of facial region micro-vibration, and improves the comfort and use convenience of the measured person. Multi-scale wavelet denoising excludes motion artifacts, baseline drift and interference caused by expression changes; three-wheel peak-valley screening algorithm deeply excavates the essential characteristics of facial micro-vibration signals, and improves the recognition accuracy of signal peak-valley pairs; the abnormality elimination and self-feedback parameter adjustment mechanism constructed by the method improves the calculation accuracy of heart rate, respiration and blood oxygen saturation.

[0071] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.​

Claims

1. A method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images, characterized in that: A camera is used to capture the subject's facial micro-vibration video in real time. The ROI algorithm is used to lock the key facial areas, and the RGB micro-vibration change signal is extracted to replace the red and near-infrared light signals. The RGB micro-vibration time series signal is subjected to five-layer wavelet transform decomposition, signal reconstruction, three rounds of peak and valley screening, and outlier removal. The yolov11 model is then fine-tuned and trained with feedback to calculate the heart rate, respiration, and blood oxygen physiological indicators. The specific steps include: S1. Set up a monitoring device at the testing site. The monitoring device includes a camera unit, a central processing unit, and a display screen, and set the operating mode; S2. Start the detection device and identify the person passing by. The camera unit records 5-10 seconds of video in real time and transmits it to the central processor for pre-processing. S3: The central processing unit locks the facial region of interest through the ROI algorithm, extracts the time series signal of the RGB three-channel micro-vibration changes of each pixel point, and selects two sets of stable RGB micro-vibration time series signals; S4, processing the two selected groups of RGB micro-vibration time series signals; S5. Fine-tune the training based on the yolov11 model to determine the calibration coefficient, and substitute the processed two sets of RGB micro-vibration time series signals into the physiological index calculation formula; S6. The test results are displayed on the display screen and stored in the cloud. If the test results are continuously lower than the preset threshold, the system will automatically issue an alarm. If the test result value is within the preset threshold range, the physiological indicators of the subject are normal.

2. The method for detecting heart rate, respiration, and blood oxygen saturation based on micro-vibration images according to claim 1, characterized in that: Specifically, in S1, the camera unit is a camera equipped with a ring fill light to ensure stable facial area video under different ambient lighting conditions. The camera is connected to the central controller via an industrial-grade USB 3.0 interface, outputting a three-channel raw video stream in real time at a frame rate of no less than 30fps. The video acquisition process is synchronously managed by the timing module in the central processing unit to ensure a one-to-one correspondence between the RGB three-channel image frames of each pixel point, facilitating subsequent accurate calculations.

3. The method for detecting heart rate, respiration, and blood oxygen saturation based on micro-vibration images according to claim 2, characterized in that: Video preprocessing includes the following steps: S21. The collected raw video enters the keyframe extraction and preprocessing submodule. Using a Laplacian-based clarity evaluation algorithm and a local illumination uniformity assessment model, each frame is scored for quality without perception. Blurred, overexposed, or low-quality frames with significant shadows are removed, while retaining keyframes that accurately reflect skin micro-vibrations. S22. Grayscale the image and perform preliminary noise filtering to lay the foundation for subsequent ROI detection and micro-vibration time series signal extraction.

4. The method for detecting heart rate, respiration, and blood oxygen saturation based on micro-vibration images according to claim 3, characterized in that: In S3, the following steps are specifically included: S31, the micro-vibration signal extraction submodule first calls the facial feature point detection algorithm ROI to locate the region of interest and performs pixel-level alignment on the region; S32. For the temporal changes in pixel brightness within each ROI area, three sets of original micro-vibration time series signals are extracted from the RGB channels respectively. Two sets of stable micro-vibration time series signals are initially selected to obtain a dual-channel signal containing heart rate, respiration, and blood oxygen information, but which is mixed with high-frequency noise and low-frequency drift.

5. The method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images according to claim 4, characterized in that: S4 specifically includes the following steps: S41, multi-scale wavelet denoising submodule uses the above wavelet db4 as the mother wavelet to perform five-layer decomposition on the two selected RGB micro-vibration time series signals to remove baseline drift artifacts and high-frequency noise. After denoising, the signal clearly retains the periodic micro-vibration characteristics in morphology; S42, reconstructing a pure signal in the heart rate frequency band and the respiratory frequency band to calculate the heart rate and respiration; S43. The three-round peak-valley screening submodule further processes the signal. The first round calculates the mean and standard deviation of the signal values ​​corresponding to all peak points and valley points, preliminarily finding all possible peaks and valleys. The second round compares the signal values ​​and time intervals of adjacent peak points and valley points, screening out peak-valley indexes that do not meet the time interval requirements and eliminating pseudo-peaks and valleys that are too close or too far apart. The third round finds the indexes of abnormal points that significantly deviate from the mean for data cleaning, further optimizing and locking the valid peak-valley pairs for each cardiac cycle. S44. For cycles that exceed the parameter range and have abnormal morphology, the three-round peak-valley screening submodule automatically discards them; when the continuous abnormal cycles exceed the preset ratio, the three-round peak-valley screening submodule feeds back to the central processor, restarts the key frame extraction and prompts the operator to adjust the acquisition conditions.

6. The method for detecting heart rate, respiration and blood oxygen saturation based on micro-vibration images according to claim 5, characterized in that: S5 specifically includes the following steps: S51. The formula for calculating heart rate and respiration is: Among them, x and y are two sets of selected RGB micro-vibration time series signals. After multi-scale wavelet transform denoising and signal reconstruction, the abnormal frequency is removed by filtering and the time series signals x(heart) and y(heart) with a frequency range of 1.0–2.0 Hz and x(breath) and y(breath) with a frequency range of 0.2–0.5 Hz are screened out. S52. Calculate blood oxygen saturation: AC x =mean_peak_x-mean_valley_x; AC y =mean_peak_y-mean_vally_y; DC x =mean_valley_x; DC y =mean_valley_y; Among them, AC is the alternating current component of the signal, DC is the direct current component of the signal, x and y are two sets of RGB micro-vibration time series signals, mean_peak_x / y is the final peak average, and mean_valley_x / y is the final valley average; A and B are calibration coefficients. Specifically, they are fine-tuned and determined through yolov11 model training. The RGB micro-vibration time series and the blood oxygen data measured by the finger-clip photoelectric volume pulse wave sensor are put into the yolov11 model training and compared with the blood oxygen data measured by this method. The A and B calibration coefficients are fine-tuned by self-feedback and finally determined.

Images