PTT estimation method, system, terminal and medium based on multi-camera equipment

Abstract

The present invention provides a PTT estimation method, system, terminal and medium based on a multi-camera device, which specifically relates to the field of biomedical engineering technology. The scheme includes: starting at least two cameras of the multi-camera device, and using the multi-camera device to synchronously collect RGB videos of different skin areas of the user; based on each RGB video, extracting PPG signals of the corresponding skin areas respectively; detecting all peaks of the two PPG signals in the same time window, and obtaining the PTT estimation result based on the time relationship between all peaks of the two PPG signals. The scheme uses the front and rear cameras to collect RGB videos of different skin areas in the same time period during normal use of the smart device, so as to realize the non-sensing collection of the entire PTT estimation process. The data collection process is convenient and flexible, and can reduce the cost of estimating PTT and improve the estimation accuracy of pulse wave transmission time.

Description

Technical Field

[0001] The present invention relates to the field of biomedical engineering technology, and in particular to a PTT estimation method, system, terminal and medium based on multi-camera equipment. Background Art

[0002] At present, cuffless blood pressure estimation methods mainly include multi-point pulse transit time (PTT) estimation, multi-band PTT estimation and waveform estimation. Among them, multi-band PTT estimation requires high-frame rate cameras and is difficult to be portable; waveform estimation requires a large amount of high-quality photoplethysmogram (PPG) signals for model training, which is difficult to deploy in practice. In contrast, the multi-point PTT method is simple to implement and has more practical application value.

[0003] Existing multi-point PTT estimation methods usually require the use of different external devices to collect pulse signals at different locations to estimate multi-point PTT, such as PTT between heart ECG and fingertip PPG. This blood pressure estimation method needs to rely on multiple external devices, has high measurement costs and is not convenient and flexible enough. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a PTT estimation method, system, terminal and medium based on a multi-camera device, aiming to solve the problems that the multi-point PTT estimation method in the prior art needs to rely on multiple external devices, has high measurement costs and is not convenient and flexible enough.

[0005] In order to achieve the above object, the first aspect of the present invention provides a PTT estimation method based on a multi-camera device, comprising:

[0006] Starting at least two cameras of a multi-camera device, and using the multi-camera device to synchronously capture RGB videos of different skin areas of a user;

[0007] Based on each of the RGB videos, extracting PPG signals of corresponding skin areas respectively;

[0008] All peaks of the two PPG signals in the same time window are detected, and a PTT estimation result is obtained based on the time relationship between all the peaks of the two PPG signals.

[0009] Optionally, extracting the PPG signal of the corresponding skin area based on each segment of the RGB video includes:

[0010] Based on each of the RGB videos, a series of RGB images of the corresponding skin area are obtained;

[0011] Determine the skin chromaticity change of the corresponding skin area at each pixel point based on the pixel value of each pixel point in each frame of the RGB image;

[0012] Based on the skin chromaticity change of each pixel point in each skin area, a PPG signal of the corresponding skin area is determined.

[0013] Optionally, determining the PPG signal of the corresponding skin area based on the skin chromaticity change of each pixel in each skin area includes:

[0014] Calculating the mean of the pixel values ​​of all the pixels in each skin area to determine the mean of the skin chromaticity change of the corresponding skin area;

[0015] The skin chromaticity deviation between the skin chromaticity change at each pixel point and the corresponding skin chromaticity change mean of each skin area is calculated, and the PPG signal of the corresponding skin area is determined based on the ratio between the skin chromaticity deviation of each pixel point and the corresponding skin chromaticity change mean.

[0016] Optionally, obtaining a PTT estimation result based on the time relationship between all the peaks of the two PPG signals includes:

[0017] Calculate the timestamps of all the peaks of the two PPG signals;

[0018] Based on the timestamps of all the peaks, calculating the time delay between the nearest neighbor peaks of the two PPG signals;

[0019] Based on the average of all said time delays, a PTT estimation result is obtained.

[0020] Optionally, obtaining a PTT estimation result based on an average value of all the time delays includes:

[0021] The two PPG signals are defined as a first PPG signal and a second PPG signal, and the wavelength of light in the target color channel corresponding to the first PPG signal is longer;

[0022] respectively calculating average time intervals between respective peaks of the first PPG signal and the second PPG signal to obtain a first average time interval and a second average time interval;

[0023] If the PTT estimation result is greater than zero, the PTT estimation result is equal to an average of time delays of all nearest neighbor peaks between the second PPG signal and the first PPG signal, and the PTT estimation result is less than the first average time interval;

[0024] If the PTT estimation result is less than zero, the PTT estimation result is equal to the average of the time delays of all nearest neighbor peaks between the first PPG signal and the second PPG signal, and the absolute value of the PTT estimation result is less than the second average time interval.

[0025] Optionally, after obtaining the PTT estimation result, the method further includes:

[0026] The PTT estimation result is input into a trained blood pressure estimation model to obtain the blood pressure estimation result of the user, wherein the blood pressure estimation model is used to reflect the mapping relationship between pulse wave transmission time and blood pressure.

[0027] Optionally, the synchronously collecting RGB videos of different skin areas of the user using the multi-camera device includes:

[0028] Synchronously capturing an RGB video of a first skin area and an RGB video of a second skin area of ​​a user using the multi-camera device;

[0029] A target detection algorithm is used to detect and crop the RGB video of the first skin area and / or the RGB video of the second skin area to obtain the RGB video of the first target area and the RGB video of the second target area.

[0030] A second aspect of the present invention provides a PTT estimation system based on a multi-camera device, the system comprising:

[0031] An information acquisition module, used to activate the front-end and rear-end camera functions of the multi-camera device, and use the multi-camera device to synchronously acquire RGB videos of different skin areas of the user;

[0032] A PPG signal extraction module, used to extract the PPG signal of the corresponding skin area based on each RGB video segment;

[0033] The PTT estimation module is used to detect all the peaks of the two PPG signals in the same time window, and obtain a PTT estimation result based on the time relationship between all the peaks of the two PPG signals.

[0034] A third aspect of the present invention provides an intelligent terminal, which includes a memory for storing executable instructions; and a processor for calling and running the executable instructions in the memory to execute any one of the steps of the above-mentioned PTT estimation method based on a multi-camera device.

[0035] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions. When the program instructions are executed by a processor, any step of the above-mentioned PTT estimation method based on a multi-camera device is implemented.

[0036] Compared with the prior art, the beneficial effects of this solution are as follows:

[0037] The present invention uses a smart device equipped with multiple cameras, and uses the front and rear cameras to collect RGB videos of different skin areas in the same time period during normal use of the smart device, so as to realize the non-sensing collection of the entire PTT estimation process. The data collection process is convenient and flexible, which can not only reduce the cost of estimating PTT, but also significantly improve the user experience. And by constructing the correlation between skin chromaticity changes and PPG signals, it is realized by calculating the skin chromaticity changes of different skin areas to extract the PPG signals of the corresponding skin areas, so that PTT can be accurately estimated. At the same time, the front and rear cameras in the multi-camera device used in the present invention are both RGB cameras, so that the multi-band fusion noise reduction method is used to fuse the RGB signals to generate a PPG signal that is robust to motion and illumination changes, which is conducive to enhancing the robustness of PTT estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 It is a flow chart of the PTT estimation method based on multi-camera equipment of the present invention;

[0040] Figure 2 It is a module schematic diagram of the PTT estimation process based on a multi-camera device of the present invention;

[0041] Figure 3 It is a schematic diagram of blood pressure estimation of the present invention;

[0042] Figure 4 It is a schematic diagram of the PTT estimation system module based on multi-camera equipment of the present invention;

[0043] Figure 5 It is a schematic diagram of the structure of the intelligent terminal of the present invention. DETAILED DESCRIPTION

[0044] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0045] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0046] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0048] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0049] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0051] Cuffless blood pressure estimation methods mainly include multi-point pulse transit time (PTT) estimation, multi-band PTT estimation and waveform estimation. Among them, multi-band PTT estimation needs to rely on high-frame rate cameras, which is difficult to be portable; waveform estimation requires a large number of high-quality photoplethysmogram (PPG) signals for model training, which is difficult to deploy in practice. In contrast, the multi-point PTT method is simple to implement and has more practical application value. Existing multi-point PTT estimation methods usually require the use of different external devices to collect pulse signals at different locations to estimate multi-point PTT, such as PTT between heart ECG and fingertip PPG. This blood pressure estimation method needs to rely on multiple external devices, has high measurement costs, and is not convenient and flexible enough.

[0052] With the popularity of smart phones and the enhancement of their functions, they are usually equipped with front and rear cameras, and can even simultaneously enable the front and rear camera functions to capture videos. Based on the defects of existing multi-point PTT estimation methods and smart devices such as smart phones that have both front and rear camera functions, the present invention proposes a PTT estimation method based on a multi-camera device, which is deployed on electronic devices such as computers and servers, and is applied to scenarios where smart devices equipped with front and / or rear cameras are used to assist in detecting user PTT, and is aimed at using the smart device to simultaneously measure videos or a series of frame images of different skin areas of the user to achieve multi-point PTT estimation. The type of the above-mentioned smart device is not limited, and can be a smart device such as a smart phone, a camera, etc. that is equipped with a front and / or rear camera and can simultaneously enable the camera function of at least two cameras, hereinafter collectively referred to as a multi-camera device.

[0053] This embodiment uses a smartphone that is actually used in daily scenarios and is equipped with a front camera and a rear camera, and can turn on the front camera and rear camera functions at the same time, as an exemplary example of a multi-camera device to explain in detail the specific process of using a multi-camera device to achieve multi-point PTT estimation. However, this exemplary example is not intended to limit the protection scope of the multi-camera device. All multi-camera devices that can implement the method of this embodiment are within the protection scope of the present invention. Specifically, Figure 1 and Figure 2 As shown, the steps of the method in this embodiment include:

[0054] Step S100: start at least two cameras of a multi-camera device, and use the multi-camera device to synchronously capture RGB videos of different skin areas of a user.

[0055] Specifically, at least two cameras of the multi-camera device are turned on, such as at least two front cameras or rear cameras, or at least one front camera and one rear camera, to enable the shooting function of the multi-camera device, and the cameras corresponding to the turned-on cameras are aimed at different skin areas of the user, and RGB videos of different skin areas of the user are synchronously collected. The skin area can be flexibly selected according to the convenience of using the multi-camera device, or the skin area that is conducive to improving the accuracy of PTT estimation can be selected based on experience, and this embodiment does not make specific restrictions. The camera in the multi-camera device used in this embodiment is an RGB camera, so that the RGB signal is fused by a multi-band fusion noise reduction method, and the RGB signals of different bands or spectral ranges are fused, and the complementarity between these RGB signals is used to reduce noise, and a PPG signal that is robust to motion and illumination changes is generated, which is conducive to enhancing the robustness of PTT estimation. It is easy to understand that the multiple cameras configured by the multi-camera device in this embodiment refer to a smart device configured with multiple cameras that have the function of separately collecting images of target areas.

[0056] In a preferred embodiment, the method of synchronously collecting RGB videos of different skin areas of a user using the multi-camera device includes:

[0057] The multi-camera device is used to synchronously capture the RGB video of the first skin area and the RGB video of the second skin area of ​​the user; the target detection algorithm is used to detect and crop the RGB video of the first skin area and / or the RGB video of the second skin area to obtain the RGB video of the first target area and the RGB video of the second target area.

[0058] Specifically, the smartphone used in this embodiment has the ability to turn on the light, front camera and rear camera at the same time, and the front camera can use structured light or screen light to fill in the light for the first skin area illuminated, and the rear camera can use flash to fill in the light for the second skin area illuminated. Among them, the first skin area includes skin areas such as cheeks, earlobes, and necks, and the second skin area is a hand or foot area such as the fingertips of any finger or toe. Specifically, the first skin area is illuminated by the front camera, and the second skin area is illuminated by the rear camera, and the RGB video of the first skin area and the RGB video of the second skin area of ​​the user are synchronously collected. In the process of collecting RGB videos, the corresponding relationship between video frames is tracked by setting corresponding marks or timestamps to ensure that the two RGB videos are synchronized in time, which is convenient for subsequent processing. For example, in daily use scenarios, the front camera of the smartphone is used to capture RGB videos containing faces; in phone answering scenarios, the front camera of the smartphone is used to capture RGB videos containing earlobes or other skin areas. In both scenarios, the rear camera can be used to capture RGB videos containing only the finger area by placing the fingertips of any finger or toe above the rear camera.

[0059] Considering that in scenarios such as answering a phone call or daily use, the RGB video of the first skin area captured by the front camera may include non-skin areas such as hair, clothes, and decorations, the present embodiment uses a target detection algorithm to detect and crop the RGB video of the first skin area to obtain an RGB video that only includes the first target area such as the face or neck. Since the RGB video of the second skin area can be captured by placing the tip of any finger or toe above the rear camera, the RGB video of the second skin area often only includes the RGB video of the hand or foot area. At this time, there is no need to perform target detection, and the RGB video of the second skin area is regarded as the RGB video of the second target area. If the RGB video of the second skin area includes non-skin areas such as hair, clothes, and decorations, the target detection algorithm can also be used to detect and crop to obtain an RGB video of the second target area that only includes the skin area. It should be stated that the present embodiment uses a smartphone equipped with a front camera and a rear camera as an exemplary embodiment of a multi-camera device, but the smartphone is not intended to limit the scope of protection of the present invention. Any device equipped with at least two cameras that can independently capture images of the target area is within the scope of protection of the present invention.

[0060] It can be seen that this embodiment uses a smart device equipped with multiple cameras, and utilizes at least two cameras to simultaneously collect RGB videos of different skin areas in the same time period during normal use of the smart device. The face and fingertip areas are conveniently collected during the use of the smart device. Therefore, it is preferred to collect RGB videos of the face and / or fingertip areas to achieve seamless collection of the entire PTT estimation process, which can not only reduce the cost of estimating PTT, but also significantly improve the user experience.

[0061] It is worth noting that in order to improve the availability and effectiveness of RGB videos, the RGB videos used in subsequent processing steps default to the RGB videos of the first target area and the second target area obtained in step S100 after preprocessing such as denoising and contrast enhancement.

[0062] Step S200: based on each segment of the RGB video, extract the PPG signal of the corresponding skin area respectively.

[0063] Specifically, based on each segment of the RGB video, a series of RGB images of the corresponding skin area are obtained; based on the pixel value of each pixel in each frame of the RGB image in the skin area, the skin chromaticity change of the corresponding skin area at each pixel is determined; based on the skin chromaticity change of each pixel in each skin area, the PPG signal of the corresponding skin area is determined.

[0064] The RGB video of the first target area and the RGB video of the second target area are framed to obtain several frames of RGB images of the first target area and several frames of RGB images of the second target area. Since the Cb and Cr components in the YCbCr color space are insensitive to changes in illumination and can better reflect changes in skin color, this embodiment converts the RGB image to the YCbCr color space. In the YCbCr space, the Y component represents brightness, and the Cb and Cr components represent chromaticity. PPG signals are usually related to chromaticity changes, so it is necessary to focus on the Cb and Cr components. Then, the pixel values ​​of each pixel in each frame of the image are averaged, the mean of the Cb and Cr components are calculated, and a time series of these average values ​​is constructed over time (i.e., a sequence of frames) to extract the skin chromaticity changes of the corresponding skin area. Finally, based on the skin chromaticity changes of each pixel in each skin area, signal processing techniques such as fast Fourier transform are used to identify peaks in the frequency domain, or bandpass filters are used to extract signals within a specific frequency range to determine periodic changes corresponding to the heartbeat frequency, thereby extracting the PPG signal waveforms of the first target area and the second target area.

[0065] In this embodiment, the frame image in the RGB video is converted into a color space to improve the recognition accuracy of the skin chromaticity change, and based on the skin chromaticity change of each pixel in each skin area, the periodic change corresponding to the heart rate is determined, so as to indirectly obtain the PPG signal of each skin area, thereby realizing the construction of the correlation relationship between the skin chromaticity change and the PPG signal.

[0066] In a preferred embodiment, determining the PPG signal of the corresponding skin area based on the change of the skin chromaticity in each skin area includes:

[0067] Perform global pixel averaging on each frame of the RGB video of the first target area and the second target area to obtain the pixel mean of each pixel in each channel, that is, the pixel mean of the three channels R, G, and B. Then, through color space conversion, the Cb and Cr components of each pixel are used to characterize the skin chromaticity change of the first target area. and skin color changes in the second target area . Calculate the mean of the pixel values ​​of all pixels in the first target area and the second target area respectively, and determine the mean of the skin chromaticity change of the first target area and the second target area and Calculate the skin chromaticity deviation between the skin chromaticity change at each pixel point of each skin area and the corresponding skin chromaticity change mean, and determine the PPG signal of the first target area and the second target area based on the ratio between the skin chromaticity deviation of each pixel point and the corresponding skin chromaticity change mean and ,Right now:

[0068] ,

[0069] in, This formula means that and Perform the DC normalization operation to obtain the optimized and This embodiment uses a DC normalization method to optimize the two-way PPG signal. According to actual application requirements, signal filtering or waveform detrending operations can be used to remove long-term trend components from the signal to make the signal more stable, thereby obtaining an optimized and .

[0070] Step S300: Detect all the peaks of the two PPG signals in the same time window, and obtain a PTT estimation result based on the time relationship between all the peaks of the two PPG signals.

[0071] Specifically, the timestamps of all the peaks of the two PPG signals are calculated; based on the timestamps of all the peaks, the time delays between the nearest neighbor peaks of the two PPG signals are calculated; and based on the average value of all the time delays, the PTT estimation result is obtained.

[0072] For the signal and , use a sliding window (such as a window length of 5-30s) to divide the two PPG signals into multiple segments, and for the segments in the same time window, detect all the peaks in the two PPG signal segments and the timestamps corresponding to each peak, and use all the peaks in one of the PPG signal segments as reference points to locate the nearest lagged peak (i.e., the nearest neighbor peak) in the other PPG signal segment. For example: All the peaks of the fragment are taken as the reference. Find the distance between each The time delay between the peaks of the two PPG signal segments is calculated and the average value is taken as the final estimation result of the pulse wave transmission time (PTT).

[0073] In this embodiment, by collecting all the peaks of the two PPG signals corresponding to the two skin areas in the same time window, the time delay between each group of nearest neighbor peaks is calculated, and the average value of the time delay between the nearest neighbor peaks is used as the final pulse wave conduction time. This method can reduce the impact of signal noise caused by user movement, etc., and improve the accuracy of the time delay between the two PPG signals, which is beneficial to improve the estimation accuracy of the pulse wave conduction time.

[0074] In a preferred embodiment, obtaining a PTT estimation result based on an average value of all the time delays includes:

[0075] The two PPG signals are defined as a first PPG signal and a second PPG signal, and the wavelength of light in the target color channel corresponding to the first PPG signal is longer; the average time interval between the peaks of the first PPG signal and the second PPG signal is calculated respectively to obtain a first average time interval and a second average time interval; if the PTT estimation result is greater than zero, the PTT estimation result is equal to the average value of the time delays of all the nearest neighboring peaks between the second PPG signal and the first PPG signal, and the PTT estimation result is less than the first average time interval; if the PTT estimation result is less than zero, the PTT estimation result is equal to the average value of the time delays of all the nearest neighboring peaks between the first PPG signal and the second PPG signal, and the absolute value of the PTT estimation result is less than the second average time interval.

[0076] Specifically, considering that the two PPG signal waveforms of different skin areas actually measured may cause noise due to the user's movement, the user should try to stay still during the measurement. In addition, there are multiple groups of corresponding peak feature points in the two PPG signal segments in the same time window, and the timestamps of all peak feature points are represented as , Considering that some peaks may deviate from the actual pulse due to noise, the PTT corresponding to the pulse transmission delay between each group of nearest neighbor peaks used to calculate the pulse wave transmission time needs to meet the following constraints, namely:

[0077] when When:

[0078] ,

[0079] in, is an adjustable parameter, ; , represents the PPG signal of all second target areas ( ) is the average interval between wave peaks.

[0080] when When:

[0081] ,

[0082] in, is an adjustable parameter, ; , represents all the first target area PPG signals ( ) is the average interval between wave peaks.

[0083] Therefore, the pulse transmission delay between each group of nearest neighbor peaks in the two PPG signal waveforms within the same time window that meets the above constraints is selected, and the average value of all these pulse transmission delays is calculated as the final pulse wave conduction time.

[0084] In this embodiment, by setting constraints on the pulse transmission delay between each group of nearest neighbor peaks in the two PPG signal waveforms of different skin areas within the same time window, the deviation caused by noise can be effectively avoided, which is beneficial to further improve the accuracy and effectiveness of the pulse wave transmission time finally obtained.

[0085] In a preferred embodiment, after obtaining the PTT estimation result, the method includes:

[0086] The PTT estimation result is input into a trained blood pressure estimation model to obtain the blood pressure estimation result of the user, wherein the blood pressure estimation model is used to reflect the mapping relationship between pulse wave transmission time and blood pressure.

[0087] Specifically, in the training stage of the blood pressure estimation model, common machine learning methods, least squares regression algorithms, etc. are used to construct an initial blood pressure estimation model, which is used to reflect the mapping relationship between pulse wave conduction time and blood pressure. The initial blood pressure estimation model is fitted and calibrated using multimodal data consisting of physiological signals related to blood pressure, such as signal waveform features related to human blood pressure, multi-band PTT features, human heart rate, and PTT features of different skin areas of the human body (including but not limited to the skin of the face and fingertips) in clinical medical data, to obtain a trained blood pressure estimation model. Among them, the signal waveform features related to human blood pressure include but are not limited to the time proportion of systolic and diastolic pressure in the blood pressure waveform, the relative height of the descending gorge, the relative height of the dicrotic wave, the amplitude of the signal peak, and the waveform features of the electrocardiogram, atrial depolarization wave (P wave), ventricular depolarization wave (QRS complex) and atrial repolarization wave (T wave). Obviously, since this implementation uses multimodal data as a training sample for training the initial blood pressure estimation model, the trained blood pressure estimation model has good robustness and generalization ability, which is conducive to improving the accuracy of the model in estimating blood pressure. In the inference stage of the blood pressure estimation model, the actual PTT measured in step S300 is input into the trained blood pressure estimation model to obtain the blood pressure estimation result of the user.

[0088] For example, Figure 3 As shown, for daily use scenarios or phone answering scenarios, the front camera of the smartphone is used to collect RGB video of the face or ear area, and the rear camera is used to collect RGB video of the index fingertip area. The method of the present invention is used to extract the PPG signals of the face or ear area and the PPG signals of the index fingertip area, respectively, and the PTT between each group of nearest neighbor peaks in the same time window is calculated based on the two PPG signals. Finally, the PTT is input into the trained blood pressure estimation model, and the PTT estimation value of the systolic blood pressure (SBP) and the PTT estimation value of the diastolic blood pressure (DBP) are calculated, thereby outputting the blood pressure value.

[0089] Furthermore, various physiological indicators such as heart rate monitoring, emotion recognition, and sleep quality assessment can be monitored based on the PTT estimation results.

[0090] like Figure 4 As shown, corresponding to the above-mentioned PTT estimation method based on a multi-camera device, an embodiment of the present invention further provides a PTT estimation system based on a multi-camera device, and the above-mentioned PTT estimation system based on a multi-camera device includes:

[0091] An information collection module 410 is used to start at least two cameras of a multi-camera device and use the multi-camera device to synchronously collect RGB videos of different skin areas of a user;

[0092] A PPG signal extraction module 420, for extracting PPG signals of corresponding skin areas based on each segment of the RGB video;

[0093] The PTT estimation module 430 is used to detect all the peaks of the two PPG signals in the same time window, and obtain a PTT estimation result based on the time relationship between all the peaks of the two PPG signals.

[0094] Specifically, in this embodiment, the specific functions of the above-mentioned PTT estimation system based on multi-camera equipment can also refer to the corresponding description in the above-mentioned PTT estimation method based on multi-camera equipment, which will not be repeated here.

[0095] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The smart terminal can be used to execute the PTT estimation method based on a multi-camera device provided in the above embodiment. For the sake of brevity, it is not repeated here. The smart terminal includes: a processor, the processor is coupled to a memory, the memory is used to store a computer program or instruction, and the processor is used to execute the computer program or instruction stored in the memory, so that the method in the above method embodiment is executed.

[0096] The present invention also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0097] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0098] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A PTT estimation method based on a multi-camera device, characterized in that: The following steps are involved: Starting at least two cameras of a multi-camera device, and using the multi-camera device to synchronously capture an RGB video of a first skin area and an RGB video of a second skin area of ​​a user; In the process of collecting RGB video, a timestamp is set to track the correspondence between video frames to ensure that the RGB video of the first skin area and the RGB video of the second skin area are synchronized in time, wherein the first skin area includes the cheek, earlobe, and neck skin area, and the second skin area is the hand or foot area; Based on each of the RGB videos, a series of RGB images of the corresponding skin area are obtained; Convert the RGB image to a YCbCr color space to obtain a Cb component and a Cr component of each pixel in each frame of the RGB image; Calculating average values ​​of the Cb component and the Cr component, and constructing a time series of the average values ​​over time; Based on the time series, determining the skin chromaticity change of the corresponding skin area at each pixel point; Calculating the mean of the pixel values ​​of all the pixels in each skin area to determine the mean of the skin chromaticity change of the corresponding skin area; Calculating the skin chromaticity deviation between the skin chromaticity change at each pixel point of each skin area and the corresponding skin chromaticity change mean value, and determining the PPG signal of the corresponding skin area based on the ratio between the skin chromaticity deviation at each pixel point and the corresponding skin chromaticity change mean value; Calculate the timestamps of all peaks of the two PPG signals; Based on the timestamps of all the peaks, calculating the time delay between the nearest neighbor peaks of the two PPG signals; Based on the average of all said time delays, a PTT estimation result is obtained.

2. The PTT estimation method based on multi-camera equipment according to claim 1, characterized in that: After obtaining the PTT estimation result, the method further comprises: The PTT estimation result is input into a trained blood pressure estimation model to obtain the blood pressure estimation result of the user, wherein the blood pressure estimation model is used to reflect the mapping relationship between pulse wave transmission time and blood pressure.

3. The PTT estimation method based on multi-camera equipment according to claim 1, characterized in that: The method of synchronously collecting the RGB video of the first skin area and the RGB video of the second skin area of ​​the user by using the multi-camera device includes: A target detection algorithm is used to detect and crop the RGB video of the first skin area and / or the RGB video of the second skin area to obtain the RGB video of the first target area and the RGB video of the second target area.

4. The PTT estimation method based on multi-camera equipment according to claim 1, characterized in that: The obtaining of a PTT estimation result based on an average value of all the time delays comprises: The two PPG signals are defined as a first PPG signal and a second PPG signal, and the wavelength of light in the target color channel corresponding to the first PPG signal is longer; respectively calculating average time intervals between respective peaks of the first PPG signal and the second PPG signal to obtain a first average time interval and a second average time interval; If the PTT estimation result is greater than zero, the PTT estimation result is equal to an average of time delays of all nearest neighbor peaks between the second PPG signal and the first PPG signal, and the PTT estimation result is less than the first average time interval; If the PTT estimation result is less than zero, the PTT estimation result is equal to the average of the time delays of all nearest neighbor peaks between the first PPG signal and the second PPG signal, and the absolute value of the PTT estimation result is less than the second average time interval.

5. A PTT estimation system based on a multi-camera device, characterized in that: The system comprises: An information acquisition module, used to start at least two cameras of a multi-camera device, and use the multi-camera device to synchronously acquire an RGB video of a first skin area and an RGB video of a second skin area of ​​a user; A PPG signal extraction module, used to extract the PPG signal of the corresponding skin area based on each RGB video segment; A PTT estimation module, used to calculate the timestamps of all the peaks of the two PPG signals; based on the timestamps of all the peaks, calculate the time delay between the nearest neighbor peaks of the two PPG signals; based on the average value of all the time delays, obtain a PTT estimation result; The information collection module comprises: In the process of collecting RGB video, a timestamp is set to track the correspondence between video frames to ensure that the RGB video of the first skin area and the RGB video of the second skin area are synchronized in time, wherein the first skin area includes the cheek, earlobe, and neck skin area, and the second skin area is the hand or foot area; The PPG signal extraction module comprises: obtaining a series of RGB images of the corresponding skin area based on each segment of the RGB video; Convert the RGB image to a YCbCr color space to obtain a Cb component and a Cr component of each pixel in each frame of the RGB image; Calculating average values ​​of the Cb component and the Cr component, and constructing a time series of the average values ​​over time; Based on the time series, determining the skin chromaticity change of the corresponding skin area at each pixel point; Calculating the mean of the pixel values ​​of all the pixels in each skin area to determine the mean of the skin chromaticity change of the corresponding skin area; The skin chromaticity deviation between the skin chromaticity change at each pixel point and the corresponding skin chromaticity change mean of each skin area is calculated, and the PPG signal of the corresponding skin area is determined based on the ratio between the skin chromaticity deviation at each pixel point and the corresponding skin chromaticity change mean.

6. An intelligent terminal, characterized in that: include: A memory for storing executable instructions; A processor is used to call and run the executable instructions in the memory to execute the steps of the PTT estimation method based on a multi-camera device as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by the processor, the PTT estimation method based on a multi-camera device as described in any one of claims 1 to 4 is implemented.

Images