Blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction

Abstract

The present invention discloses a blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction. The method selects an area of ​​interest from a video frame of a user's facial video, divides the area of ​​interest of each video frame in turn, and determines each local area of ​​interest; screens each local area of ​​interest of the video frame based on the R channel signal and the G channel signal of each local area of ​​interest to obtain each candidate area of ​​interest; determines the target area of ​​interest corresponding to each video frame according to each candidate area of ​​interest; uses an amplitude ratio method to calculate the R channel signal and the G channel signal of the target area of ​​interest to calculate the peripheral blood oxygen saturation of the video frame, thereby determining the user's blood oxygen change trend according to the peripheral blood oxygen saturation of each video frame. The method solves the problem in the related art that the accuracy of peripheral blood oxygen saturation estimation is affected due to the different penetration depths of red light and green light in the skin.

Description

Technical Field

[0001] The present invention relates to the field of biomedical engineering, and in particular to a blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction. Background Art

[0002] In current remote health testing, peripheral blood oxygen saturation (SpO 2 ) is crucial for the physiological health assessment of the human body.

[0003] In the related technology, the technology of non-contact optical physiological detection of peripheral blood oxygen saturation based on cameras has developed rapidly, making continuous and non-invasive detection of various physiological parameters possible. In the field of non-contact health detection, non-contact research on blood oxygen includes: using a combination of red light and infrared wavelengths to detect peripheral blood oxygen saturation through the amplitude ratio method (RR, Ratio to Ratio). Due to the deep penetration depth of red light and infrared wavelengths, errors caused by differences in the penetration depth of photons of different wavelengths can be effectively avoided, but there is no active infrared light source in many usage scenarios, which limits the use of this method. Another method is to use a combination of green light and red light in visible light for detection through a dual ratio method, but due to the different penetration depths of red light and green light in the skin, the accuracy of peripheral blood oxygen saturation estimation is affected. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction are provided, aiming to solve the problem in the related art that the accuracy of peripheral blood oxygen saturation estimation is affected by the different penetration depths of red light and green light in the skin.

[0005] The technical solution adopted by the present invention to solve the problem is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a blood oxygen detection method based on multi-band pulse conduction, wherein the method comprises:

[0007] Obtain a facial video corresponding to the user, select a region of interest for each video frame in the facial video, and determine an initial region of interest;

[0008] Sequentially divide the initial regions of interest of each video frame in the facial video to determine each local region of interest;

[0009] Screening each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame, determining each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range;

[0010] Determine a target region of interest of each video frame according to each candidate region of interest of the video frame;

[0011] The amplitude ratio method is used to determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target region of interest of each video frame.

[0012] In one implementation method, the initial region of interest of each video frame in the facial video is divided to determine each local region of interest, including:

[0013] Slide the preset sliding window on the initial region of interest of the video frame, and perform mean filtering on the R channel information and the G channel of the initial region of interest located in the preset sliding window respectively;

[0014] The area where the initial region of interest is located within the preset sliding window is taken as the local region of interest.

[0015] In one implementation method, each local region of interest of each video frame is screened based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame to determine each candidate region of interest, including:

[0016] Performing band-pass filtering on the R channel signal and the G channel signal of each local region of interest to determine the R channel local pulse signal and the G channel local pulse signal;

[0017] Calculate the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference;

[0018] The local regions of interest are screened according to the first local phase differences to determine candidate regions of interest.

[0019] In one implementation method, calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine the first local phase difference includes:

[0020] Performing window function weighting and Fourier transformation on the R channel local pulse signal and the G channel local pulse signal of each local region of interest to obtain the R channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain;

[0021] Obtain a main frequency index, and search on the G channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain based on the main frequency index to determine the R channel local main frequency and the G channel local main frequency;

[0022] The phase difference between the local main frequency of the R channel and the local main frequency of the G channel is calculated to determine a first local phase difference.

[0023] In one implementation method, a method for obtaining a primary frequency index includes:

[0024] Calculate the average value of the R channel signal and the G channel signal of the initial region of interest respectively to determine the global region of interest;

[0025] Perform Butterworth filtering and Fourier transform on the global region of interest to determine the global pulse signal in the frequency domain;

[0026] The frequency index corresponding to the maximum amplitude of the global pulse signal in the frequency domain is used as the main frequency index.

[0027] In one implementation method, determining a target region of interest of each video frame according to each candidate region of interest of the video frame includes:

[0028] Combining the candidate regions of interest corresponding to each video frame to determine a combined region of interest corresponding to the video frame;

[0029] Obtain each mask combination, process the combined region of interest using each mask combination, and determine each processed combined region of interest;

[0030] The phase differences of the R channel signal and the G channel signal of each processed combined region of interest are calculated respectively to determine each second local phase difference;

[0031] The processed combined regions of interest are selected based on the second local phase differences to determine a target region of interest.

[0032] In one implementation method, a method for obtaining each mask combination includes:

[0033] Randomly generate a preset number of masks;

[0034] A preset number of masks multiplied by 70% are selected from the preset number of masks for permutation and combination to determine each mask combination.

[0035] In a second aspect, an embodiment of the present invention further provides a blood oxygen detection device based on multi-band pulse conduction, wherein the blood oxygen detection device based on multi-band pulse conduction includes:

[0036] An initial region of interest determination module is used to obtain a facial video corresponding to the user, select a region of interest for each video frame in the facial video, and determine an initial region of interest;

[0037] A local region of interest division module is used to sequentially divide the initial region of interest of each video frame in the facial video to determine each local region of interest;

[0038] A local region of interest screening module is used to screen each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame, and determine each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range;

[0039] A target region of interest determination module, used to determine a target region of interest of each video frame according to each candidate region of interest of the video frame;

[0040] The peripheral blood oxygen saturation calculation module is used to determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target area of ​​interest of each video frame by using the amplitude ratio method.

[0041] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-mentioned blood oxygen detection methods based on multi-band pulse conduction; and the processor is used to execute the program.

[0042] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement any of the above-mentioned blood oxygen detection methods based on multi-band pulse conduction.

[0043] Beneficial effects of the present invention: The embodiment of the present invention selects an area of ​​interest for a video frame of a user's facial video, divides the area of ​​interest of each video frame in turn, and determines each local area of ​​interest; screens each local area of ​​interest of the video frame based on the R channel signal and the G channel signal of each local area of ​​interest to obtain each candidate area of ​​interest; determines the target area of ​​interest corresponding to each video frame according to each candidate area of ​​interest; calculates the peripheral blood oxygen saturation of the video frame by using the amplitude ratio method to calculate the R channel signal and the G channel signal of the target area of ​​interest, thereby determining the blood oxygen change trend of the user according to the peripheral blood oxygen saturation of each video frame. The problem in the related art that the accuracy of peripheral blood oxygen saturation estimation is affected by the different penetration depths of red light and green light in the skin is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1It is a flow chart of a blood oxygen detection method based on multi-band pulse conduction provided in an embodiment of the present invention.

[0046] Figure 2 It is a flowchart of a specific embodiment of a blood oxygen detection method based on multi-band pulse conduction provided in an embodiment of the present invention.

[0047] Figure 3 It is a schematic diagram of the penetration paths of light waves of different wavelengths through the skin provided by an embodiment of the present invention.

[0048] Figure 4 Schematic diagram of different mask combinations provided by an embodiment of the present invention.

[0049] Figure 5 It is a schematic diagram of the multi-wavelength pulse transmission time difference of the same combination of interest regions after the action of different mask combinations provided by an embodiment of the present invention.

[0050] Figure 6 It is a schematic diagram of the amplitude ratio of the same combination of regions of interest after different mask combinations are applied, provided by an embodiment of the present invention.

[0051] Figure 7 2 is a diagram of an experimental setup provided by an embodiment of the present invention.

[0052] Figure 8 It is a schematic diagram of the linear regression results of peripheral blood oxygen saturation and amplitude ratio provided by an embodiment of the present invention.

[0053] Fig. 9 It is a schematic diagram of peripheral blood oxygen saturation detection results of different users provided by an embodiment of the present invention.

[0054] Fig.10 It is a schematic diagram of the internal modules of the blood oxygen detection device based on multi-band pulse conduction provided by an embodiment of the present invention.

[0055] Fig.11 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention discloses a blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0059] In the related technology, the technology of non-contact optical physiological detection of peripheral blood oxygen saturation based on cameras has developed rapidly, making continuous and non-invasive detection of various physiological parameters possible. In the field of non-contact health detection, non-contact research on blood oxygen includes: using a combination of red light and infrared wavelengths to detect peripheral blood oxygen saturation through the amplitude ratio method. Due to the deep penetration depth of red light and infrared wavelengths, errors caused by differences in the penetration depth of photons of different wavelengths can be effectively avoided, but there is no active infrared light source in many usage scenarios, which limits the use of this method. Another method is to use a combination of green light and red light in visible light for detection through a dual ratio method, but due to the different penetration depths of red light and green light in the skin, the accuracy of peripheral blood oxygen saturation estimation is affected.

[0060] In view of the above-mentioned defects of the prior art, the present invention provides a blood oxygen detection method based on multi-band pulse conduction, wherein the method selects an area of ​​interest from a video frame of a user's facial video, divides the area of ​​interest of each video frame in turn, and determines each local area of ​​interest; screens each local area of ​​interest of the video frame based on the R channel signal and the G channel signal of each local area of ​​interest to obtain each candidate area of ​​interest; determines the target area of ​​interest corresponding to each video frame according to each candidate area of ​​interest; calculates the peripheral blood oxygen saturation of the video frame by using the amplitude ratio method to calculate the R channel signal and the G channel signal of the target area of ​​interest, thereby determining the blood oxygen change trend of the user according to the peripheral blood oxygen saturation of each video frame. The problem in the related art that the accuracy of peripheral blood oxygen saturation estimation is affected by the different penetration depths of red light and green light in the skin is solved.

[0061] Exemplary methods:

[0062] like Figure 1 As shown, the method includes:

[0063] Step S100: Obtain a facial video corresponding to the user, select a region of interest for each video frame in the facial video, and determine an initial region of interest.

[0064] Specifically, when the user needs to detect changes in blood oxygen, an RGB camera (three-primary color camera) with a notch filter (blocking light with a wavelength of 580-605nm from passing through) can be placed above the user's location to shoot the user's face from a bird's-eye view to obtain the user's corresponding facial video. Each shooting time can be 10 to 15 minutes. In order to reduce the processing of invalid data, the region of interest is selected for each video frame of the facial video to obtain the region of interest of each video frame. All subsequent processing is based on the region of interest (ROI) to improve the efficiency of data processing. At the same time, in order to obtain the user's blood oxygen information based on the facial video, the region of interest is generally set to a skin area such as the forehead or cheek with a complete area and sufficient capillaries.

[0065] Furthermore, in order to reduce the amount of data processing and improve the efficiency of detecting the user's blood oxygen, a facial video segment with valid data can be selected from the user's corresponding facial video, and then a region of interest can be selected based on the facial video segment, and then subsequent processing can be performed based on the region of interest to obtain the user's blood oxygen change trend. Considering that the user may not be stable at the beginning of the video shooting and may shake, the facial video segment selects a video frame 10 seconds after the start frame of the facial video to ensure the stability of the selected region of interest in the subsequent video to ensure signal quality.

[0066] Step S200: sequentially divide the initial region of interest of each video frame in the facial video to determine each local region of interest.

[0067] In simple terms, Figure 2 As shown, the region of interest of each video frame in the facial video is processed in sequence by sliding the first preset sliding window on the facial video. For example, the first preset sliding window size is set to 15 seconds or 900 frames, and the step size is 1 second or 60 frames to slide on the facial video frame, and the region of interest of the video frame in the first preset sliding window is divided in sequence to obtain each local region of interest corresponding to each video frame.

[0068] In one implementation, the initial region of interest of each video frame in the facial video is divided to determine each local region of interest, including:

[0069] Slide the preset sliding window on the initial region of interest of the video frame, and perform mean filtering on the R channel information and the G channel of the initial region of interest located in the preset sliding window respectively;

[0070] The area where the initial region of interest is located within the preset sliding window is taken as the local region of interest.

[0071] Specifically, the preset sliding window acting on the region of interest is the second preset sliding window. The second preset sliding window can be set to a size of 11×11 (pixels) with a step size of 1. The second preset sliding window is slid on the region of interest, and the R channel information (red light channel information) and G channel information (green light channel information) corresponding to the region of interest located in the second preset sliding window are mean filtered. At the same time, the region in the second preset sliding window is used as the local region of interest.

[0072] Step S300, screening each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame, determining each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range.

[0073] Specifically, Figure 3As shown, the R channel signal represents the signal after the red light is reflected by the skin layer, and the G channel signal represents the signal after the green light is reflected by the skin. In peripheral blood oxygen saturation monitoring based on RGB cameras, large errors occurred when estimating the peripheral blood oxygen saturation on multiple subjects using red light wavelengths and green light wavelengths, and studies have shown that different penetration depths are one of the main reasons for the different accuracy of peripheral blood oxygen saturation estimation results under the amplitude ratio method (mainly reflected in the difference in skin thickness). Therefore, solving the problem of the difference in penetration depth between red light wavelengths and green light wavelengths is crucial to improving the accuracy and consistency of peripheral blood oxygen saturation estimation based on RGB cameras.

[0074] The larger the multi-wavelength pulse transmission time (MW-PTT), the more significant the difference in the penetration depth of photons of different wavelengths in the skin. However, when the pulse wave signal generated by red and green wavelengths is used to calculate the multi-wavelength pulse transmission time, the results are often unstable. Sometimes the multi-wavelength pulse transmission time is so small that it can be ignored, and sometimes it is so obvious that it can be used for blood pressure prediction. This shows that in some skin areas, the penetration depths of red and green wavelengths are very close, resulting in extremely small or close to zero multi-wavelength pulse transmission time. This instability is due to the differences in penetration of different wavelengths of light into different skin types and areas. In the estimation of peripheral blood oxygen saturation, the consistency of peripheral blood oxygen saturation prediction between different skin areas of the same individual and the consistency between individuals with different skin characteristics (such as skin color, thickness, etc.) can be achieved by minimizing this depth difference, that is, minimizing the multi-wavelength pulse transmission time. Therefore, by selecting those skin areas where the multi-wavelength pulse transmission time is minimal or close to zero for peripheral blood oxygen saturation prediction, the amplitude ratio method error caused by differences in photon penetration depth and individual differences can be reduced, ultimately improving the accuracy of peripheral blood oxygen saturation estimation.

[0075] This embodiment screens each local region of interest based on the R channel signal and the G channel signal to obtain each candidate region of interest, ensuring that the difference in multi-wavelength pulse transmission time corresponding to the R channel signal and the G channel signal in the candidate region of interest is within a preset range (relatively small), so that the R channel signal and the G channel signal can be extracted based on the candidate region of interest to calculate the peripheral blood oxygen saturation.

[0076] In one implementation, each local region of interest of each video frame is screened based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame to determine each candidate region of interest, including:

[0077] Step S301, performing bandpass filtering on the R channel signal and the G channel signal of each local region of interest to determine the R channel local pulse signal and the G channel local pulse signal;

[0078] Step S302, calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference;

[0079] Step S303: Screen the local regions of interest according to the first local phase differences to determine candidate regions of interest.

[0080] Specifically, when the acquired red light signal and green light signal are reflected back by the same skin layer, the multi-wavelength pulse transmission time corresponding to the pulse signal in the red light signal is approximately the same as the multi-wavelength pulse transmission time corresponding to the pulse signal in the green light signal. The multi-wavelength pulse transmission time can be represented by the phase of each light signal in the frequency domain. Therefore, firstly, the R channel signal and the G channel signal of each local region of interest are band-pass filtered to extract the R channel local pulse signal corresponding to the R channel of each local region of interest and the G channel local pulse signal corresponding to the G channel. This embodiment adopts a band-pass filter of 0.7Hz-3Hz (Hertz); then, the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest is calculated to obtain a first local phase difference. The first local phase difference is used to reflect whether the red light signal and the green light signal in the local region of interest are reflected by the same skin layer, so that each local region of interest is screened according to the first local phase difference. When the first local phase difference corresponding to a local region of interest is greater than π or less than 0.003, it indicates that the penetration depth difference between red light and green light in the local region of interest is large or the noise influence leads to a large estimation error, and the invalid local region of interest is removed; when the first local phase difference corresponding to a local region of interest is less than π and greater than 0.003, the local region of interest is used as the initial region of interest. This embodiment ensures higher quality and more consistent signals for the calculation of peripheral blood oxygen saturation by screening the local regions of interest.

[0081] In one implementation, calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine the first local phase difference includes:

[0082] Step S3021, performing window function weighting and Fourier transform on the R channel local pulse signal and the G channel local pulse signal of each local region of interest to obtain the R channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain;

[0083] Step S3022, obtaining a main frequency index, searching on the G channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain based on the main frequency index, and determining the R channel local main frequency and the G channel local main frequency;

[0084] Step S3023: Calculate the phase difference between the local main frequency of the R channel and the local main frequency of the G channel to determine a first local phase difference.

[0085] Specifically, a Hanning window is used to perform window function weighting on the R channel local pulse signal and the G channel local pulse signal of each region of interest to reduce spectrum leakage and improve spectrum resolution; a fast Fourier transform is used to convert the R channel local pulse signal and the G channel local pulse signal in the time domain into the frequency domain respectively; based on the main frequency index, the main frequencies corresponding to the R channel local pulse signal and the G channel local pulse signal in the frequency domain are found on the R channel local pulse signal and the G channel local pulse signal in the frequency domain to obtain the R channel local main frequency and the G channel local main frequency; an angle function is used to calculate the phases of the R channel local main frequency and the G channel local main frequency on the main frequency, so as to obtain the first local phase difference based on the phase difference between the R channel local main frequency and the G channel local main frequency.

[0086] In one implementation, a method for obtaining a primary frequency index includes:

[0087] Calculate the average value of the R channel signal and the G channel signal of the initial region of interest respectively to determine the global region of interest;

[0088] Perform Butterworth filtering and Fourier transform on the global region of interest to determine the global pulse signal in the frequency domain;

[0089] The frequency index corresponding to the maximum amplitude of the global pulse signal in the frequency domain is used as the main frequency index.

[0090] Specifically, the pixel averages of the R channel and the G channel of the region of interest of each video frame are calculated respectively to obtain the global region of interest corresponding to each video frame; the global region of interest is subjected to Butterworth filtering to remove the interference caused by camera jitter, electronic noise, ambient light, breathing, and non-pulse signals in the global region of interest, and the global pulse signal is extracted from the global region of interest after Butterworth filtering, and the global pulse signal is Fourier transformed to convert the global region of interest from the time domain to the frequency domain to obtain the global pulse signal in the frequency domain; the frequency index of the maximum amplitude is found from the global pulse signal in the frequency domain, that is, the main frequency index. In one implementation, since the G channel is usually stable and of high quality and can effectively reflect the pulse activity of an individual, the G channel signal in the global pulse signal is Fourier transformed to find the frequency index of the maximum amplitude as the main frequency index.

[0091] Step S400: determining a target ROI of each video frame according to each candidate ROI of the video frame.

[0092] Simply put, each candidate region of interest is a region where the pulse transmission time corresponding to red light and green light is the smallest (that is, the phase difference is the smallest). By combining the candidate regions of interest, a combined region of interest with the smallest pulse transmission time corresponding to red light and green light can be constructed. The target region of interest corresponding to the video frame can be determined based on the combined region of interest.

[0093] In one implementation, determining a target region of interest of each video frame according to each candidate region of interest of the video frame includes:

[0094] Step S401, combining candidate regions of interest corresponding to each video frame to determine a combined region of interest corresponding to the video frame;

[0095] Step S402, obtaining each mask combination, and processing the combined region of interest using each mask combination respectively, to determine each processed combined region of interest;

[0096] Step S403, respectively calculating the phase difference of the R channel signal and the G channel signal of each processed combined region of interest to determine each second local phase difference;

[0097] Step S404: selecting each processed combined region of interest based on each second local phase difference to determine a target region of interest.

[0098] Specifically, first, each candidate region of interest is combined to obtain a combined region of interest with the minimum phase difference (pulse transmission time) of the pulse signal of red light and green light. Each mask combination is obtained, and the combined region of interest is processed by the mask combination so that the R channel signal and the G channel signal in the processed combined region of interest are returned through the same skin layer. The R channel signal and the G channel signal are extracted from the processed combined region of interest, and the phase difference of the R channel signal and the G channel signal is calculated, specifically including: performing window function weighting and Fourier transform on the R channel pulse signal corresponding to the R channel signal of the processed combined region of interest and the G channel pulse signal corresponding to the G channel signal to obtain the R channel pulse signal in the frequency domain and the G channel pulse signal in the frequency domain; obtaining the main frequency index, searching on the R channel pulse signal in the frequency domain and the G channel pulse signal in the frequency domain based on the main frequency index, and determining the main frequency of the R channel pulse signal in the frequency domain and the main frequency of the G channel pulse signal in the frequency domain; calculating the phase difference between the main frequency of the R channel pulse signal in the frequency domain and the main frequency of the G channel pulse signal in the frequency domain, and determining the second local phase difference. After processing the combined ROI with different mask combinations, different processed combined ROIs are obtained. Therefore, the second local phase differences corresponding to the processed combined ROIs are also different. The processed combined ROI with the smallest second local phase difference is selected as the target ROI, such as Figure 4 , Figure 5 shown.

[0099] The method for obtaining each mask combination includes: randomly generating a preset number of masks; selecting a preset number of masks multiplied by 70% of the masks from the preset number of masks for permutation and combination to determine each mask combination. For example, considering that 70% of red light and green light come from the same skin layer, 10 groups of masks are randomly generated through a Monte Carlo simulation experiment, and 120 groups of mask combinations are generated according to the number of combinations of 7 selected from 10 in the permutation and combination.

[0100] Step S500: Determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target region of interest of each video frame using an amplitude ratio method.

[0101] In simple terms, the R channel signal and the G channel signal in the target region of interest come from the same skin layer. At this time, the amplitude ratio method can be used to calculate the peripheral blood oxygen saturation based on the R channel signal and the G channel signal in the target region of interest as the peripheral blood oxygen saturation of the video frame where the target region of interest is located, such as Figure 6 shown.

[0102] The peripheral blood oxygen saturation is calculated based on the R channel signal and the G channel signal in the target region of interest using an amplitude ratio method, including:

[0103] ;

[0104] is the peripheral blood oxygen saturation, and is a constant obtained through experiment or calibration, is the amplitude ratio;

[0105] ;

[0106] is the intensity of the R channel signal, is the intensity of the G channel signal, It is the reference value of the light intensity ratio of the R channel signal and the G channel signal.

[0107] In one implementation, after the peripheral blood oxygen saturation corresponding to each video frame is obtained, the blood oxygen change trend of the user can be determined according to the peripheral blood oxygen saturation corresponding to each video frame in the facial video.

[0108] In simple terms, a linear regression is performed on the peripheral blood oxygen saturation corresponding to each video frame of the facial video to obtain the user's blood oxygen change trend. In addition, a linear regression can also be performed on the amplitude ratio to observe the user's blood oxygen change trend.

[0109] Based on the above embodiments, the present invention conducted a specific experiment, and the experimental process includes: the user wears a pulse oximeter on one hand to obtain standard peripheral blood oxygen saturation, and wears a detection bracelet (various bracelets or watches that can detect blood oxygen) on the other hand as a reference aid for cross-validation with the data obtained by the pulse oximeter; only when the readings of the two devices are consistent, the true value of the peripheral blood oxygen saturation is considered valid. Under ambient light conditions without specific lighting control, an RGB camera with a notch filter is used to capture the user's facial video at a bird's-eye view, and each recording lasts 10 to 15 minutes. The specific experimental setting is shown in the figure below. Figure 7 After experiments and comparison with the existing blood oxygen detection method, the method of this embodiment has a better detection effect, and its effect diagram is shown in Figure 8 , Fig. 9 shown.

[0110] Based on the above embodiments, the present invention also provides a blood oxygen detection device based on multi-band pulse conduction, such as Fig.10 As shown, the device comprises:

[0111] The initial region of interest determination module 01 is used to obtain a facial video corresponding to the user, select a region of interest for each video frame in the facial video, and determine an initial region of interest;

[0112] The local region of interest division module 02 is used to sequentially divide the initial region of interest of each video frame in the facial video to determine each local region of interest;

[0113] The local region of interest screening module 03 is used to screen each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame, and determine each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range;

[0114] A target region of interest determination module 04 is used to determine a target region of interest of each video frame according to each candidate region of interest of the video frame;

[0115] The peripheral blood oxygen saturation calculation module 05 is used to determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target region of interest of each video frame by adopting the amplitude ratio method.

[0116] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Fig.11As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a blood oxygen detection method based on multi-band pulse conduction is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0117] Those skilled in the art will understand that Fig.11 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one implementation, one or more programs are stored in the memory of the terminal, and the terminal is configured to be executed by one or more processors. The one or more programs include instructions for performing a blood oxygen detection method based on multi-band pulse conduction.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] In summary, the present invention discloses a blood oxygen detection method, device, terminal and medium based on multi-band pulse conduction. The method selects an area of ​​interest from a video frame of a user's facial video, divides the area of ​​interest of each video frame in turn, and determines each local area of ​​interest; screens each local area of ​​interest of the video frame based on the R channel signal and the G channel signal of each local area of ​​interest to obtain each candidate area of ​​interest; determines the target area of ​​interest corresponding to each video frame according to each candidate area of ​​interest; calculates the peripheral blood oxygen saturation of the video frame by using the amplitude ratio method to calculate the R channel signal and the G channel signal of the target area of ​​interest, thereby determining the blood oxygen change trend of the user according to the peripheral blood oxygen saturation of each video frame. The problem in the related art that the accuracy of peripheral blood oxygen saturation estimation is affected by the different penetration depths of red light and green light in the skin is solved.

[0121] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A blood oxygen detection method based on multi-band pulse conduction, characterized in that: The method comprises: Obtaining a facial video corresponding to the user, selecting a region of interest for each video frame in the facial video, and determining an initial region of interest; Sequentially dividing the initial region of interest of each video frame in the facial video to determine each local region of interest; Screening each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame to determine each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range; Determine a target region of interest of each video frame according to each candidate region of interest of the video frame; Determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target region of interest of each video frame by using an amplitude ratio method; Screening the local regions of interest of each video frame based on the R channel signal and the G channel signal corresponding to the local regions of interest of each video frame to determine candidate regions of interest includes: Performing bandpass filtering on the R channel signal and the G channel signal of each local region of interest to determine the R channel local pulse signal and the G channel local pulse signal; Calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference; Screening each of the local regions of interest according to each of the first local phase differences to determine each of the candidate regions of interest; Calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference includes: Performing window function weighting and Fourier transformation on the R channel local pulse signal and the G channel local pulse signal of each local region of interest to obtain the R channel local pulse signal and the G channel local pulse signal in the frequency domain; Obtain a main frequency index, and search the G channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain based on the main frequency index to determine the R channel local main frequency and the G channel local main frequency; The phase difference between the R channel local main frequency and the G channel local main frequency is calculated to determine the first local phase difference.

2. The blood oxygen detection method based on multi-band pulse conduction according to claim 1, characterized in that: Dividing the initial region of interest of each video frame in the facial video to determine each local region of interest includes: Sliding on the initial region of interest of the video frame based on a preset sliding window, performing mean filtering on the R channel information and the G channel of the initial region of interest located in the preset sliding window respectively; The area where the initial region of interest is located within the preset sliding window is taken as the local region of interest.

3. The blood oxygen detection method based on multi-band pulse conduction according to claim 1, characterized in that: The method for obtaining the main frequency index includes: Calculating average values ​​of the R channel signal and the G channel signal of the initial region of interest respectively to determine a global region of interest; Performing Butterworth filtering and Fourier transform on the global region of interest to determine a global pulse signal in the frequency domain; The frequency index corresponding to the maximum amplitude of the global pulse signal in the frequency domain is used as the main frequency index.

4. The blood oxygen detection method based on multi-band pulse conduction according to claim 1, characterized in that: Determining a target region of interest of each video frame according to each candidate region of interest of the video frame includes: Combining the candidate regions of interest corresponding to each video frame to determine a combined region of interest corresponding to the video frame; Acquire each mask combination, respectively use each mask combination to process the combined region of interest, and determine each processed combined region of interest; Respectively calculating the phase difference between the R channel signal and the G channel signal of each of the processed combined regions of interest to determine each second local phase difference; The processed combined regions of interest are selected based on the second local phase differences to determine a target region of interest.

5. The blood oxygen detection method based on multi-band pulse conduction according to claim 4, characterized in that: Methods for obtaining each mask combination include: Randomly generate a preset number of masks; A preset number of masks multiplied by 70% of the preset number of masks are selected from the preset number of masks for arrangement and combination to determine each mask combination.

6. A blood oxygen detection device based on multi-band pulse conduction, characterized in that: The device comprises: An initial region of interest determination module is used to obtain a facial video corresponding to the user, select a region of interest for each video frame in the facial video, and determine an initial region of interest; A local region of interest division module is used to sequentially divide the initial region of interest of each video frame in the facial video to determine each local region of interest; A local region of interest screening module is used to screen each local region of interest of each video frame based on the R channel signal and the G channel signal corresponding to each local region of interest of the video frame, and determine each candidate region of interest, wherein the difference in multi-wavelength pulse transmission time between the R channel signal and the G channel signal in the candidate region of interest is within a preset threshold range; A target region of interest determination module, configured to determine a target region of interest of each video frame according to each of the candidate regions of interest of the video frame; A peripheral blood oxygen saturation calculation module, used to determine the peripheral blood oxygen saturation corresponding to each video frame based on the R channel signal and the G channel signal of the target region of interest of each video frame by using an amplitude ratio method; Screening the local regions of interest of each video frame based on the R channel signal and the G channel signal corresponding to the local regions of interest of each video frame to determine candidate regions of interest includes: Performing bandpass filtering on the R channel signal and the G channel signal of each local region of interest to determine the R channel local pulse signal and the G channel local pulse signal; Calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference; Screening each of the local regions of interest according to each of the first local phase differences to determine each of the candidate regions of interest; Calculating the phase difference between the R channel local pulse signal and the G channel local pulse signal of each local region of interest to determine a first local phase difference includes: Performing window function weighting and Fourier transformation on the R channel local pulse signal and the G channel local pulse signal of each local region of interest to obtain the R channel local pulse signal and the G channel local pulse signal in the frequency domain; Obtain a main frequency index, and search the G channel local pulse signal in the frequency domain and the G channel local pulse signal in the frequency domain based on the main frequency index to determine the R channel local main frequency and the G channel local main frequency; The phase difference between the R channel local main frequency and the G channel local main frequency is calculated to determine the first local phase difference.

7. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the blood oxygen detection method based on multi-band pulse conduction as described in any one of claims 1-5; and the processor is used to execute the program.

8. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are loaded and executed by the processor to implement the steps of the blood oxygen detection method based on multi-band pulse conduction as described in any one of claims 1 to 5 above.

Images