A non-contact blood oxygen saturation measurement method and system

Abstract

The present invention provides a non-contact blood oxygen saturation measurement method and system, which obtains a video signal containing a person to be measured, pre-processes the video signal, extracts the face area therein; tracks the region of interest in the face area, performs three-channel separation on the region of interest, and obtains the original imaging photoplethysmography signal; extracts the red and blue channel signals in the original imaging photoplethysmography signal, and performs filtering processing respectively, and extracts the ratio of the average values ​​of the peak and trough as a characteristic value; and calculates the blood oxygen saturation value according to the characteristic value. The present invention can effectively reduce the influence of motion artifacts, allowing the subject to perform non-contact blood oxygen saturation measurement with a slight shake.

Description

Technical Field

[0001] The invention belongs to the field of measurement technology, and in particular relates to a non-contact blood oxygen saturation measurement method and system based on K-nearest neighbor foreground extraction. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the acceleration of the pace of social work and life and the intensification of aging, the incidence of cardiovascular diseases has also increased year by year. How to accurately and quickly measure physiological parameters closely related to the cardiovascular system, such as blood oxygen saturation, has become increasingly important. At present, the mainstream method for measuring blood oxygen saturation is still contact-based, that is, it is necessary to use professional instruments to place sensors at specific locations on the body surface for measurement. Since the contact detection method has high requirements on the placement of the sensor, the operator's technical level, and the measurement equipment and site, the convenience of measurement is greatly reduced. Therefore, in recent years, many studies have begun to focus on non-contact blood oxygen saturation measurement methods. Imaging Photoplethysmography (IPPG) technology is a non-contact physiological parameter detection technology developed in recent years.

[0004] However, in order to prevent the influence of motion artifacts, the existing non-contact blood oxygen saturation measurement methods based on IPPG technology often require the measured object to remain still in order to obtain a relatively fixed sensitive area and a higher quality photoelectric volume pulse wave. If the measured person moves slightly during the measurement process, motion artifacts will be caused and high-energy noise will be introduced into the detected pulse wave waveform. If this noise cannot be effectively removed, it will be harmful to the blood oxygen saturation measurement. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a non-contact blood oxygen saturation measurement method and system. The present invention can obtain a pulse wave with a higher signal-to-noise ratio through filtering processing on the basis of the original imaging photoplethysmography signal, thereby meeting the requirements for accurate and reliable blood oxygen saturation measurement under the condition of small-amplitude motion artifacts.

[0006] According to some embodiments, the present invention adopts the following technical solutions:

[0007] A non-contact blood oxygen saturation measurement method comprises the following steps:

[0008] Obtain a video signal containing the person to be tested, pre-process the video signal, and extract the face area therein;

[0009] Tracking a region of interest in a face region, performing three-channel separation on the region of interest, and obtaining an original imaging photoplethysmography signal;

[0010] Extract the red and blue channel signals from the original imaging photoplethysmography signal, filter them separately, and extract the ratio of the average values ​​of the peak and trough as the characteristic value;

[0011] Calculate the blood oxygen saturation value based on the characteristic value.

[0012] As an optional implementation, the specific process of preprocessing the video signal includes: converting the video into a series of frame images by using frame processing, and automatically detecting the face in the first frame image by using the AdaBoost algorithm.

[0013] As an optional implementation, the specific process of tracking the area of ​​interest in the face area includes: using KNN foreground extraction to generate an alpha matrix of a video frame, separating the tracking area from each frame in the video sequence, obtaining each subframe, aligning the previous subframe and the current subframe image, calculating the ternary map of the current subframe, combining the ternary map of the previous subframe, performing alpha registration estimation, and calculating the alpha matrix of the current subframe.

[0014] As a further limitation, before registering the previous subframe and the current subframe images, each subframe is first converted to grayscale and then non-rigidly registered.

[0015] As a further limitation, the ternary map calculates a threshold using a multi-level Otsu threshold algorithm, and divides the image into three regions, namely a foreground region, a background region and an unknown region, according to the calculated threshold.

[0016] As a further limitation, after the alpha matrix of the current subframe is calculated, it is recombined with the alpha matrix of the previous frame to obtain the final alpha matrix of the current frame.

[0017] As an optional implementation, when filtering the red and blue channel signals, a bidirectional long short-term memory network filtering model or a long short-term memory network filtering model is used to filter the two-channel waveforms to obtain a pulse wave waveform with a higher signal-to-noise ratio.

[0018] As an optional implementation, the specific process of calculating the blood oxygen saturation value according to the characteristic value includes: measuring the blood oxygen saturation using a dual-wavelength method:

[0019] SPO 2 =A+B·R

[0020]

[0021]

[0022] Among them, A and B are empirical constants, which are obtained by linear fitting with the data measured by the contact oxygen saturation detection device--the finger-clip blood oxygen and meter through breath holding experiment. R is the similar absorption coefficient and the maximum difference absorption coefficient of the two hemoglobins, and D is the IPPG signal under the red channel. λ1 IPPG signal D under the blue channel λ2 The ratio of is the peak average value of the red channel, is the trough average of the red channel, is the peak average value of the blue channel, is the average trough value of the blue channel.

[0023] A non-contact blood oxygen saturation measurement system, comprising:

[0024] A video acquisition unit, configured to acquire a video signal containing a person to be tested;

[0025] The video processing module is configured to pre-process the video signal to extract the face area therein; track the region of interest in the face area, perform three-channel separation on the region of interest, and obtain the original imaging photoplethysmography signal;

[0026] The feature value extraction module is configured to extract the red and blue channel signals from the original imaging photoplethysmography signal, perform filtering processing on them respectively, and extract the ratio of the average values ​​of the peak and the trough as the feature value;

[0027] The calculation module is configured to calculate the blood oxygen saturation value according to the characteristic value.

[0028] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0029] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention uses the KNN-based foreground extraction technology to track the target area, which can effectively reduce the influence of motion artifacts and allow the subject to perform non-contact blood oxygen saturation measurement in the case of slight shaking.

[0032] The present invention uses a BLSTM network structure to filter the original IPPG signal. Due to the fitting process of regressing the input signal and the standard signal, a pulse wave signal with a higher signal-to-noise ratio can be obtained while retaining most of the information of the signal waveform, such as the dicrotic wave.

[0033] Compared with traditional filtering methods, such as wavelet transform, the present invention can obtain a better pulse wave result, which is more beneficial for non-contact blood oxygen saturation detection.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 is a schematic diagram of the measurement system of this embodiment;

[0037] Figure 2 Schematic diagram of the measurement process of this embodiment;

[0038] Figure 3 is a flow chart of the target tracking algorithm of this embodiment;

[0039] Figure 4 is a schematic diagram of the BLSTM network algorithm of this embodiment, the entire information flow of the signal is in state C, and the input and output values ​​of the signal are in h;

[0040] Figure 5 : is a BLSTM filter network structure diagram of this embodiment;

[0041] Figure 6 The BLSTM network training and testing model and parameter annotations for each layer in this embodiment;

[0042] Among them, 1 is a fill light unit, 2 is a video acquisition unit, 3 is a video signal processing unit, and 4 is a subject;

[0043] i is frame processing, ii is the AdaBoost face detection algorithm used, iii is the KNN foreground extraction algorithm, iv is the Butterworth bandpass filter for filtering, the BLSTM algorithm is used for regression fitting, and v is the eigenvalue, which is the ratio of the average value of the peak and trough of the red channel to the average value of the peak and trough of the blue channel. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] Embodiment 1:

[0048] like Figure 2 As shown, a non-contact blood oxygen saturation measurement method, after obtaining the video stream, needs to use frame processing to convert the video into a series of frame images, uses the AdaBoost algorithm to automatically detect the face in the first frame image, and then tracks the region of interest (ROI). The present invention uses the KNN foreground extraction algorithm to track the selected region of interest. Since there are more capillaries on both sides of the nose and cheek area of ​​the human face and less occlusion, this area is selected as the region of interest. The region needs to be as stable as possible when the measured object shakes. After the ROI area is determined, three-channel separation is required to obtain the original IPPG signals of the red, green and blue channels. Subsequently, the red and blue two-channel signals need to be filtered to obtain a pulse wave with a higher signal-to-noise ratio. The present invention uses the BLSTM algorithm to filter the two-channel waveform. After obtaining the pulse wave waveform with a higher signal-to-noise ratio, the ratio of the average value of the peak and trough is extracted as a characteristic value, and substituted into the empirical formula for blood oxygen saturation measurement to obtain the blood oxygen saturation value.

[0049] Fast-tracking methods to achieve goals such as Figure 3 As shown. Let F β (m t ,n t) is the color intensity of the input video at positions m = 0, 1, ..., M and n = 0, 1, ..., N and time t, where β represents the red, green and blue bands, i.e. β∈{R,G,B}. To initialize the matching process, the alpha matrix of the first frame is calculated by giving it and its user-generated target region as input. KNN foreground extraction is applied to generate the alpha values ​​(αmatte) of the video frames so that pixels with similar appearance have similar alpha values. With this in mind, the feature vector is defined:

[0050] F(m,n)=[cos(h),sin(h),s,v,m,n] (1)

[0051] Where h, s and v represent HSV coordinates. The similarity between two pixels is calculated by defining a kernel function:

[0052]

[0053] where C is the least upper bound of ||F(m,n)-F(m1,n1)||.

[0054] In order to reduce the computational complexity, the moving area is tracked and separated from the video sequence. The present invention uses L1 tracker for real-time tracking. Assume A β (m t ,n t ) is the center of the window containing the target, A β The position at frame (t+1) is:

[0055]

[0056] where ξ represents the L1 tracker, λ = [λ a ,λ b ,λ l ] represent the target template, trivial template and l respectively 2 norm parameter. τ is the number of iterations of the accelerated proximal gradient (APG) method used to solve the minimization model for the L1 tracker. ι represents the number of samples used to sparsely represent the target and trivial template. θ is the bias of the samples in the particle filter frame, which provides an estimate of the target in the frame. is the number of particles, and s represents the size of the template.

[0057] The tracking region is separated from each frame in the video sequence to obtain subframe X β (m t ,n t ). From now on, determining the video image target area will only be performed in subframes, which, after being added back, are recombined with the alpha matrix of the previous frame to obtain the final alpha value of the current frame.

[0058] The user needs to provide the TRIMAP for the first frame and use image registration to estimate the TRIMAP for the next frame. To do this, each subframe is first converted to grayscale Z(m t ,n t ). Then a non-rigid registration is performed. The transformation is calculated as:

[0059]

[0060] Where p = [m t ]-1,q=[n t ]-1,u=m t -[m t ],v=n t -[n t ]. x and B y is the basis function defined, and non-rigid registration provides control points that define the degree of non-rigid deformation

[0061] TRIMAP divides the image into three regions, namely the foreground region is marked as 1, the background region is marked as 0, and the unknown region is marked as 0.5. Therefore, in the estimated TRIMAP, T(m t ,n t ) Two thresholds are obtained through the multi-level Otsu threshold algorithm, namely:

[0062] (γ 1 ,γ 2 )=Δ[T(m t ,n t ),2] (5)

[0063] Where Δ represents a multi-level threshold operation.

[0064] The threshold obtained by the multi-level Otsu threshold algorithm is used to divide TRIMAP into three regions. β (m t ,n t ) and the corresponding TRIMAP, T(m t ,n t ) is passed to the extraction process to estimate the α value α of the corresponding subframe d (m t ,n t The final α value α(m t ,n t ):

[0065] α(m t ,n t )=α d (m t ,n t )+α(m t-1 ,nt-1 ) (6)

[0066] The second is the training process of the BLSTM filter model. The BLSTM filter model was trained and tested using the Multimodal Spontaneous Emotion Corpus-Heart Rate Database (MSEC-HRD), which takes emotion recognition into account, so the video contains a lot of facial movements and expressions. Each video is 30 to 60 seconds long, with a resolution of 1040×1392 pixels and a frame rate of 25 frames per second. The real signal is a pulse wave signal with a sampling rate of 1000Hz obtained by the BIOPAC150 data acquisition system. The training model structure diagram is shown in the figure below. Figure 4 As shown. The pulse wave signal obtained from the video signal is used as input and put into the BLSTM training model, and the output is the pulse wave signal measured by the data acquisition system. Assume that the original IPPG signal size is [b, L, 1], the first layer parameters of the training model are units = 125, Dropout = 0.2, RS = T (RS means return sequences), the second layer parameters are units = 125, Dropout = 0.2, RS = T (RS means return sequences), the third layer parameters are units = 125, Dropout = 0.2, RS = F (RS means return sequences) The last layer is a fully connected layer with a parameter of 125 and an output signal size of [b, L, 1]. Figure 6 The arrow in the middle indicates the size of the signal transmitted between layers, and b is the batch size. The trained model can be used to filter the pulse wave signal.

[0067] You can first choose to filter using a Butterworth or Chebyshev bandpass filter, and then use the trained BLSTM regression model to filter the signal.

[0068] The LSTM network structure is as follows Figure 4 As shown. The key to LSTM lies in the state of the cell (the range of the dotted box 1) and the horizontal line above the cell. The cell state is similar to a conveyor belt, and it is easy for information to flow unchanged on it. However, it is impossible to add or delete information with only the horizontal line above, but it is achieved through the gate structure. The gate can selectively let information pass, mainly through a sigmoid neural layer and a point-by-point multiplication operation (dashed box 2). LSTM uses three such basic structures to protect and control information. These three gates are input gate, forget gate and output gate.

[0069] The calculation formula of the LSTM network structure is:

[0070] f t =σ(W f ·[h t-1 ,x t ]+b f ) (7)

[0071] i t =σ(W i ·[h t-1 ,x t ]+b i ) (8)

[0072]

[0073]

[0074] O t =σ(W O ·[h t-1 ,x t ]+b O ) (11)

[0075] h t =O t *tanh(C t ) (12)

[0076] Among them, x t is the current input, h t-1 is the hidden state of the previous unit, C t-1 is the cell state of the previous unit, h t and C t are the hidden state and cell state of the current unit.

[0077] After filtering the original pulse wave signal using the BLSTM structure, a signal with a higher signal-to-noise ratio is obtained, and the red and blue channel signals are extracted, and the blood oxygen saturation is measured using the dual-wavelength method. The empirical formula is:

[0078] SPO 2 =A+B·R (13)

[0079]

[0080]

[0081] Among them, A and B are empirical constants, which are obtained by linear fitting with the data measured by the breath-holding experiment and the contact oxygen saturation detection device (such as the finger-clip blood oxygen and meter). The calculation of parameter R is to find the similar absorption coefficient and the maximum difference absorption coefficient of the two hemoglobins. Reflected in the imaging photoplethysmography technology, it is necessary to determine the IPPG signal under the corresponding two wavelength light channels. Extract the characteristic values ​​of the two channels for calculation. is the peak average value of the red channel, is the trough average of the red channel, is the peak average value of the blue channel, is the average trough value of the blue channel.

[0082] When setting up the system, it is necessary to simultaneously capture video and measure blood oxygen saturation with standard instruments so that linear fitting can be performed based on standard values.

[0083] Embodiment 2:

[0084] A non-contact blood oxygen saturation measurement system includes a video acquisition unit. A subject stands in front of the system and the video acquisition unit acquires a video. Figure 1 As shown, during the acquisition process, a fill light unit can be added to perform a fill light operation. After obtaining a stable video stream, the video analysis unit analyzes the video signal to obtain a blood oxygen saturation value.

[0085] It also includes a video signal processing unit for executing the signal processing steps of the first embodiment.

[0086] Embodiment three:

[0087] The only difference between the third embodiment and the first embodiment is that the BLSTM is replaced by the LSTM.

[0088] They should be considered as the same invention. When training the BLSTM network, the present invention uses a four-layer network structure for training, and 125 units are used. If there is a plan to only change the number of network layers and the number of units, it should be considered as the same invention. When extracting the pulse wave characteristic value, the present invention uses the ratio of the peak and trough values ​​of the red and blue channels as the characteristic value. If there is a plan to use statistical data such as the average and standard deviation of the red and blue channels as the characteristic value, it should be considered as the same invention.

[0089] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A non-contact blood oxygen saturation measurement method, characterized in that: The following steps are involved: Obtain a video signal containing the person to be tested, pre-process the video signal, and extract the face area therein; Tracking a region of interest in a face region, performing three-channel separation on the region of interest, and obtaining an original imaging photoplethysmography signal; The region of interest is the sides of the nose and cheek area of ​​the face; The specific process of tracking the region of interest in the face region includes: using KNN foreground extraction to generate alpha values ​​of video frames so that pixels with similar appearance have similar alpha values, and defining a feature vector: F(m,n)=[cos(h),sin(h),s,v,m,n] Wherein, m=0, 1, ..., M and n=0, 1, ..., N represent the position of the input video; h, s and v represent HSV coordinates; Define the kernel function to calculate the similarity between two pixels: Among them, C is the minimum upper bound of ||F(m,n)-F(m1,n1)||; (m,n)(m1,n1) are two pixels; Using L1 tracker for real-time tracking, let A β (m t ,n t ) is the center of the window containing the target, A β The position at frame (t+1) is: where ξ represents the L1 tracker, λ = [λ a ,λ b ,λ l ] represent the target template, trivial template and l2 norm parameter respectively; τ is the number of iterations of the accelerated proximal gradient method, which is used to solve the minimization model of the L1 tracker; ι represents the number of samples used to sparsely represent the target and trivial template; θ is the deviation of the samples in the particle filter framework, which provides an estimate of the target in the frame; is the number of particles, s represents the size of the template; The tracking area is separated from each frame in the video sequence to obtain each subframe, the previous subframe and the current subframe image are registered, the ternary map of the current subframe is calculated, and the alpha registration estimation is performed in combination with the ternary map of the previous subframe to calculate the alpha matrix of the current subframe. After the alpha matrix of the current subframe is calculated, it is recombined with the alpha matrix of the previous frame to obtain the final alpha matrix of the current frame; Before registering the previous subframe and the current subframe images, each subframe is first converted to grayscale and then non-rigidly registered; The ternary map calculates a threshold using a multi-level Otsu threshold algorithm, and divides the image into three regions, namely a foreground region, a background region, and an unknown region, according to the calculated threshold; Extract the red and blue channel signals from the original imaging photoplethysmography signal, filter them separately, and extract the ratio of the average values ​​of the peak and trough as the characteristic value; when filtering the red and blue channel signals, first use the Butterworth bandpass filter for filtering, and then use the four-layer bidirectional long short-term memory network filtering model to filter the two channel waveforms to obtain a pulse wave waveform with a high signal-to-noise ratio; Calculate the blood oxygen saturation value based on the characteristic value.

2. The non-contact blood oxygen saturation measurement method according to claim 1, characterized in that: The specific process of preprocessing the video signal includes: using frame processing to convert the video into a series of frame images, and using the AdaBoost algorithm to automatically detect the face in the first frame image.

3. The non-contact blood oxygen saturation measurement method according to claim 1, characterized in that: The specific process of calculating the blood oxygen saturation value according to the characteristic value includes: measuring the blood oxygen saturation using the dual wavelength method: SPO2=A+B·R Among them, A and B are empirical constants, which are obtained by linear fitting with the data measured by the contact oxygen saturation detection device--the finger-clip blood oxygen and meter through breath holding experiment. R is the similar absorption coefficient and the maximum difference absorption coefficient of the two hemoglobins, and D is the IPPG signal under the red channel. λ1 IPPG signal D under the blue channel λ2 The ratio of is the peak average value of the red channel, is the trough average of the red channel, is the peak average value of the blue channel, is the average trough value of the blue channel.

4. A non-contact blood oxygen saturation measurement system, using the non-contact blood oxygen saturation measurement method according to claim 1, characterized in that: include: A video acquisition unit, configured to acquire a video signal containing a person to be tested; The video processing module is configured to pre-process the video signal and extract the face area therein; Tracking a region of interest in a face region, performing three-channel separation on the region of interest, and obtaining an original imaging photoplethysmography signal; The feature value extraction module is configured to extract the red and blue channel signals from the original imaging photoplethysmography signal, perform filtering processing on them respectively, and extract the ratio of the average values ​​of the peak and the trough as the feature value; The calculation module is configured to calculate the blood oxygen saturation value according to the characteristic value.

5. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a non-contact blood oxygen saturation measurement method as described in any one of claims 1 to 3.

6. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the non-contact blood oxygen saturation measurement method according to any one of claims 1 to 3 are completed.

Images