Physiological parameter detection method and device based on RGB-NIR combination, terminal and storage medium

Abstract

The invention relates to the technical field of biomedical engineering. The invention discloses a physiological parameter detection method and device based on RGB-NIR combination, a terminal and a storage medium. The accuracy of physiological parameter detection can be improved. The physiological parameter detection method based on RGB-NIR combination comprises the following steps: acquiring a target RGB signal and a first target NIR signal corresponding to the face of a target object; performing correction processing on the first target NIR signal by adopting a first preset algorithm to obtain a second target NIR signal; performing fusion processing on the target RGB signal and the second target NIR signal to obtain a fusion signal; heart rate feature extraction processing is conducted on the fusion signal, and the target heart rate of the target object is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomedical engineering. More particularly, the present application relates to a physiological parameter detection method and device based on RGB-NIR combination, a terminal and a storage medium. BACKGROUND

[0002] The existing physiological parameter (e.g. heart rate and / or respiratory rate) detection method usually collects multiple frames of RGB images, extracts RGB signals (which can be understood as pulse wave signals) from the multiple frames of RGB images, and then detects physiological parameters from the RGB signals. However, in actual application, the imaging quality of the RGB images is disturbed by factors such as changes in environmental illumination, shadow disturbance and motion artifacts, resulting in poor quality of the RGB signals extracted from the multiple frames of RGB images, so that the finally detected physiological parameters deviate from the actual values, reducing the accuracy of detecting physiological parameters. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a physiological parameter detection method and device based on RGB-NIR combination, a terminal and a storage medium, which can improve the accuracy of detecting physiological parameters. The embodiments of the present application mainly achieve the purpose by the following technical solutions: In a first aspect, the present application provides a physiological parameter detection method based on RGB-NIR combination, comprising: obtaining a target RGB signal and a first target NIR signal corresponding to a face of a target object; correcting the first target NIR signal using a first preset algorithm to obtain a second target NIR signal; fusing the target RGB signal and the second target NIR signal to obtain a fused signal; extracting a heart rate feature from the fused signal to obtain a target heart rate of the target object.

[0004] According to an embodiment of the present application, the step of obtaining the first target NIR signal comprises: sequentially obtaining multiple frames of original images of the target object, wherein the odd-numbered frames of images in the multiple frames of original images are all RGB images, and the even-numbered frames of images are all NIR images; adjusting the pixel values of all NIR images using the pixel values of all RGB images to obtain a first target image set; extracting a NIR signal from the first target image set to obtain the first target NIR signal.

[0005] According to an embodiment of the present application, the exposure times corresponding to the odd-numbered frames of images and the even-numbered frames of images in the multiple frames of original images are different.

[0006] According to one embodiment of the present application, the step of adjusting the pixel values of all NIR images by using the pixel values of all RGB images to obtain the first target image set comprises: dividing the plurality of original images into a plurality of image subsets in time sequence, each image subset comprising one odd frame image and one even frame image; performing pixel calculation processing on each pixel of the odd frame image in each image subset according to a preset ratio to obtain a to-be-adjusted pixel corresponding to each pixel of the odd frame image in each image subset; in each image subset, performing calculation processing on the to-be-adjusted pixels corresponding to a first pixel of the even frame image and a second pixel of the odd frame image according to a second preset algorithm to obtain a target pixel corresponding to the first pixel of the even frame image in each image subset, wherein the first pixel is any one pixel of the even frame image in each image subset, and the second pixel is a pixel in the odd frame image of each image subset located at the same pixel position as the first pixel; constructing a modified even frame image corresponding to each image subset by using the target pixels corresponding to all pixels of the even frame image in each image subset; constructing the first target image set by using the modified even frame images corresponding to all image subsets.

[0007] According to one embodiment of the present application, the first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm.

[0008] According to one embodiment of the present application, the step of performing fusion processing on the target RGB signal and the second target NIR signal to obtain a fusion signal comprises: performing fusion processing on the signal of at least one channel in the target RGB signal and the second target NIR signal by using a chrominance method or a pulse blood volume method to obtain a fusion signal.

[0009] According to one embodiment of the present application, the physiological parameter detection method based on RGB-NIR combination further comprises: obtaining a third target NIR signal corresponding to the chest of the target object; performing respiratory frequency extraction processing on the third target NIR signal to obtain a target respiratory frequency of the target object.

[0010] In a second aspect, the embodiment of the present application provides a physiological parameter detection device based on RGB-NIR combination, comprising: an acquisition module configured to acquire a target RGB signal and a first target NIR signal corresponding to the face of a target object; a correction module, configured to correct the first target NIR signal by using a first preset algorithm to obtain a second target NIR signal; a fusion module, configured to fuse the target RGB signal and the second target NIR signal to obtain a fusion signal; a heart rate feature extraction module, configured to extract a heart rate feature from the fusion signal to obtain a target heart rate of the target object.

[0011] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory to execute the steps of the physiological parameter detection method based on RGB-NIR combination provided in the first aspect of the present application.

[0012] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, configured to store a computer program, and the computer program causes a computer to execute the steps of the physiological parameter detection method based on RGB-NIR combination provided in the first aspect of the present application.

[0013] The beneficial effects of the embodiments of the present application include: The embodiments of the present application add the second target NIR signal which is corrected by the first preset algorithm on the basis of the target RGB signal. Since the second target NIR signal is not sensitive to the interference of factors such as light change, shadow disturbance and / or motion artifact of the environment, the second target NIR signal has strong stability. The embodiments of the present application use the strong stability characteristics of the second target NIR signal to compensate for the poor quality of the target RGB signal. Specifically, the target RGB signal and the second target NIR signal are fused into a fusion signal. Then, the target heart rate of the target object is extracted from the fusion signal. Thus, compared with the prior art of extracting physiological parameters by using only RGB signals, the embodiments of the present application can reduce the detection deviation of physiological parameters (i.e. target heart rate) and improve the accuracy of detecting physiological parameters. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 The flowchart of the physiological parameter detection method based on RGB-NIR combination of the present application in some embodiments; Figure 2 A principle block diagram of the physiological parameter detection device based on RGB-NIR combination in some embodiments of the present application; Figure 3 A principle block diagram of the terminal device in some embodiments of the present application. DETAILED DESCRIPTION

[0016] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0017] It should be noted that the terms "first", "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0018] The term "exemplary" or "for example" or the like is used to mean serving as an example, instance, or illustration. Any implementation described herein as "exemplary" or "for example" should not be construed as being preferred or advantageous over other implementations. The term "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0019] The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or devices.

[0020] Unless otherwise defined, all technical and scientific terms used in the specification of the present application have the same meaning as commonly understood by one skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The term "and / or" used in the specification of the present application includes any and all combinations of one or more related listed items.

[0021] The specific embodiments of the present application are further described below in conjunction with the drawings.

[0022] Reference Figure 1 As shown in FIG. 1, a flow chart of a physiological parameter detection method based on RGB-NIR combination is provided in the first aspect of the embodiments of the present application. Figure 1 In the embodiments of the present application, the physiological parameter detection method based on RGB-NIR combination comprises steps S1, S2, S3 and S4.

[0023] S1, obtaining a target RGB signal and a first target NIR signal corresponding to a face of a target object.

[0024] In the embodiments of the present application, R of RGB represents red, G represents green, and B represents blue. NIR (Near Infrared) is near-infrared light.

[0025] Further, the step of obtaining the first target NIR signal comprises: obtaining a plurality of frames of original images of the target object in time sequence, wherein odd-numbered frames of images in the plurality of frames of original images are all RGB images, and even-numbered frames of images are all NIR images; adjusting pixel values of all the NIR images using pixel values of all the RGB images to obtain a first target image set; and performing NIR signal extraction processing on the first target image set to obtain the first target NIR signal.

[0026] It should be understood that, in the embodiments of the present application, two different exposure times are adopted, and the same camera is used to alternately collect the RGB images and the NIR images. In this way, the embodiments of the present application can maintain a high signal-to-noise ratio under various ambient light conditions, effectively improve the monitoring accuracy of heart rate and breathing rate, and at the same time, retain a good visual picture. In other embodiments, different cameras can be used to collect the RGB images and the NIR images, which can be set by those skilled in the art according to actual needs. The camera is an RGB camera.

[0027] Further, the exposure times corresponding to the odd-numbered frames of images and the even-numbered frames of images in the plurality of frames of original images are different. In other embodiments, the exposure times corresponding to the odd-numbered frames of images and the even-numbered frames of images in the plurality of frames of original images can also be the same.

[0028] The odd-numbered frames of images in the plurality of frames of original images adopt a long exposure time, and the even-numbered frames of images in the plurality of frames of original images adopt a short exposure time. For example, the long exposure time can be 20 milliseconds, and the short exposure time can be 5 milliseconds. In other embodiments, the specific values of the long exposure time and the short exposure time can be set by those skilled in the art according to actual needs.

[0029] The long exposure time can be a conventional exposure time.

[0030] The use of the long exposure time can fully capture the slight changes of the skin reflection of the target object under ambient light, which is crucial for detecting physiological signals such as heart rate, respiration, etc.

[0031] When the camera uses the short exposure time, the near-infrared light source needs to be turned on to irradiate the target object, so as to obtain the NIR image. Due to the high intensity of the near-infrared light source and the short distance between the near-infrared light source and the target object, if a conventional exposure time is used, the collected NIR image will be overexposed. To avoid this situation, the embodiments of the present application can control the camera through a program to capture the reflection image containing the near-infrared light source using the short exposure time, so as to ensure that the collected NIR image is not overexposed, and the image reflection mainly comes from the near-infrared light.

[0032] Further, the step of adjusting the pixel values of all the NIR images by using the pixel values of all the RGB images to obtain the first target image set comprises: dividing the plurality of original images into a plurality of image subsets in time sequence, each image subset comprising one odd frame image and one even frame image; performing pixel calculation processing on each pixel of the odd frame image in each image subset according to a preset ratio to obtain the to-be-adjusted pixel corresponding to each pixel of the odd frame image in each image subset; in each image subset, performing calculation processing on the to-be-adjusted pixels corresponding to the first pixel of the even frame image and the second pixel of the odd frame image according to a second preset algorithm to obtain the target pixel corresponding to the first pixel of the even frame image in each image subset, wherein the first pixel is any one pixel of the even frame image in each image subset, and the second pixel is a pixel in the odd frame image of each image subset which is located at the same pixel position as the first pixel; constructing the modified even frame image corresponding to each image subset by using the target pixels corresponding to all the pixels of the even frame image in each image subset; and constructing the first target image set by using the modified even frame images corresponding to all the image subsets.

[0033] Each image subset belongs to one acquisition cycle. The first frame of each image subset is an RGB image, and the second frame is an NIR image.

[0034] Further, the step of performing pixel calculation processing on each pixel of the odd frame image in each image subset according to a preset ratio to obtain the to-be-adjusted pixel corresponding to each pixel of the odd frame image in each image subset comprises: multiplying each pixel of the RGB image in each image subset by to obtain the to-be-adjusted pixel corresponding to each pixel of the RGB image in each image subset. It can be understood that the preset ratio is In other embodiments, the preset ratio can also be other ratios, which can be set by those skilled in the art according to actual needs.

[0035] Further, in each image subset, the corresponding to-be-adjusted pixels of the first pixels of the even-numbered frame images and the second pixels of the odd-numbered frame images are calculated and processed according to a second preset algorithm to obtain the target pixels corresponding to the first pixels of the even-numbered frame images in each image subset, wherein the first pixels are any one of the pixels of the even-numbered frame images in each image subset, and the second pixels are the pixels in the same pixel position as the first pixels in the odd-numbered frame images of each image subset. The step includes: in each image subset, subtracting the corresponding to-be-adjusted pixels of the second pixels of the odd-numbered frame images from the first pixels of the even-numbered frame images to obtain the target pixels corresponding to the first pixels of the even-numbered frame images in each image subset. This step can eliminate the ambient light signal common to the two frames of images in each image subset to obtain a relatively pure NIR image.

[0036] Further, the modified even-numbered frame images corresponding to all image subsets are sequentially formed into the first target image set.

[0037] All images in the first target image set are NIR images, that is, near-infrared light images.

[0038] Further, the step of performing NIR signal extraction processing on the first target image set to obtain the first target NIR signal includes: performing face recognition processing on each image in the first target image set by using a face detection method to obtain a first region of interest of each image in the first target image set; estimating a first target feature of each pixel point in the first region of interest of each image in the first target image set in a plurality of first predetermined directions between consecutive frames by using an optical flow method; performing accumulation and calculation processing on all first target features of each pixel point in the first region of interest of each image in the first target image set in each first predetermined direction between consecutive frames to obtain a first displacement signal corresponding to each first predetermined direction of each pixel point in the first region of interest of each image in the first target image set; calculating the average value of all first displacement signals of each first predetermined direction to obtain a first physiological motion signal corresponding to each first predetermined direction; and performing fusion processing on the first physiological motion signals of all first predetermined directions to obtain the first target NIR signal.

[0039] The face detection method can be YuNet, which is a lightweight, fast, and accurate face detection method. In other embodiments, the face detection method can also be other methods, which can be set by a person skilled in the art according to actual needs.

[0040] The plurality of first predetermined directions are the horizontal direction and the vertical direction. In other embodiments, a person skilled in the art can set other directions according to actual needs.

[0041] The first target feature is a velocity component. In other embodiments, a person skilled in the art can set it to other features according to actual needs, such as an acceleration component.

[0042] Further, the step of obtaining the target RGB signal comprises: sequentially constructing a second target image set from all odd frame images in the plurality of original images; performing face recognition processing on each image in the second target image set by using the face detection method to obtain a second region of interest of each image in the second target image set; estimating a second target feature of each pixel point in the second region of interest of each image in the second target image set in a plurality of second predetermined directions between consecutive frames by using the optical flow method; performing accumulation and calculation processing on all second target features of each pixel point in the second region of interest of each image in the second target image set in each second predetermined direction between consecutive frames to obtain a second displacement signal corresponding to each second predetermined direction of each pixel point in the second region of interest of each image in the second target image set; calculating the average value of all second displacement signals of each second predetermined direction to obtain a second physiological motion signal corresponding to each second predetermined direction; and performing fusion processing on the second physiological motion signals of all second predetermined directions to obtain the target RGB signal.

[0043] The plurality of second predetermined directions are horizontal and vertical directions. In other embodiments, a person skilled in the art can set other directions according to actual needs.

[0044] The second target feature is a velocity component. In other embodiments, a person skilled in the art can set it to other features according to actual needs, such as an acceleration component.

[0045] The target RGB signal comprises R channel signals, G channel signals and B channel signals. S2, performing correction processing on the first target NIR signal by using a first preset algorithm to obtain a second target NIR signal.

[0046] The first target NIR signal and the target RGB signal have a time difference therebetween, more specifically, a periodic offset in time. If the two signals are directly fused, the time axis will be misaligned, affecting the signal phase consistency, and thus reducing the accuracy of the central rate feature extraction of the fused signal. Therefore, the first target NIR signal needs to be corrected.

[0047] The first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm. In other embodiments, the first preset algorithm can also be other algorithms, which can be set by a person skilled in the art according to actual needs.​

[0048] The use of the cubic spline interpolation algorithm or the linear interpolation algorithm can increase the sampling frequency by up to 2 times. Regarding the new signal sequence obtained after interpolation (i.e., the second target NIR signal), the "new points" generated by the interpolation are located precisely near the timestamp positions of the RGB image.

[0049] The second target NIR signal is a signal that is time-aligned with the target RGB signal. Acquiring the second target NIR signal can eliminate the half-frame time delay caused by time-division multiplexing.

[0050] Further, when the first preset algorithm is a cubic spline interpolation algorithm, step S2 includes: recording all time points of the first target NIR signal as a first time point set; generating a second time point set based on the first time point set; performing calculations on each time point in the first time point set and each time point in the second time point set according to the cubic spline interpolation formula to obtain the signal value corresponding to each time point in the second time point set; and connecting the signal values ​​corresponding to all time points in the first time point set and the signal values ​​corresponding to all time points in the second time point set in chronological order to obtain the second target NIR signal.

[0051] Further, the step of generating a second set of time points based on the first set of time points includes: adding the first and second time points in the first set of time points and dividing by 2 to obtain the first time point in the second set of time points; adding the second and third time points in the first set of time points and dividing by 2 to obtain the second time point in the second set of time points; adding the third and fourth time points in the first set of time points and dividing by 2 to obtain the third time point in the second set of time points; and so on, adding the (N-1)th and Nth time points in the first set of time points and dividing by 2 to obtain the (N-1)th time point in the second set of time points. N is the length of the third set of time points.

[0052] Furthermore, the calculation formula for obtaining the signal value corresponding to each time point in the second time point set by performing calculations on each time point in the first time point set and each time point in the second time point set according to the cubic spline interpolation formula is as follows: ; in, It is the second time point in the set. The signal value corresponding to each time point; It is the first time point in the set. The first coefficient corresponding to each time point; is the second coefficient corresponding to the first time point in the first time point set; is the second coefficient corresponding to the first time point in the first time point set; is the second time point in the second time point set; is the second time point in the second time point set; is the second time point in the second time point set; is the second time point in the second time point set; is the third coefficient corresponding to the first time point in the first time point set; is the third coefficient corresponding to the first time point in the first time point set; is the third coefficient corresponding to the first time point in the first time point set. is the third coefficient corresponding to the first time point in the first time point set. is the length of the third time point set.

[0053] All the first coefficients, all the second coefficients, all the third coefficients, and all the fourth coefficients are preset by those skilled in the art according to actual needs.

[0054] Further, in the case that the first preset algorithm is a linear interpolation algorithm, the S2 step includes: recording all the time points of the first target NIR signal as a third time point set; generating a fourth time point set based on the third time point set; calculating and processing each time point in the fourth time point set, each time point in the third time point set, and the signal value corresponding to each time point in the third time point set according to a linear interpolation formula, to obtain the signal value corresponding to each time point in the fourth time point set; connecting the signal values corresponding to all the time points in the third time point set and the signal values corresponding to all the time points in the fourth time point set in time sequence, to obtain the second target NIR signal.

[0055] Further, the step of generating a fourth time point set based on the third time point set includes: obtaining the first time point in the fourth time point set by adding the first time point and the second time point in the third time point set and dividing the sum by 2; obtaining the second time point in the fourth time point set by adding the second time point and the third time point in the third time point set and dividing the sum by 2; obtaining the third time point in the fourth time point set by adding the third time point and the fourth time point in the third time point set and dividing the sum by 2; and so on, obtaining the N-1th time point in the fourth time point set by adding the N-1th time point and the Nth time point in the third time point set and dividing the sum by 2.

[0056] Further, the calculation formula for calculating and processing each time point in the fourth time point set, each time point in the third time point set, and the signal value corresponding to each time point in the third time point set according to a linear interpolation formula to obtain the signal value corresponding to each time point in the fourth time point set is:​ ; in, It is the fourth time point set mentioned above. The signal values ​​corresponding to each time point, the length of the fourth time point set is... ; It is the third time point set The signal value corresponding to each time point; It is the third time point set The signal value corresponding to each time point; It is the third time point set A point in time; It is the third time point set A point in time; It is the fourth time point set mentioned above. A specific point in time. .

[0057] S3. Perform fusion processing on the target RGB signal and the second target NIR signal to obtain a fused signal.

[0058] Further, step S3 includes: fusing the signal of at least one channel of the target RGB signal and the second target NIR signal using the chromaticity-based method (CHROM) or the pulse blood volume method (PBV) to obtain a fused signal.

[0059] More specifically, the R channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the G channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the B channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the R channel signal and the G channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the R channel signal and the B channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the B channel signal and the G channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal; or the R channel signal, the G channel signal and the B channel signal in the target RGB signal and the second target NIR signal can be fused by using the chroma method or the pulse blood volume method to obtain a fused signal.

[0060] Further, the step of fusing the R channel signal, the G channel signal and the B channel signal in the target RGB signal and the second target NIR signal by using the chroma method to obtain a fused signal comprises: extracting a first chroma component, a second chroma component and a target chroma change rate from the R channel signal, the G channel signal and the B channel signal of the target RGB signal; calculating and processing the first chroma component, the second chroma component and the target chroma change rate according to a third preset algorithm to obtain a first weight coefficient; calculating and processing the first weight coefficient according to a fourth preset algorithm to obtain a second weight coefficient; and fusing the R channel signal, the G channel signal, the B channel signal of the target RGB signal and the second target NIR signal based on the first weight coefficient and the second weight coefficient to obtain a fused signal.

[0061] Further, the step of extracting the first chroma component, the second chroma component and the target chroma variation rate from the R channel signal, the G channel signal and the B channel signal of the target RGB signal comprises: dividing the target RGB signal into at least three RGB sub-signals; extracting the corresponding chroma maximum value and the corresponding chroma minimum value of each RGB sub-signal from each RGB sub-signal; calculating the corresponding chroma component of each RGB sub-signal based on the corresponding chroma maximum value and the corresponding chroma minimum value of each RGB sub-signal; taking the corresponding chroma component of the first RGB sub-signal in the at least three RGB sub-signals as the first chroma component; taking the corresponding chroma component of the second RGB sub-signal in the at least three RGB sub-signals as the second chroma component; taking the maximum chroma component among the corresponding chroma components of all RGB sub-signals as the first target component; taking the minimum chroma component among the corresponding chroma components of all RGB sub-signals as the second target component; and obtaining the target chroma variation rate by subtracting the second target component from the first target component.

[0062] The step of extracting the corresponding chroma maximum value and the corresponding chroma minimum value of each RGB sub-signal from each RGB sub-signal can be implemented by using the existing technology.

[0063] Further, taking the first RGB sub-signal as an example, if the red channel value is greater than the green channel value and the blue channel value in the corresponding chroma maximum value of the first RGB sub-signal, the calculation formula of the step of calculating the corresponding chroma component of the first RGB sub-signal based on the corresponding chroma maximum value and the corresponding chroma minimum value of the first RGB sub-signal is: ; wherein, is the corresponding chroma component of the first RGB sub-signal, i.e., the first chroma component; is the green channel value in the corresponding chroma maximum value of the first RGB sub-signal; is the blue channel value in the corresponding chroma maximum value of the first RGB sub-signal; is the corresponding chroma maximum value of the first RGB sub-signal; is the corresponding chroma minimum value of the first RGB sub-signal.

[0064] Further, if the green channel value is greater than the red channel value and the blue channel value in the corresponding chroma maximum value of the first RGB sub-signal, the calculation formula of the step of calculating the corresponding chroma component of the first RGB sub-signal based on the corresponding chroma maximum value and the corresponding chroma minimum value of the first RGB sub-signal is: ; wherein, is the red channel value in the chroma maximum value corresponding to the first segment of RGB sub-signals.

[0065] Further, in the chroma maximum value corresponding to the first segment of RGB sub-signals, if the blue channel value is greater than the red channel value and the green channel value, the calculation formula of the step of calculating the chroma component corresponding to the first segment of RGB sub-signals based on the chroma maximum value and the chroma minimum value corresponding to the first segment of RGB sub-signals is: .

[0066] Further, taking the second segment of RGB sub-signals as an example, in the chroma maximum value corresponding to the second segment of RGB sub-signals, if the red channel value is greater than the green channel value and the blue channel value, the calculation formula of the step of calculating the chroma component corresponding to the second segment of RGB sub-signals based on the chroma maximum value and the chroma minimum value corresponding to the second segment of RGB sub-signals is: ; wherein, is the second chroma component corresponding to the second segment of RGB sub-signals; is the green channel value in the chroma maximum value corresponding to the second segment of RGB sub-signals; is the blue channel value in the chroma maximum value corresponding to the second segment of RGB sub-signals; is the chroma maximum value corresponding to the second segment of RGB sub-signals; is the chroma minimum value corresponding to the second segment of RGB sub-signals.

[0067] Further, in the chroma maximum value corresponding to the second segment of RGB sub-signals, if the green channel value is greater than the red channel value and the blue channel value, the calculation formula of the step of calculating the chroma component corresponding to the second segment of RGB sub-signals based on the chroma maximum value and the chroma minimum value corresponding to the second segment of RGB sub-signals is: ; wherein, is the red channel value in the chroma maximum value corresponding to the second segment of RGB sub-signals.

[0068] Further, in the chroma maximum value corresponding to the second segment of RGB sub-signals, if the blue channel value is greater than the red channel value and the green channel value, the calculation formula of the step of calculating the chroma component corresponding to the second segment of RGB sub-signals based on the chroma maximum value and the chroma minimum value corresponding to the second segment of RGB sub-signals is: .

[0069] Further, the first chroma component, the second chroma component and the target chroma change rate are calculated according to a third preset algorithm, and a calculation formula of the step of obtaining the first weight coefficient is: ; ; ; wherein, is a standard chroma change rate; is the target chroma change rate; is a first weight total of R channel signals, G channel signals and B channel signals in the target RGB signal; is the first weight coefficient.

[0070] Further, the first weight coefficient is calculated according to a fourth preset algorithm, and a calculation formula of the step of obtaining a second weight coefficient is: .

[0071] Further, the R channel signals, the G channel signals, the B channel signals of the target RGB signal and the second target NIR signal are fused based on the first weight coefficient and the second weight coefficient, and a calculation formula of the step of obtaining a fusion signal is: ; wherein, is the fusion signal; is the R channel signal of the target RGB signal; is the G channel signal of the target RGB signal; is the B channel signal of the target RGB signal; is the second target NIR signal.

[0072] Further, the R channel signals, the G channel signals, the B channel signals of the target RGB signal and the second target NIR signal are fused by using the pulse blood volume method, and the step of obtaining the fusion signal comprises: extracting a first blood volume change rate, a second blood volume change rate and a target blood volume change rate from the R channel signals, the G channel signals and the B channel signals of the target RGB signal; calculating the first blood volume change rate, the second blood volume change rate and the target blood volume change rate according to a fifth preset algorithm to obtain a third weight coefficient; calculating the third weight coefficient according to a sixth preset algorithm to obtain a fourth weight coefficient; and fusing the R channel signals, the G channel signals, the B channel signals of the target RGB signal and the second target NIR signal based on the third weight coefficient and the fourth weight coefficient to obtain the fusion signal.

[0073] Further, the step of extracting the first blood volume change rate, the second blood volume change rate and the target blood volume change rate from the R channel signal, the G channel signal and the B channel signal of the target RGB signal comprises: obtaining a pulse wave signal of the target object in a resting state as a base signal, the base signal having the same time length as the target RGB signal; calculating a blood volume change rate corresponding to each signal value in the target RGB signal based on each signal value in the base signal and each signal value in the target RGB signal; taking the blood volume change rate corresponding to a first signal value in the target RGB signal as the first blood volume change rate; taking the blood volume change rate corresponding to a second signal value in the target RGB signal as the second blood volume change rate; taking the largest blood volume change rate among all blood volume change rates as a third target component; taking the smallest blood volume change rate among all blood volume change rates as a fourth target component; and obtaining the target blood volume change rate by subtracting the fourth target component from the third target component.

[0074] Further, taking the first signal value in the target RGB signal as an example, a calculation formula for calculating the blood volume change rate corresponding to the first signal value in the target RGB signal based on the first signal value in the base signal and the first signal value in the target RGB signal is: ; wherein, is the blood volume change rate corresponding to the first signal value in the target RGB signal, i.e. the first blood volume change rate; is the first signal value in the target RGB signal; is the first signal value in the base signal.

[0075] Further, taking the second signal value in the target RGB signal as an example, a calculation formula for calculating the blood volume change rate corresponding to the second signal value in the target RGB signal based on the second signal value in the base signal and the second signal value in the target RGB signal is: ; wherein, is the blood volume change rate corresponding to the second signal value in the target RGB signal, i.e. the second blood volume change rate; is the second signal value in the target RGB signal; is the second signal value in the base signal.

[0076] Further, the first blood volume change rate, the second blood volume change rate and the target blood volume change rate are calculated according to a fifth preset algorithm, and a calculation formula of the step of obtaining the third weight coefficient is as follows: ; ; ; wherein, is a standard blood volume change rate; is the target blood volume change rate; is a second weight total amount of R channel signals, G channel signals and B channel signals in the target RGB signal; is the third weight coefficient.

[0077] Further, the third weight coefficient is calculated according to a sixth preset algorithm, and a calculation formula of the step of obtaining a fourth weight coefficient is as follows: .

[0078] Further, the R channel signals, the G channel signals, the B channel signals of the target RGB signal and the second target NIR signal are fused based on the third weight coefficient and the fourth weight coefficient, and a calculation formula of the step of obtaining a fusion signal is as follows: .

[0079] Other channel fusion methods can refer to the specific steps of “fusing the R channel signals, the G channel signals and the B channel signals in the target RGB signal and the second target NIR signal to obtain a fusion signal by using a chroma method” and “fusing the R channel signals, the G channel signals and the B channel signals in the target RGB signal and the second target NIR signal to obtain a fusion signal by using a pulse blood volume method”.

[0080] The fusion signal is a remote photoplethysmography (rPPG) signal.

[0081] S4, performing heart rate feature extraction processing on the fusion signal to obtain a target heart rate of the target object.

[0082] Further, the S4 step includes: performing fast Fourier transform processing on the fusion signal to obtain a first frequency spectrum signal; performing frequency spectrum peak value identification processing on the first frequency spectrum signal to obtain a first target frequency spectrum peak value; and taking the first target frequency spectrum peak value as the target heart rate.

[0083] Through the above embodiments, the embodiments of the present application add the second target NIR signal modified by the first preset algorithm on the basis of the target RGB signal. Since the second target NIR signal is not sensitive to the interference of factors such as illumination change, shadow disturbance and / or motion artifact of the environment, the second target NIR signal has strong stability. The embodiments of the present application use the strong stability characteristics of the second target NIR signal to compensate for the poor quality of the target RGB signal, and the specific compensation process is to fuse the target RGB signal and the second target NIR signal into a fusion signal. Then, the target heart rate of the target object is extracted from the fusion signal. Thus, compared with the prior art of extracting physiological parameters only by using the RGB signal, the embodiments of the present application can reduce the detection deviation of the physiological parameters (i.e. the target heart rate) and improve the accuracy of detecting the physiological parameters.

[0084] In some embodiments, before the S2 step, the RGB-NIR joint-based physiological parameter detection method further includes filtering each channel signal in the target RGB signal by using a fourth-order Butterworth band-pass filter. This step can remove the frequency components irrelevant to the heart rate.

[0085] The frequency range of the fourth-order Butterworth band-pass filter is Hz.

[0086] In some embodiments, the RGB-NIR joint-based physiological parameter detection method further includes: obtaining a third target NIR signal corresponding to the chest of the target object; and performing a respiratory frequency extraction process on the third target NIR signal to obtain a target respiratory frequency of the target object.

[0087] Further, the step of obtaining the third target NIR signal corresponding to the chest of the target object includes: performing chest region recognition processing on each image in the first target image set to obtain a third region of interest of each image in the first target image set; estimating a third target feature of each pixel point in the third region of interest of each image in the first target image set in a plurality of third predetermined directions between consecutive frames by using the optical flow method; performing accumulation and calculation processing on all third target features of each pixel point in the third region of interest of each image in the first target image set in each third predetermined direction between consecutive frames to obtain a third displacement signal corresponding to each third predetermined direction of each pixel point in the third region of interest of each image in the first target image set between consecutive frames; calculating the average value of all third displacement signals of each third predetermined direction to obtain a third physiological motion signal corresponding to each third predetermined direction; and performing fusion processing on the third physiological motion signals of all third predetermined directions to obtain the third target NIR signal.

[0088] The plurality of third predetermined directions are horizontal and vertical directions. In other embodiments, other directions can be set by those skilled in the art according to actual needs.

[0089] The third target feature is a velocity component. In other embodiments, other features such as an acceleration component can be set by those skilled in the art according to actual needs.

[0090] Further, the step of performing a respiratory frequency extraction process on the third target NIR signal to obtain a target respiratory frequency of the target object includes: performing a fast Fourier transform process on the third target NIR signal to obtain a second frequency spectrum signal; performing a frequency spectrum peak value identification process on the second frequency spectrum signal to obtain a second target frequency spectrum peak value; and taking the second target frequency spectrum peak value as the target respiratory frequency.

[0091] The embodiments of the present application can realize stable and alternating RGB and NIR image acquisition on a single camera system, effectively avoid overexposure through intelligent exposure and light source control, and obtain high-quality multi-frame original images, thereby realizing accurate and stable non-contact measurement of heart rate and respiration.

[0092] Reference Figure 2 As shown in the figure, it is a principle block diagram of a physiological parameter detection device based on RGB-NIR combination provided by the second aspect of the embodiments of the present application. In Figure 2 The physiological parameter detection device 100 based on RGB-NIR combination includes: An acquisition module 101 is configured to acquire a target RGB signal and a first target NIR signal corresponding to a face of a target object. A correction module 102 is configured to perform a correction process on the first target NIR signal by using a first preset algorithm to obtain a second target NIR signal. A fusion module 103 is configured to perform a fusion process on the target RGB signal and the second target NIR signal to obtain a fusion signal. A heart rate feature extraction module 104 is configured to perform a heart rate feature extraction process on the fusion signal to obtain a target heart rate of the target object.

[0093] The third aspect of the embodiments of the present application provides a terminal device, and a principle block diagram of the terminal device can be as shown in the figure Figure 3The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected through a system bus. The processor is configured to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the physiological parameter detection method based on RGB-NIR combination. The display screen can be a liquid crystal display screen or an electronic ink display screen. The temperature sensor is pre-set inside the terminal device and is configured to detect the operating temperature of the internal device.

[0094] Those skilled in the art can understand that Figure 3 The block diagram shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the terminal device to which the present application is applied. The specific terminal device can include more or fewer components than those shown in the figure, or combine some components, or have a different component arrangement.

[0095] In some embodiments, the present application provides a terminal device including a processor and a memory for storing a computer program, wherein the processor is configured to call and run the computer program stored in the memory to execute the steps of the physiological parameter detection method based on RGB-NIR combination provided in the first aspect of the present application.

[0096] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the steps of the physiological parameter detection method based on RGB-NIR combination provided in the first aspect of the present application.

[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] The technical features of the above embodiments can be combined without changing the basic principles of the present application. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered as the scope of the present application.

[0099] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting physiological parameters based on RGB-NIR combined method, characterized in that, include: Obtain the target RGB signal and the first target NIR signal corresponding to the face of the target object; The first target NIR signal is modified using a first preset algorithm to obtain the second target NIR signal; The target RGB signal and the second target NIR signal are fused to obtain a fused signal; The fused signal is processed to extract heart rate features, thereby obtaining the target heart rate of the target object.

2. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The steps for acquiring the first target NIR signal include: Multiple frames of original images of the target object are acquired sequentially, wherein odd-numbered frames are RGB images and even-numbered frames are NIR images. The pixel values ​​of all NIR images are adjusted using the pixel values ​​of all RGB images to obtain the first target image set; The first target image set is subjected to NIR signal extraction processing to obtain the first target NIR signal.

3. The physiological parameter detection method based on RGB-NIR combined according to claim 2, characterized in that, The exposure times for odd-numbered and even-numbered frames in the multi-frame original images are different.

4. The physiological parameter detection method based on RGB-NIR combined according to claim 2, characterized in that, The steps of adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain the first target image set include: The original images of the multiple frames are divided into multiple image subsets according to time sequence. Each image subset includes one odd-numbered frame image and one even-numbered frame image. Pixel calculation processing is performed on each pixel of the odd-numbered frames in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frames in each image subset; In each image subset, the pixels to be adjusted corresponding to the first pixel of the even-numbered frame image and the second pixel of the odd-numbered frame image are calculated and processed according to the second preset algorithm to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset. The first pixel is any pixel of the even-numbered frame image in each image subset, and the second pixel is the pixel in the odd-numbered frame image of each image subset that is located at the same pixel position as the first pixel. The target pixels corresponding to all pixels of the even-numbered frames in each image subset are used to construct the modified even-numbered frame image for each image subset. The modified even-numbered frame images corresponding to all image subsets constitute the first target image set.

5. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm.

6. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The steps of fusing the target RGB signal and the second target NIR signal to obtain a fused signal include: The signal of at least one channel of the target RGB signal and the second target NIR signal are fused using the colorimetric method or the pulse blood volume method to obtain a fused signal.

7. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The physiological parameter detection method based on RGB-NIR combination also includes: Acquire the third target NIR signal corresponding to the chest of the target object; The respiratory rate of the target object is obtained by performing respiratory rate extraction processing on the NIR signal of the third target.

8. A physiological parameter detection device based on RGB-NIR combined, characterized in that, include: The acquisition module is used to acquire the target RGB signal and the first target NIR signal corresponding to the face of the target object; The correction module is used to correct the first target NIR signal using a first preset algorithm to obtain the second target NIR signal; The fusion module is used to fuse the target RGB signal and the second target NIR signal to obtain a fused signal; The heart rate feature extraction module is used to perform heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

9. A terminal device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the steps of the physiological parameter detection method based on RGB-NIR combined as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the steps of the physiological parameter detection method based on RGB-NIR combined as described in any one of claims 1 to 7.

Images