Physiological parameter detection method, device, terminal and storage medium based on RGB-NIR combination

Abstract

This application relates to the field of biomedical engineering technology. This application discloses a method, device, terminal, and storage medium for detecting physiological parameters based on RGB-NIR combined detection, which can improve the accuracy of physiological parameter detection. The method for detecting physiological parameters based on RGB-NIR combined detection includes acquiring a target RGB signal and a first target NIR signal corresponding to the face of a target object; performing a first preset algorithm to correct the first target NIR signal to obtain a second target NIR signal; fusing the target RGB signal and the second target NIR signal to obtain a fused signal; and performing heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

Description

Technical Field

[0001] This application relates to the field of biomedical engineering technology. More specifically, this application relates to a method, device, terminal, and storage medium for detecting physiological parameters based on RGB-NIR combined methods. Background Technology

[0002] Existing methods for detecting physiological parameters (such as heart rate and / or respiratory rate) typically involve first acquiring multiple frames of RGB images, extracting the RGB signals (which can be understood as pulse wave signals) from these images, and then detecting the physiological parameters from these RGB signals. However, in practical applications, factors such as changes in ambient lighting, shadow disturbances, and motion artifacts interfere with the imaging quality of RGB images, resulting in poor quality of the RGB signals extracted from multiple frames. This leads to discrepancies between the final detected physiological parameters and the actual values, reducing the accuracy of the physiological parameter detection. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, terminal, and storage medium for detecting physiological parameters based on RGB-NIR combined methods, which can improve the accuracy of detecting physiological parameters. This application is mainly achieved through the following technical solutions:

[0004] A first aspect of this application provides a method for detecting physiological parameters based on RGB-NIR combined methods, including:

[0005] Obtain the target RGB signal and the first target NIR signal corresponding to the face of the target object;

[0006] The first target NIR signal is modified using a first preset algorithm to obtain the second target NIR signal;

[0007] The target RGB signal and the second target NIR signal are fused to obtain a fused signal;

[0008] The fused signal is processed to extract heart rate features, thereby obtaining the target heart rate of the target object.

[0009] According to one embodiment of this application, the step of acquiring the first target NIR signal includes:

[0010] Multiple frames of original images of the target object are acquired sequentially, wherein odd-numbered frames are RGB images and even-numbered frames are NIR images.

[0011] The pixel values ​​of all NIR images are adjusted using the pixel values ​​of all RGB images to obtain the first target image set;

[0012] The first target image set is subjected to NIR signal extraction processing to obtain the first target NIR signal.

[0013] According to one embodiment of this application, the exposure times for odd-numbered frames and even-numbered frames in the multi-frame original images are different.

[0014] According to one embodiment of this application, the step of adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain a first target image set includes:

[0015] The original images of the multiple frames are divided into multiple image subsets according to time sequence. Each image subset includes one odd-numbered frame image and one even-numbered frame image.

[0016] Pixel calculation processing is performed on each pixel of the odd-numbered frames in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frames in each image subset;

[0017] In each image subset, the pixels to be adjusted corresponding to the first pixel of the even-numbered frame image and the second pixel of the odd-numbered frame image are calculated and processed according to the second preset algorithm to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset. The first pixel is any pixel of the even-numbered frame image in each image subset, and the second pixel is the pixel in the odd-numbered frame image of each image subset that is located at the same pixel position as the first pixel.

[0018] The target pixels corresponding to all pixels of the even-numbered frames in each image subset are used to construct the modified even-numbered frame image for each image subset.

[0019] The modified even-numbered frame images corresponding to all image subsets constitute the first target image set.

[0020] According to one embodiment of this application, the first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm.

[0021] According to one embodiment of this application, the step of fusing the target RGB signal and the second target NIR signal to obtain a fused signal includes:

[0022] The signal of at least one channel of the target RGB signal and the second target NIR signal are fused using the colorimetric method or the pulse blood volume method to obtain a fused signal.

[0023] According to one embodiment of this application, the physiological parameter detection method based on RGB-NIR combined further includes:

[0024] Acquire the third target NIR signal corresponding to the chest of the target object;

[0025] The respiratory rate of the target object is obtained by performing respiratory rate extraction processing on the NIR signal of the third target.

[0026] A second aspect of this application provides a physiological parameter detection device based on RGB-NIR combined, comprising:

[0027] The acquisition module is used to acquire the target RGB signal and the first target NIR signal corresponding to the face of the target object;

[0028] The correction module is used to correct the first target NIR signal using a first preset algorithm to obtain the second target NIR signal;

[0029] The fusion module is used to fuse the target RGB signal and the second target NIR signal to obtain a fused signal;

[0030] The heart rate feature extraction module is used to perform heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

[0031] A third aspect of this application provides a terminal device, including a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the steps of the physiological parameter detection method based on RGB-NIR combined provided in the first aspect of this application.

[0032] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the RGB-NIR combined physiological parameter detection method provided in the first aspect of this application.

[0033] The beneficial effects of the embodiments of this application include:

[0034] This application embodiment adds a second target NIR signal, corrected by a first preset algorithm, to the target RGB signal. Because the second target NIR signal is insensitive to interference from environmental factors such as changes in illumination, shadow disturbances, and / or motion artifacts, it possesses strong stability. This application embodiment utilizes this strong stability characteristic of the second target NIR signal to compensate for the poor quality of the target RGB signal. Specifically, the compensation process involves fusing the target RGB signal and the second target NIR signal into a fused signal. Then, the target heart rate of the target object is extracted from the fused signal. Therefore, compared to existing technologies that only use RGB signals to extract physiological parameters, this application embodiment can reduce the detection bias of physiological parameters (i.e., target heart rate) and improve the accuracy of physiological parameter detection. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 The flowcharts for some embodiments of the physiological parameter detection method based on RGB-NIR combined according to this application are shown below;

[0037] Figure 2 This is a block diagram illustrating the principle of the RGB-NIR combined physiological parameter detection device in some embodiments of this application;

[0038] Figure 3 This is a schematic block diagram of the terminal device of this application in some embodiments. Detailed Implementation

[0039] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0040] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0041] The terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0042] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0043] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0044] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0045] refer to Figure 1 The diagram shown is a flowchart of a physiological parameter detection method based on RGB-NIR combined, provided in the first aspect of an embodiment of this application. Figure 1 The physiological parameter detection method based on RGB-NIR combination includes steps S1, S2, S3 and S4.

[0046] S1. Obtain the target RGB signal and the first target NIR signal corresponding to the face of the target object.

[0047] In the embodiments of this application, R represents red, G represents green, and B represents blue in RGB. NIR (Near Infrared) is near-infrared light.

[0048] Further, the step of acquiring the first target NIR signal includes: acquiring multiple frames of original images of the target object in a time sequence, wherein the odd-numbered frames of the multiple frames are all RGB images and the even-numbered frames are all NIR images; adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain a first target image set; and performing NIR signal extraction processing on the first target image set to obtain the first target NIR signal.

[0049] It should be understood that the embodiments of this application employ two different exposure times, using the same camera to alternately acquire the RGB image and the NIR image. In this way, the embodiments of this application can maintain a high signal-to-noise ratio under various ambient light conditions and effectively improve the monitoring accuracy of heart rate and respiratory rate, while retaining good visual quality. In other embodiments, different cameras can be used to acquire the RGB image and the NIR image, and the specific settings can be configured by those skilled in the art according to actual needs. The camera is an RGB camera.

[0050] Furthermore, the exposure times for odd-numbered and even-numbered frames in the multi-frame original image are different. In other embodiments, the exposure times for odd-numbered and even-numbered frames in the multi-frame original image may also be the same.

[0051] In the multi-frame original image, odd-numbered frames use a long exposure time, and even-numbered frames use a short exposure time. For example, the long exposure time can be 20 milliseconds, and the short exposure time can be 5 milliseconds. In other embodiments, the specific values ​​of the long and short exposure times can be set by those skilled in the art according to actual needs.

[0052] The long exposure time can be the same as the regular exposure time.

[0053] The use of the long exposure time can fully capture the subtle changes in the skin reflection of the target object under ambient light, which is crucial for detecting physiological signals such as heart rate and respiration.

[0054] While using the short exposure time, the camera also needs to illuminate the target object with a near-infrared light source to obtain the NIR image. Because the near-infrared light source has high intensity and is close to the target object, using a conventional exposure time would result in overexposure of the acquired NIR image. To avoid this, embodiments of this application can programmatically control the camera to capture a reflected image containing the near-infrared light source using a short exposure time, thereby ensuring that the acquired NIR image is not overexposed, and that the image reflection primarily originates from near-infrared light.

[0055] Further, the step of adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain the first target image set includes: dividing the multiple original frames of images into multiple image subsets in chronological order, each image subset including one odd-numbered frame image and one even-numbered frame image; performing pixel calculation processing on each pixel of the odd-numbered frame image in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frame image in each image subset; performing calculation processing on the pixel to be adjusted corresponding to the first pixel of the even-numbered frame image and the second pixel of the odd-numbered frame image in each image subset according to a second preset algorithm to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset, wherein the first pixel is any pixel of the even-numbered frame image in each image subset, and the second pixel is the pixel in the odd-numbered frame image of each image subset that is located at the same pixel position as the first pixel; constructing the modified even-numbered frame image corresponding to each image subset using the target pixels corresponding to all pixels of the even-numbered frame image in each image subset; and constructing the first target image set using the modified even-numbered frame images corresponding to all image subsets.

[0056] Each image subset belongs to one acquisition cycle. The first frame of each image subset is an RGB image, and the second frame is an NIR image.

[0057] Further, the step of performing pixel calculation processing on each pixel of the odd-numbered frames in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frames in each image subset includes: multiplying each pixel of the RGB image in each image subset by a preset ratio. This involves obtaining the pixels to be adjusted corresponding to each pixel in the RGB image of each image subset. This can be understood as the preset ratio being... In other embodiments, the preset ratio can be other ratios, which can be set by those skilled in the art according to actual needs.

[0058] Further, in each image subset, the adjustment pixels corresponding to the first pixel of the even-numbered frame image and the second pixel of the odd-numbered frame image are calculated and processed according to the second preset algorithm to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset. Here, the first pixel is any pixel in the even-numbered frame image of each image subset, and the second pixel is a pixel in the odd-numbered frame image of each image subset that is located at the same pixel position as the first pixel. This step includes: in each image subset, subtracting the adjustment pixel corresponding to the second pixel of the odd-numbered frame image from the first pixel of the even-numbered frame image to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset. This step can eliminate the ambient light signal shared by the two frames in each image subset, obtaining a relatively clean NIR image.

[0059] Furthermore, in this embodiment of the application, the modified even-numbered frame images corresponding to all image subsets are used to construct the first target image set in chronological order.

[0060] All images in the first target image set are NIR images, that is, near-infrared light images.

[0061] Further, the step of performing NIR signal extraction processing on the first target image set to obtain the first target NIR signal includes: performing face recognition processing on each frame of the first target image set using a face detection method to obtain a first region of interest (ROI) for each frame of the first target image set; estimating first target features of each pixel in the first ROI of each frame of the first target image set in multiple first predetermined directions between consecutive frames using an optical flow method; accumulating and calculating all first target features of each pixel in the first ROI of each frame of the first target image set in each first predetermined direction between consecutive frames to obtain a first displacement signal corresponding to each pixel in the first ROI of each frame of the first target image set in each first predetermined direction between consecutive frames; calculating the average value of all first displacement signals in each first predetermined direction to obtain a first physiological motion signal corresponding to each first predetermined direction; and fusing the first physiological motion signals in all first predetermined directions to obtain the first target NIR signal.

[0062] The face detection method can be YuNet, a lightweight, fast, and accurate face detection method. In other embodiments, the face detection method can also be other methods, which can be set by those skilled in the art according to actual needs.

[0063] The plurality of first predetermined directions are horizontal and vertical directions. In other embodiments, those skilled in the art can set other directions according to actual needs.

[0064] The first target feature is a velocity component. In other embodiments, those skilled in the art can set it to other features, such as an acceleration component, according to actual needs.

[0065] Further, the step of acquiring the target RGB signal includes: constructing a second target image set by sequentially assembling all odd-numbered frames in the multiple original images; performing face recognition processing on each frame in the second target image set using the face detection method to obtain a second region of interest (ROI) for each frame in the second target image set; estimating second target features of each pixel within the second ROI of each frame in the second target image set in multiple second predetermined directions between consecutive frames using the optical flow method; accumulating and calculating all second target features of each pixel within the second ROI of each frame in the second target image set in each second predetermined direction between consecutive frames to obtain a second displacement signal corresponding to each pixel within the second ROI of each frame in the second target image set in each second predetermined direction between consecutive frames; calculating the average value of all second displacement signals in each second predetermined direction to obtain a second physiological motion signal corresponding to each second predetermined direction; and fusing all second physiological motion signals in the second predetermined directions to obtain the target RGB signal.

[0066] The plurality of second predetermined directions are horizontal and vertical directions. In other embodiments, those skilled in the art can set other directions according to actual needs.

[0067] The second target feature is a velocity component. In other embodiments, those skilled in the art can set it to other features, such as an acceleration component, according to actual needs.

[0068] The target RGB signal includes an R channel signal, a G channel signal, and a B channel signal. S2, The first target NIR signal is corrected using a first preset algorithm to obtain a second target NIR signal.

[0069] There is a time difference between the first target NIR signal and the target RGB signal; more specifically, there is a time difference between them. The period shift, if directly fused with the two signals, will cause time axis misalignment, affecting signal phase consistency and thus reducing the accuracy of central rate feature extraction from the fused signal. Therefore, it is necessary to correct the first target NIR signal.

[0070] The first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm. In other embodiments, the first preset algorithm may also be other algorithms, which can be set by those skilled in the art according to actual needs.

[0071] The use of the cubic spline interpolation algorithm or the linear interpolation algorithm can increase the sampling frequency by up to 2 times. Regarding the new signal sequence obtained after interpolation (i.e., the second target NIR signal), the "new points" generated by the interpolation are located precisely near the timestamp positions of the RGB image.

[0072] The second target NIR signal is a signal that is time-aligned with the target RGB signal. Acquiring the second target NIR signal can eliminate the half-frame time delay caused by time-division multiplexing.

[0073] Further, when the first preset algorithm is a cubic spline interpolation algorithm, step S2 includes: recording all time points of the first target NIR signal as a first time point set; generating a second time point set based on the first time point set; performing calculations on each time point in the first time point set and each time point in the second time point set according to the cubic spline interpolation formula to obtain the signal value corresponding to each time point in the second time point set; and connecting the signal values ​​corresponding to all time points in the first time point set and the signal values ​​corresponding to all time points in the second time point set in chronological order to obtain the second target NIR signal.

[0074] Further, the step of generating a second set of time points based on the first set of time points includes: adding the first and second time points in the first set of time points and dividing by 2 to obtain the first time point in the second set of time points; adding the second and third time points in the first set of time points and dividing by 2 to obtain the second time point in the second set of time points; adding the third and fourth time points in the first set of time points and dividing by 2 to obtain the third time point in the second set of time points; and so on, adding the (N-1)th and Nth time points in the first set of time points and dividing by 2 to obtain the (N-1)th time point in the second set of time points. N is the length of the third set of time points.

[0075] Furthermore, the calculation formula for obtaining the signal value corresponding to each time point in the second time point set by performing calculations on each time point in the first time point set and each time point in the second time point set according to the cubic spline interpolation formula is as follows:

[0076] ;

[0077] in, It is the second time point in the set. The signal value corresponding to each time point; It is the first time point in the set. The first coefficient corresponding to each time point; It is the first time point in the set. The second coefficient corresponding to each time point; It is the second time point in the set. A point in time; It is the first time point in the set. A point in time; It is the first time point in the set. The third coefficient corresponding to each time point; It is the first time point in the set. The fourth coefficient corresponding to each time point. , It is the length of the third time point set.

[0078] All of the first coefficients, all of the second coefficients, all of the third coefficients, and all of the fourth coefficients are preset and can be set by those skilled in the art according to actual needs.

[0079] Further, when the first preset algorithm is a linear interpolation algorithm, step S2 includes: recording all time points of the first target NIR signal as a third time point set; generating a fourth time point set based on the third time point set; calculating and processing the signal value corresponding to each time point in the fourth time point set, each time point in the third time point set, and each time point in the third time point set according to the linear interpolation formula to obtain the signal value corresponding to each time point in the fourth time point set; and connecting the signal values ​​corresponding to all time points in the third time point set and the signal values ​​corresponding to all time points in the fourth time point set in chronological order to obtain the second target NIR signal.

[0080] Further, the step of generating a fourth time point set based on the third time point set includes: adding the first and second time points in the third time point set and dividing by 2 to obtain the first time point in the fourth time point set; adding the second and third time points in the third time point set and dividing by 2 to obtain the second time point in the fourth time point set; adding the third and fourth time points in the third time point set and dividing by 2 to obtain the third time point in the fourth time point set; and so on, adding the (N-1)th and Nth time points in the third time point set and dividing by 2 to obtain the (N-1)th time point in the fourth time point set.

[0081] Furthermore, the signal values ​​corresponding to each time point in the fourth time point set, each time point in the third time point set, and each time point in the third time point set are calculated and processed according to the linear interpolation formula, and the calculation formula for the signal value corresponding to each time point in the fourth time point set is as follows:

[0082] ;

[0083] in, It is the fourth time point set mentioned above. The signal values ​​corresponding to each time point, the length of the fourth time point set is... ; It is the third time point set The signal value corresponding to each time point; It is the third time point set The signal value corresponding to each time point; It is the third time point set A point in time; It is the third time point set A point in time; It is the fourth time point set mentioned above. A specific point in time. .

[0084] S3. Perform fusion processing on the target RGB signal and the second target NIR signal to obtain a fused signal.

[0085] Further, step S3 includes: fusing the signal of at least one channel of the target RGB signal and the second target NIR signal using the chromaticity-based method (CHROM) or the pulse blood volume method (PBV) to obtain a fused signal.

[0086] More specifically, the R channel signal of the target RGB signal and the second target NIR signal can be fused using a colorimetric method or a pulse blood volume method to obtain a fused signal; or, the G channel signal of the target RGB signal and the second target NIR signal can be fused using a colorimetric method or a pulse blood volume method to obtain a fused signal; or, the B channel signal of the target RGB signal and the second target NIR signal can be fused using a colorimetric method or a pulse blood volume method to obtain a fused signal; or, the R channel signal and G channel signal of the target RGB signal can be fused with the second target NIR signal using a colorimetric method or a pulse blood volume method to obtain a fused signal; or, the R channel signal and G channel signal of the target RGB signal can be fused with the second target NIR signal using a colorimetric method or a pulse blood volume method. Alternatively, the R and B channels of the target RGB signal are fused with the second target NIR signal using a colorimetric method or a pulse blood volume method to obtain a fused signal; or the B and G channels of the target RGB signal are fused with the second target NIR signal using a colorimetric method or a pulse blood volume method to obtain a fused signal; or the R, G, and B channels of the target RGB signal are fused with the second target NIR signal using a colorimetric method or a pulse blood volume method to obtain a fused signal.

[0087] Further, the step of fusing the R-channel, G-channel, and B-channel signals of the target RGB signal with the second target NIR signal using a chromaticity method to obtain a fused signal includes: extracting a first chromaticity component, a second chromaticity component, and a target chromaticity change rate from the R-channel, G-channel, and B-channel signals of the target RGB signal; calculating the first chromaticity component, the second chromaticity component, and the target chromaticity change rate according to a third preset algorithm to obtain a first weighting coefficient; calculating the first weighting coefficient according to a fourth preset algorithm to obtain a second weighting coefficient; and fusing the R-channel, G-channel, B-channel signals of the target RGB signal and the second target NIR signal based on the first weighting coefficient and the second weighting coefficient to obtain a fused signal.

[0088] Further, the step of extracting the first chromaticity component, the second chromaticity component, and the target chromaticity change rate from the R channel signal, G channel signal, and B channel signal of the target RGB signal includes: dividing the target RGB signal into at least three RGB sub-signals; extracting the maximum and minimum chromaticity values ​​corresponding to each RGB sub-signal; calculating the chromaticity component corresponding to each RGB sub-signal based on the maximum and minimum chromaticity values; using the chromaticity component corresponding to the first RGB sub-signal among the at least three RGB sub-signals as the first chromaticity component; using the chromaticity component corresponding to the second RGB sub-signal among the at least three RGB sub-signals as the second chromaticity component; using the largest chromaticity component among all the chromaticity components corresponding to all RGB sub-signals as the first target component; using the smallest chromaticity component among all the chromaticity components corresponding to all RGB sub-signals as the second target component; and subtracting the second target component from the first target component to obtain the target chromaticity change rate.

[0089] The step of "extracting the maximum and minimum chromaticity values ​​of each RGB sub-signal from each RGB sub-signal" can be achieved using existing technologies.

[0090] Furthermore, taking the first RGB sub-signal as an example, if the red channel value is greater than the green channel value and the blue channel value in the maximum chromaticity value corresponding to the first RGB sub-signal, then the calculation formula for calculating the chromaticity component corresponding to the first RGB sub-signal based on the maximum and minimum chromaticity values ​​corresponding to the first RGB sub-signal is as follows:

[0091] ;

[0092] in, It is the chromaticity component corresponding to the first RGB sub-signal, that is, the first chromaticity component; It is the green channel value in the maximum chromaticity value corresponding to the first RGB sub-signal; It is the blue channel value in the maximum chromaticity value corresponding to the first RGB sub-signal; It is the maximum chromaticity value corresponding to the first RGB sub-signal; It is the minimum chromaticity value corresponding to the first RGB sub-signal.

[0093] Furthermore, in the chromaticity maximum values ​​corresponding to the first RGB sub-signal segment, if the green channel value is greater than the red channel value and the blue channel value, then the calculation formula for the step of calculating the chromaticity components corresponding to the first RGB sub-signal segment based on the chromaticity maximum and minimum values ​​corresponding to the first RGB sub-signal segment is as follows:

[0094] ;

[0095] in, It is the red channel value in the maximum chromaticity value corresponding to the first RGB sub-signal.

[0096] Furthermore, if the blue channel value is greater than the red channel value and the green channel value in the maximum chromaticity value corresponding to the first RGB sub-signal, then the calculation formula for calculating the chromaticity component corresponding to the first RGB sub-signal based on the maximum and minimum chromaticity values ​​corresponding to the first RGB sub-signal is as follows:

[0097] .

[0098] Furthermore, taking the second RGB sub-signal as an example, if the red channel value is greater than the green channel value and the blue channel value in the maximum chromaticity value corresponding to the second RGB sub-signal, then the calculation formula for calculating the chromaticity component corresponding to the second RGB sub-signal based on the maximum and minimum chromaticity values ​​corresponding to the second RGB sub-signal is as follows:

[0099] ;

[0100] in, It is the chromaticity component corresponding to the second RGB sub-signal, that is, the second chromaticity component; It is the green channel value in the maximum chromaticity value corresponding to the second RGB sub-signal; It is the blue channel value in the maximum chromaticity value corresponding to the second RGB sub-signal; It is the maximum chromaticity value corresponding to the second RGB sub-signal; It is the minimum chromaticity value corresponding to the second RGB sub-signal.

[0101] Furthermore, in the chromaticity maximum value corresponding to the second RGB sub-signal, if the green channel value is greater than the red channel value and the blue channel value, then the calculation formula for the step of calculating the chromaticity component corresponding to the second RGB sub-signal based on the chromaticity maximum and minimum values ​​corresponding to the second RGB sub-signal is as follows:

[0102] ;

[0103] in, It is the red channel value in the maximum chromaticity value corresponding to the second RGB sub-signal.

[0104] Furthermore, in the chromaticity maximum value corresponding to the second RGB sub-signal, if the blue channel value is greater than the red channel value and the green channel value, then the calculation formula for calculating the chromaticity component corresponding to the second RGB sub-signal based on the chromaticity maximum and minimum values ​​corresponding to the second RGB sub-signal is as follows:

[0105] .

[0106] Furthermore, the calculation formula for obtaining the first weighting coefficient by calculating the first chromaticity component, the second chromaticity component, and the target chromaticity change rate according to the third preset algorithm is as follows:

[0107] ;

[0108] ;

[0109] ;

[0110] in, It is the standard chromaticity change rate; It is the target chromaticity change rate; It is the total first weight of the R channel signal, G channel signal and B channel signal in the target RGB signal; It is the first weighting coefficient.

[0111] Furthermore, the first weight coefficient is calculated and processed according to the fourth preset algorithm to obtain the second weight coefficient. The calculation formula for the step is: .

[0112] Furthermore, the calculation formula for the step of fusing the R channel signal, G channel signal, B channel signal of the target RGB signal and the second target NIR signal based on the first weighting coefficient and the second weighting coefficient to obtain the fused signal is as follows:

[0113] ;

[0114] in, It is the fused signal; It is the R channel signal of the target RGB signal; It is the G channel signal of the target RGB signal; It is the B channel signal of the target RGB signal; It is the second target NIR signal.

[0115] Further, the step of fusing the R-channel, G-channel, and B-channel signals of the target RGB signal with the second target NIR signal using the pulse blood volume method to obtain the fused signal includes: extracting a first blood volume change rate, a second blood volume change rate, and a target blood volume change rate from the R-channel, G-channel, and B-channel signals of the target RGB signal; calculating the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate according to a fifth preset algorithm to obtain a third weighting coefficient; calculating the third weighting coefficient according to a sixth preset algorithm to obtain a fourth weighting coefficient; and fusing the R-channel, G-channel, B-channel signals of the target RGB signal and the second target NIR signal based on the third weighting coefficient and the fourth weighting coefficient to obtain the fused signal.

[0116] Further, the step of extracting the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate from the R channel signal, G channel signal, and B channel signal of the target RGB signal includes: obtaining the pulse wave signal of the target object in a resting state as a base signal, the duration of the base signal being the same as the duration of the target RGB signal; calculating the blood volume change rate corresponding to each signal value in the target RGB signal based on each signal value in the base signal and each signal value in the target RGB signal; taking the blood volume change rate corresponding to the first signal value in the target RGB signal as the first blood volume change rate; taking the blood volume change rate corresponding to the second signal value in the target RGB signal as the second blood volume change rate; taking the largest blood volume change rate among all blood volume change rates as the third target component; taking the smallest blood volume change rate among all blood volume change rates as the fourth target component; and subtracting the fourth target component from the third target component to obtain the target blood volume change rate.

[0117] Furthermore, taking the first signal value in the target RGB signal as an example, the formula for calculating the blood volume change rate corresponding to the first signal value in the target RGB signal based on the first signal value in the base signal and the first signal value in the target RGB signal is as follows:

[0118] ;

[0119] in, It is the rate of change of blood volume corresponding to the first signal value in the target RGB signal, that is, the first rate of change of blood volume; It is the first signal value in the target RGB signal; It is the first signal value in the basic signal.

[0120] Furthermore, taking the second signal value in the target RGB signal as an example, the calculation formula for the blood volume change rate corresponding to the second signal value in the target RGB signal, based on the second signal value in the base signal and the second signal value in the target RGB signal, is as follows:

[0121] ;

[0122] in, It is the rate of change of blood volume corresponding to the second signal value in the target RGB signal, that is, the second rate of change of blood volume; It is the second signal value in the target RGB signal; It is the second signal value in the basic signal.

[0123] Furthermore, the calculation formula for obtaining the third weighting coefficient by calculating the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate according to the fifth preset algorithm is as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] in, It is the standard rate of change in blood volume; It is the rate of change of the target blood volume; It is the total second weight of the R channel signal, G channel signal and B channel signal in the target RGB signal; It is the third weighting coefficient.

[0128] Furthermore, the third weighting coefficient is calculated and processed according to the sixth preset algorithm to obtain the fourth weighting coefficient. The calculation formula for the step is: .

[0129] Furthermore, the calculation formula for the step of fusing the R channel signal, G channel signal, B channel signal of the target RGB signal and the second target NIR signal based on the third weighting coefficient and the fourth weighting coefficient to obtain the fused signal is as follows:

[0130] .

[0131] Other channel fusion methods can be implemented by referring to the specific steps of "using the chromaticity method to fuse the R channel signal, G channel signal and B channel signal in the target RGB signal with the second target NIR signal to obtain a fused signal" and "using the pulse blood volume method to fuse the R channel signal, G channel signal and B channel signal in the target RGB signal with the second target NIR signal to obtain a fused signal".

[0132] The fused signal is an rPPG (remote photoplethysmography) signal.

[0133] S4. Perform heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

[0134] Further, step S4 includes: performing a fast Fourier transform on the fused signal to obtain a first spectral signal; performing spectral peak identification processing on the first spectral signal to obtain a first target spectral peak; and using the first target spectral peak as the target heart rate.

[0135] Through the above implementation methods, this application embodiment adds a second target NIR signal, corrected by a first preset algorithm, to the target RGB signal. Since the second target NIR signal is insensitive to interference from environmental factors such as changes in illumination, shadow disturbances, and / or motion artifacts, it possesses strong stability. This application embodiment utilizes this strong stability characteristic of the second target NIR signal to compensate for the poor quality of the target RGB signal. Specifically, the compensation process involves fusing the target RGB signal and the second target NIR signal into a fused signal. Then, the target heart rate of the target object is extracted from the fused signal. Therefore, compared to existing technologies that only use RGB signals to extract physiological parameters, this application embodiment can reduce the detection bias of physiological parameters (i.e., target heart rate) and improve the accuracy of physiological parameter detection.

[0136] In some implementations, prior to step S2, the RGB-NIR combined physiological parameter detection method further includes filtering each channel of the target RGB signal using a fourth-order Butterworth bandpass filter. This step removes frequency components unrelated to heart rate.

[0137] The frequency range of the fourth-order Butterworth bandpass filter is: hertz.

[0138] In some embodiments, the physiological parameter detection method based on RGB-NIR combination further includes: acquiring a third target NIR signal corresponding to the chest of the target object; and performing respiratory rate extraction processing on the third target NIR signal to obtain the target respiratory rate of the target object.

[0139] Further, the step of acquiring the third target NIR signal corresponding to the chest of the target object includes:

[0140] Chest region recognition processing is performed on each frame of the first target image set to obtain the third region of interest (ROI) of each frame of the first target image set; the optical flow method is used to estimate the third target features of each pixel in the third ROI of each frame of the first target image set in multiple third predetermined directions between consecutive frames; all third target features of each pixel in the third ROI of each frame of the first target image set in each third predetermined direction between consecutive frames are accumulated and calculated to obtain the third displacement signal corresponding to each pixel in the third ROI of each frame of the first target image set in each third predetermined direction between consecutive frames; the average value of all third displacement signals in each third predetermined direction is calculated to obtain the third physiological motion signal corresponding to each third predetermined direction; the third physiological motion signals in all third predetermined directions are fused to obtain the third target NIR signal.

[0141] The plurality of third predetermined directions are horizontal and vertical directions. In other embodiments, those skilled in the art can set other directions according to actual needs.

[0142] The third target feature is a velocity component. In other embodiments, those skilled in the art can set it to other features, such as an acceleration component, according to actual needs.

[0143] Further, the step of performing respiratory frequency extraction processing on the third target NIR signal to obtain the target respiratory frequency of the target object includes: performing fast Fourier transform processing on the third target NIR signal to obtain a second spectral signal; performing spectral peak identification processing on the second spectral signal to obtain a second target spectral peak; and using the second target spectral peak as the target respiratory frequency.

[0144] The embodiments of this application can achieve stable, alternating RGB and NIR image acquisition on a single camera system, and effectively avoid overexposure through intelligent exposure and light source control, thereby obtaining high-quality multi-frame original images, thus achieving accurate and stable non-contact measurement of heart rate and respiration.

[0145] refer to Figure 2The diagram shown is a schematic block diagram of a physiological parameter detection device based on RGB-NIR combined, provided in the second aspect of an embodiment of this application. Figure 2 The physiological parameter detection device 100 based on RGB-NIR combined includes:

[0146] The acquisition module 101 is used to acquire the target RGB signal and the first target NIR signal corresponding to the face of the target object;

[0147] Correction module 102 is used to correct the first target NIR signal using a first preset algorithm to obtain the second target NIR signal;

[0148] The fusion module 103 is used to perform fusion processing on the target RGB signal and the second target NIR signal to obtain a fused signal;

[0149] The heart rate feature extraction module 104 is used to perform heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

[0150] A third aspect of this application provides a terminal device, the schematic diagram of which is as follows: Figure 3 As shown. The terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the terminal device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a physiological parameter detection method based on RGB-NIR combined analysis. The display screen can be a liquid crystal display (LCD) or an e-ink display. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0151] Those skilled in the art will understand that Figure 3 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0152] In some embodiments, this application provides a terminal device, which includes a processor and a memory. The memory stores a computer program, and the processor calls and runs the computer program stored in the memory to perform the steps of the physiological parameter detection method based on RGB-NIR combined provided in the first aspect of this application.

[0153] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the RGB-NIR combined physiological parameter detection method provided in the first aspect of this application.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0155] The technical features of the above embodiments can be combined without changing the basic principles of this application. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the patent protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting physiological parameters based on RGB-NIR combined, characterized in that, include: Obtain the target RGB signal and the first target NIR signal corresponding to the face of the target object; The first target NIR signal is modified using a first preset algorithm to obtain the second target NIR signal; The target RGB signal and the second target NIR signal are fused to obtain a fused signal; The fused signal is subjected to heart rate feature extraction processing to obtain the target heart rate of the target object; The step of fusing the target RGB signal and the second target NIR signal to obtain a fused signal includes: using the pulse blood volume method to fuse the R channel signal, G channel signal and B channel signal in the target RGB signal with the second target NIR signal to obtain a fused signal; The steps of fusing the R-channel, G-channel, and B-channel signals of the target RGB signal with the second target NIR signal using the pulse blood volume method to obtain a fused signal include: extracting a first blood volume change rate, a second blood volume change rate, and a target blood volume change rate from the R-channel, G-channel, and B-channel signals of the target RGB signal; calculating the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate according to a fifth preset algorithm to obtain a third weighting coefficient; calculating the third weighting coefficient according to a sixth preset algorithm to obtain a fourth weighting coefficient; and fusing the R-channel, G-channel, B-channel signals of the target RGB signal and the second target NIR signal based on the third and fourth weighting coefficients to obtain a fused signal. The steps of extracting the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate from the R channel, G channel, and B channel signals of the target RGB signal include: obtaining the pulse wave signal of the target object in a resting state as a base signal, the duration of which is the same as the duration of the target RGB signal; calculating the blood volume change rate corresponding to each signal value in the target RGB signal based on each signal value in the base signal and each signal value in the target RGB signal; taking the blood volume change rate corresponding to the first signal value in the target RGB signal as the first blood volume change rate; taking the blood volume change rate corresponding to the second signal value in the target RGB signal as the second blood volume change rate; taking the largest blood volume change rate among all blood volume change rates as the third target component; taking the smallest blood volume change rate among all blood volume change rates as the fourth target component; and subtracting the fourth target component from the third target component to obtain the target blood volume change rate. The formula for calculating the rate of change of blood volume corresponding to the first signal value of the target RGB signal based on the first signal value in the basic signal and the first signal value in the target RGB signal is as follows: ; The formula for calculating the blood volume change rate corresponding to the second signal value of the target RGB signal based on the second signal value in the basic signal and the second signal value in the target RGB signal is as follows: ; The formula for calculating the third weighting coefficient by processing the first blood volume change rate, the second blood volume change rate, and the target blood volume change rate according to the fifth preset algorithm is as follows: ; ; ; The third weighting coefficient is calculated and processed according to the sixth preset algorithm to obtain the fourth weighting coefficient. The calculation formula for the step is: ; The calculation formula for fusing the R channel signal, G channel signal, B channel signal of the target RGB signal and the second target NIR signal based on the third weighting coefficient and the fourth weighting coefficient to obtain the fused signal is as follows: ; The steps for acquiring the first target NIR signal include: acquiring multiple frames of original images of the target object in a time sequence, wherein odd-numbered frames are RGB images and even-numbered frames are NIR images; adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain a first target image set; and performing NIR signal extraction processing on the first target image set to obtain the first target NIR signal. The exposure times for odd-numbered frames and even-numbered frames in the multi-frame original images are different. Odd-numbered frames in the multi-frame original images use a long exposure time, while even-numbered frames in the multi-frame original images use a short exposure time. The steps of adjusting the pixel values ​​of all NIR images using the pixel values ​​of all RGB images to obtain a first target image set include: dividing the multiple original frames of images into multiple image subsets in chronological order, each image subset including one odd-numbered frame image and one even-numbered frame image; performing pixel calculation processing on each pixel of the odd-numbered frame image in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frame image in each image subset; performing calculation processing on the pixel to be adjusted corresponding to the first pixel of the even-numbered frame image and the second pixel of the odd-numbered frame image in each image subset according to a second preset algorithm to obtain the target pixel corresponding to the first pixel of the even-numbered frame image in each image subset, wherein the first pixel is any pixel of the even-numbered frame image in each image subset, and the second pixel is the pixel in the odd-numbered frame image of each image subset that is located at the same pixel position as the first pixel; using the target pixels corresponding to all pixels of the even-numbered frame image in each image subset to construct the modified even-numbered frame image corresponding to each image subset; and using the modified even-numbered frame images corresponding to all image subsets to construct the first target image set. The step of performing pixel calculation processing on each pixel of the odd-numbered frames in each image subset according to a preset ratio to obtain the pixel to be adjusted corresponding to each pixel of the odd-numbered frames in each image subset includes: multiplying each pixel of the RGB image in each image subset by a preset ratio. Obtain the pixel to be adjusted corresponding to each pixel of the RGB image in each image subset; in, It is the rate of change of blood volume corresponding to the first signal value in the target RGB signal, that is, the first rate of change of blood volume; It is the first signal value in the target RGB signal; It is the first signal value in the basic signal; It is the rate of change of blood volume corresponding to the second signal value in the target RGB signal, that is, the second rate of change of blood volume; It is the second signal value in the target RGB signal; It is the second signal value in the basic signal; It is the standard rate of change in blood volume; It is the rate of change of the target blood volume; It is the total second weight of the R channel signal, G channel signal and B channel signal in the target RGB signal; It is the third weighting coefficient; It is the fused signal; It is the R channel signal of the target RGB signal; It is the G channel signal of the target RGB signal; It is the B channel signal of the target RGB signal; It is the second target NIR signal.

2. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The first preset algorithm is a cubic spline interpolation algorithm or a linear interpolation algorithm.

3. The physiological parameter detection method based on RGB-NIR combined according to claim 1, characterized in that, The physiological parameter detection method based on RGB-NIR combination also includes: Acquire the third target NIR signal corresponding to the chest of the target object; The respiratory rate of the target object is obtained by performing respiratory rate extraction processing on the NIR signal of the third target.

4. A physiological parameter detection device based on RGB-NIR joint for implementing the physiological parameter detection method based on RGB-NIR joint according to any one of claims 1 to 3, characterized in that, include: The acquisition module is used to acquire the target RGB signal and the first target NIR signal corresponding to the face of the target object; The correction module is used to correct the first target NIR signal using a first preset algorithm to obtain the second target NIR signal; The fusion module is used to fuse the target RGB signal and the second target NIR signal to obtain a fused signal; The heart rate feature extraction module is used to perform heart rate feature extraction processing on the fused signal to obtain the target heart rate of the target object.

5. A terminal device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the steps of the physiological parameter detection method based on RGB-NIR combined according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the steps of the physiological parameter detection method based on RGB-NIR combined according to any one of claims 1 to 3.

Images