Remote heart rate detection method based on multispectral imaging technology

Abstract

The invention relates to a remote heart rate detection method based on a multispectral imaging technology. The method comprises the following steps of: acquiring a multi-spectral image sequence of the face of a testee by using a multi-spectral camera; separating and extracting the multispectral image sequence into a plurality of single-channel image sequences; extracting a target face region of interest of the testee from the plurality of single-channel image sequences by using a face detection algorithm, and performing face region clipping on the image sequences according to the target face region of interest; preprocessing each cut single-channel image sequence, and constructing standardized spectrum data; extracting independent components from the standardized spectrum data by using an independent component analysis method; selecting a signal corresponding to the maximum power spectral density from the independent components as a preliminary BVP signal; and filtering the preliminary BVP signal to obtain a final blood volume pulse signal. According to the non-contact detection method, interference of ambient light can be better eliminated, and the accuracy and robustness of blood volume pulse signal extraction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of heart rate detection, and in particular to a non-contact heart rate detection method based on multispectral imaging technology. Background Art

[0002] Heart rate is one of the key physiological parameters of the human body and is extremely important for cardiovascular health assessment, disease monitoring, exercise monitoring, and daily health management. Traditional heart rate detection methods are mainly contact measurement methods, such as electrocardiogram (ECG) and photoplethysmography (PPG). Although ECG measurement can accurately obtain information about the heart's electrical activity, its equipment is relatively complex and expensive, and requires professional operation and multiple electrodes to be attached to the surface of the subject's body, which greatly limits its application in daily continuous monitoring and remote detection scenarios. Although photoplethysmography is relatively simple, it usually requires special sensor equipment to be worn on the body contact part (such as fingers, wrists, etc.). This contact measurement method may cause inconvenience to users for long-term continuous monitoring, especially in some special environments or when remote, non-invasive detection is required, its application flexibility is restricted.

[0003] With the continuous advancement of science and technology, non-contact detection technologies, such as remote photoplethysmography (rPPG), have gradually become a research hotspot. Remote photoplethysmography extracts heart rate information by analyzing changes in skin color in videos. Current research focuses on RGB video. However, RGB video only contains information in the red, green, and blue bands, and its spectral information is relatively simple. During the heart rate detection process, human blood has complex absorption and reflection characteristics for light of different wavelengths, which makes it difficult for RGB video to fully capture this information. As a result, it is susceptible to interference from ambient light changes, motion artifacts, and other factors when extracting heart rate signals. Detection accuracy is limited and its scope of application is narrow. Accurately obtaining reliable heart rate data is particularly difficult in complex scenarios.

[0004] Multispectral imaging technology, with its ability to capture reflectance information from target objects across multiple spectral bands, offers a new approach for remote heart rate monitoring. Multispectral images can provide richer blood-related information. By selecting and analyzing specific wavelengths, they can reduce the impact of ambient light and motion artifacts, facilitating more accurate heart rate signal extraction. Simultaneously, the development of smartphone-based spectral imaging technology presents new opportunities. Its portability allows users to monitor their heart rate anytime, anywhere using their phones, eliminating the need for specialized testing equipment. Furthermore, the widespread availability of smartphones allows for rapid adoption of this technology across a broad user base, facilitating large-scale health data collection and analysis. Furthermore, the powerful computing power of smartphones enables local preliminary processing and analysis of collected multispectral images, reducing data transmission delays. Furthermore, these devices can be combined with cloud services to further optimize detection algorithms and store data. Summary of the Invention

[0005] In response to the technical problems existing in the above-mentioned prior art, the present invention provides a remote heart rate detection method based on multispectral imaging technology. By using a multispectral camera to acquire multispectral image sequences, the limitations of existing heart rate detection technology can be overcome, and it has extremely important practical significance and application value for achieving high-precision, remote heart rate detection.

[0006] The present invention solves its technical problems by adopting the following technical solutions:

[0007] The remote heart rate detection method based on multispectral imaging technology includes the following steps:

[0008] S100: Use a multispectral camera to collect a multispectral image sequence of the subject's face;

[0009] S200: separating and extracting the multispectral image sequence into multiple single-channel image sequences;

[0010] S300: extracting a target face region of interest of the subject from multiple single-channel image sequences using a face detection algorithm, and performing face region cropping on the image sequences according to the target face region of interest;

[0011] S400: Preprocessing each cropped single-channel image sequence to construct standardized spectrum data;

[0012] S500: extracting independent components from the standardized spectrum data using an independent component analysis method;

[0013] S600: Selecting a signal corresponding to the maximum power spectrum density among the independent components as a preliminary BVP signal;

[0014] S700: Filter the preliminary BVP signal to obtain a final blood volume pulse signal.

[0015] The advantages of the present invention are:

[0016] 1. A multispectral imaging system can capture light signals of multiple wavelengths. Different wavelengths of light penetrate human tissue to varying depths and interact with tissue, providing richer information. Furthermore, because the imaging system has multiple spectral channels, the present invention can better eliminate ambient light interference under varying lighting conditions by comprehensively analyzing different spectral information, thereby improving the accuracy and robustness of blood volume pulse signal extraction.

[0017] 2. The method of the present invention is simple to implement and can be easily integrated into mobile detection devices such as smart phones.

[0018] 3. Compared with the traditional contact heart rate detection method, the non-contact method of the present invention can realize remote detection and has better applicability in sports scenes, skin burns and neonatal patient scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the overall process of the method of the present invention.

[0020] Figure 2 4 is a flow chart of a method for extracting a target facial region of a subject from a heart rate detection video according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] In one embodiment, Figure 1 As shown, Figure 1 This is a schematic diagram of the overall process of a remote heart rate detection method based on multispectral imaging technology provided by an embodiment of the present invention. The method is applied to a computer device and includes the following steps:

[0023] S100: Use a multispectral camera to collect a multispectral image sequence of the subject's face;

[0024] S200: separating and extracting the multispectral image sequence into multiple single-channel image sequences;

[0025] S300: extracting a target face region of interest of the subject from multiple single-channel image sequences using a face detection algorithm, and performing face region cropping on the image sequences according to the target face region of interest;

[0026] S400: Preprocessing each cropped single-channel image sequence to construct standardized spectrum data;

[0027] S500: extracting independent components from the standardized spectrum data using an independent component analysis (ICA) method;

[0028] S600: Selecting the signal corresponding to the maximum power spectral density in the independent components as a preliminary BVP (Blood Volume Pulse) signal;

[0029] S700: Filter the preliminary BVP signal to obtain a final blood volume pulse signal.

[0030] In step S100, the embodiment specifically includes: the subject sits about 2 meters away from the multispectral camera and wears a neck brace to ensure that the head remains still during the acquisition process. The multispectral camera uses a 16-channel camera. The multispectral camera is used to collect a 16-channel multispectral image sequence V of the subject's face. 1 ,F 2 ,F 3 ,…,F t ,…,F T}, where V is the collected multispectral image sequence, F t is the t-th multispectral image in the multispectral image sequence V, t∈{1,2,…,T}, T is the number of frames in the multispectral image sequence V; the acquisition band of the multispectral camera is 564-706nm, a total of 16 bands, the acquisition frame rate is 30 Hz, and the resolution is 1024*1024.

[0031] In step S200, the embodiment specifically comprises: converting each frame of image F t From the composite image containing 16 channel information, separate and extract 16 images each retaining only a single channel information, thereby forming a sequence of 16 single-channel images v i is the separated single-channel image sequence, i∈{1,2,…,16}, is a single-channel image sequence v i The t-th frame single-channel image in , t∈{1,2,…,T}, T is the single-channel image sequence v i The single-channel image frame rate is 30 Hz, with a resolution of 128 x 128. Each image in the sequence fully corresponds to the facial image information captured by the corresponding channel in the original multispectral image sequence at different time points. Each single-channel sequence maintains the same time frame correspondence with the original sequence, facilitating further analysis and processing of heart rate-related features based on each single-channel sequence.

[0032] In step S300, if Figure 2As shown, the method of extracting the target face area of ​​the subject from the heart rate detection video using the face detection algorithm in this embodiment includes:

[0033] S310: Determine the value of the backend parameter used for face detection (an identifier for selecting a face detection algorithm) to determine which face detection algorithm to use for subsequent operations. This allows for flexible selection of the appropriate face detection method based on actual needs. If backend is "HC," the process begins using OpenCV's HaarCascade algorithm for face detection, which has the advantage of fast computation. If backend is "RF," the process begins using TensorFlow's RetinaFace algorithm for face detection, which has the advantage of high accuracy. If it is any other unsupported value, a ValueError exception is thrown, indicating that the face detection backend is not supported.

[0034] Determine the use of face detection algorithms for subsequent operations, including:

[0035] S311: If the value of backend is "HC", the CascadeClassifier class of OpenCV is used to load the pre-trained model, which will be based on the Haar feature in the input single-channel single-frame image. In the process of detecting the face area, after completing the face detection, the relevant information about the area where the face is located is output and stored in a variable - face area information face_zone. The face area information face_zone includes information such as the face area coordinates, area width and height. These information together define the specific position and size of the face in the image and are the basic data required for subsequent operations (such as cropping and analysis of face images). At the same time, the number of faces detected is determined. If the number is less than 1, that is, no face is detected, the range of the entire input image is returned to ensure the continuity of the subsequent processing flow. Even in the case of face detection failure, there is still data that can be passed to the subsequent steps, so that the subsequent steps can perform full image analysis; if the number is greater than or equal to 2, that is, multiple faces are detected, at this time it is necessary to select the largest face area for subsequent processing; if the number is 1, the unique face area coordinates, width and height information are directly assigned to the face area information face_zone.

[0036] S312: If the value of backend is "RF", the RetinaFace algorithm based on TensorFlow is used to input a single-channel single-frame image. For face detection, the RetinaFace algorithm uses a deep learning model to analyze facial features in an image and return a list of dictionaries. Each dictionary contains information such as the diagonal coordinates of a face region and a confidence score. The length of the dictionary is checked. If it is greater than 0, it means a face has been detected. The dictionary corresponding to the face with the highest confidence score is selected from the detected faces, and the diagonal coordinates of the face region are assigned to face_zone. If the length is less than 0, meaning no face has been detected, the entire image range is returned, and the message "ERROR: No Face Detected" is printed.

[0037] After face detection is completed, cropping is performed based on the detected face area, including:

[0038] S320: 16 single-channel image sequences are processed according to the face area detected in the previous step Cropping is performed, and the frames of each channel after cropping are scaled according to the target width and height respectively. The INTER_AREA interpolation method is used to reassemble the scaled channel images into multiple single-channel image sequences.

[0039] In step S400, this embodiment preprocesses the recombined multiple single-channel image sequences to construct standardized spectrum data, specifically including:

[0040] S410: Traverse each frame image in each single-channel image sequence after trimming, summarize the pixel values ​​of each channel, that is, first sum the pixel values ​​along the row direction, then sum the results of the first summation again along the column direction, and then divide the sum of the pixel values ​​of each channel by the total number of pixels of the frame (the height of the frame multiplied by the width) to obtain the average pixel value, and integrate these average pixel values ​​to form a spectrum array to construct the spectrum data is a single-channel spectrum array S i The tth data in , t∈{1,2,…,T}, T is the number of frames of the acquired image sequence. This spectrum data can reflect the characteristic information of each channel in the image sequence over time from another perspective.

[0041] S420: Check the validity of the constructed spectrum data. Use the judgment function to check whether there are non-numeric or infinite values ​​in the data. If these abnormal values ​​exist, throw an exception prompt and terminate the operation to ensure the quality of the data processed subsequently. The invalid data will not interfere with the subsequent heart rate signal extraction results.

[0042] S430: Standardize the obtained spectral data. Each channel of the spectral data is traversed, and the detrend function is first used to remove the linear trend portion of the data. The original data may have some linear trends over time, but these trends are not essential features related to the heart rate signal. Removing them helps to highlight the true heart rate-related changes. Then, the mean of the detrended data is subtracted and divided by the standard deviation to standardize the data for each channel. This processed data can better adhere to statistical assumptions such as normal distribution in subsequent analysis algorithms.

[0043] In step S500, this embodiment assumes that the blind source separation model is

[0044] S=W×C (1)

[0045] Where c is the source signal (including the blood volume pulse signal), W is the confusion matrix of the mixed source signal, and S is the mixed signal. This model assumes that the source signals are linearly combined into a spectral signal. The key to restoring the source signal is to calculate the inverse matrix A (unmixing matrix) of the confusion matrix W, that is, inv(W). This includes:

[0046] S510: performing whitening processing on the standardized spectrum data, including:

[0047]

[0048] Among them, U and D are the eigenvectors and eigenvalues ​​of the covariance matrix respectively, and the superscript T represents the transpose. whitened The covariance matrix of is the identity matrix:

[0049]

[0050] S520: In order to capture the high-order statistical characteristics of the signal, the fourth-order moment matrix M of the whitened data is calculated:

[0051] M ijkl =E[C i E j C k C l ]-E[C i C j ]E[C k C l ]-E[C i C k ]E[C j C l ]-E[C i C l ]E[C j C k ] (4)

[0052] Since the data has been whitened, it can be simplified to:

[0053] M ijkl =E[C i C j C k C l ] (5)

[0054] Among them, C i 、C j 、C k 、C l are different components in the source signal C respectively, i, j, k, l are indexes, and in this embodiment, i, j, k, l∈{1, 2, ..., 16}.

[0055] S530: By jointly diagonalizing multiple fourth-order moment matrices, find a rotation matrix R so that the fourth-order moment matrix is ​​as diagonal as possible in the new coordinate system. Specifically, a set of independent fourth-order moment matrices are constructed and diagonalized using the rotation matrix R:

[0056]

[0057] Among them, M (m) To traverse all (i, j, k, l) combinations (a total of 16 4 The fourth-order moment matrix containing the high-order correlation of the signal is generated by (1, 2, ..., 16 4}. are the diagonal elements of the mth fourth-order moment matrix.

[0058] S540: Solve the rotation matrix R by continuously iterating Givens rotations so that all fourth-order moment matrices are diagonalized as much as possible in the new coordinate system.

[0059] S550: Combined with whitening matrix A whitened =UD -1 / 2 U T And the rotation matrix R, get the final unmixing matrix A:

[0060] A=RA whitened (7)

[0061] S560: Finally, the mixed signal S is separated using the unmixing matrix A to obtain the estimated source signal matrix C, C = {c 1 ,c 2 ,…,c j ,…,c 16},j∈{1,2,…,16},c j is the jth independent component in the source signal matrix.

[0062] In step S600, this embodiment selects a signal corresponding to the maximum power spectrum density in the independent components as a preliminary BVP signal, which specifically includes:

[0063] S610: Perform fast Fourier transform on each channel of the source signal matrix C estimated in step S560, calculate the frequency axis and remove the DC component, and calculate the power spectrum density.

[0064] S620: Select the signal corresponding to the maximum power spectrum density as the preliminary BVP signal.

[0065] In step S700, this embodiment filters the preliminary BVP signal to obtain a final blood volume pulse signal. Specifically, a bandpass filter is applied to the preliminary BVP signal, with a low-frequency cutoff frequency of 0.7 and a high-frequency cutoff frequency of 2.5. The first element of the filtered signal is taken as the final BVP signal. This signal is the blood volume pulse signal extracted from the original spectrum data and can be used for subsequent physiological analysis and other applications.

[0066] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A remote heart rate detection method based on multispectral imaging technology, characterized in that: The method comprises the following steps: S100: Use a multispectral camera to collect a multispectral image sequence of the subject's face; S200: separating and extracting the multispectral image sequence into multiple single-channel image sequences; S300: extracting a target face region of interest of the subject from multiple single-channel image sequences using a face detection algorithm, and performing face region cropping on the image sequences according to the target face region of interest; S400: Preprocessing each cropped single-channel image sequence to construct standardized spectrum data; S5 00: extracting independent components from the standardized spectrum data using an independent component analysis method; S600: Selecting a signal corresponding to the maximum power spectrum density among the independent components as a preliminary BVP signal; S700: Filter the preliminary BVP signal to obtain a final blood volume pulse signal.

2. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: The step S100 specifically includes: the subject sits about 2 meters away from the multispectral camera, and the subject's head remains still during the acquisition process, and the multispectral camera is used to acquire a multi-channel multispectral image sequence of the subject's face.

3. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: The step S200 is specifically as follows: each frame of image F t From the composite image containing n channel information, separate and extract multiple images each retaining only a single channel information, thereby forming n single-channel image sequences v i ={f i 1 ,f i 2 ,f i 3 ,…,f i t ,…,f i T }, v i is the separated single-channel image sequence, i∈{1,2,…,n}, f i t is a single-channel image sequence v i The t-th frame single-channel image in , t∈{1,2,…,T}, T is the single-channel image sequence v i The number of frames in .

4. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: In step S300, a face detection algorithm is used to extract a target face region of interest of the subject from multiple single-channel image sequences, including: S310: According to the value of the backend parameter backend for face detection, select to use the HaarCascade algorithm or the RetinaFace algorithm for face detection.

5. The remote heart rate detection method based on multispectral imaging technology according to claim 4, characterized in that: In step S300, a face detection algorithm is used to extract a target face region of interest of the subject from multiple single-channel image sequences, including: S311: If the HaarCascade algorithm is used, the CascadeClassifier class is used to load the pre-trained model. The model is based on the Haar feature in the input single-channel single-frame image f i t Detect the face area in the image, and after completing the face detection, output the relevant information about the area where the face is located and store it in the face area information face_zone; the face area information face_zone includes the face area coordinates, area width and height information; determine the number of faces detected, if the number is less than 1, that is, no face is detected, then return the range of the entire input image; if the number is greater than or equal to 2, that is, multiple faces are detected, then select the largest face area for subsequent processing; if the number is 1, directly assign the unique face area coordinates, width and height information to the face area information face_zone.

6. The remote heart rate detection method based on multispectral imaging technology according to claim 4, characterized in that: In step S300, a face detection algorithm is used to extract a target face region of interest of the subject from multiple single-channel image sequences, including: S312: If the RetinaFace algorithm is used, a deep learning model is adopted to analyze the features of the face in the image and return a dictionary list, where each dictionary contains the two diagonal coordinates of the face area and confidence information; the dictionary length is determined. If the length is greater than 0, it means that a face is detected. The dictionary corresponding to the face with the highest confidence is selected from the detected face information, and the two diagonal coordinates of the face area are assigned to the face area information face_zone. The face area information face_zone includes the face area coordinates, area width and height information. If the length is not greater than 0, that is, no face is detected, the entire image range is returned.

7. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: In step S300, the face region of the image sequence is cropped according to the target face region of interest, including: S320: Cropping multiple single-channel image sequences according to the target face region of interest obtained by the face detection algorithm, and scaling the cropped frames of each channel according to the target width and height respectively, and recombining the scaled channel images into multiple single-channel image sequences using the INTER_AREA interpolation method.

8. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: The step S400 is specifically as follows: S410: traverse each frame image in each cropped single-channel image sequence, summarize the pixel values ​​of each channel, that is, first sum the pixel values ​​along the row direction, then sum the results of the first summation again along the column direction, and then divide the sum of the pixel values ​​of each channel by the total number of pixels in the frame to obtain an average pixel value, and integrate these average pixel values ​​to form a spectrum array, thereby constructing spectrum data; S420: Performing a data validity check on the constructed spectrum data, using a judgment function to check whether there are non-numeric or infinite values ​​in the data. If these abnormal values ​​exist, the step is terminated. S430: Standardize the obtained spectrum data: traverse each channel of the spectrum data, first use the detrending function to remove the linear trend part of the data, then subtract the mean of the detrended data and divide it by the standard deviation to standardize the data of each channel.

9. The remote heart rate detection method based on multispectral imaging technology according to claim 1, characterized in that: The step S500 is specifically as follows: First assume that the blind source separation model is S = W × C Where C is the source signal containing the blood volume pulse signal, W is the confusion matrix of the mixed source signal, and S is the mixed signal; S510: performing whitening processing on the standardized spectrum data; S520: In order to capture the high-order statistical characteristics of the signal, the fourth-order moment matrix of the whitened data is calculated; S530: performing joint diagonalization on multiple fourth-order moment matrices to find a rotation matrix R such that the fourth-order moment matrix is ​​as diagonal as possible in the new coordinate system; S540: Solve the rotation matrix R by continuously iterating the Givens rotation; S550: combining the whitening matrix and the rotation matrix R to obtain an unmixing matrix; S560: Use the unmixing matrix to separate the mixed signal S to obtain an estimated source signal matrix.

10. The remote heart rate detection method based on multispectral imaging technology according to claim 9, characterized in that: The step S600 is specifically as follows: S610: Perform fast Fourier transform on each channel of the source signal matrix estimated in step S560, calculate the frequency axis and remove the DC component, and calculate the power spectrum density; S620: Select the signal corresponding to the maximum power spectrum density as the preliminary BVP signal.

Images