A non-contact visual sensing physiological parameter detection method, system and device

Abstract

The application discloses a non-contact visual physiological parameter detection method, system and device, and relates to the technical field of visual physiological index measurement. The method comprises the following steps: collecting facial video data under different wave bands, and extracting a facial region of interest (ROI); transforming the facial ROI video data of different wave bands into a multi-color space respectively, and extracting corresponding space-time feature spectrums respectively; inputting each space-time feature spectrum into a physiological detection network, and preliminarily extracting feature maps of the space-time feature spectrums; inputting each feature map into a multi-scale dense visual state space module, and outputting corresponding feature spectrums; and fusing each feature spectrum by using a multi-time-space map spectrum visual mangaba fusion module (MSTM-VMF) to obtain output features. The method effectively integrates the information of various space-time feature spectrums by means of a fusion prediction module, realizes the simultaneous detection of various physiological parameters, and improves the monitoring efficiency.

Description

Technical Field

[0001] This invention relates to the field of visual physiological index measurement technology, specifically to a non-contact method, system, and device for detecting physiological parameters of visual perception. Background Technology

[0002] Physiological parameter monitoring is a crucial medical procedure in emergency care and chronic disease monitoring for the injured and sick. However, routine physiological monitoring largely relies on contact-based methods, such as multi-parameter physiological monitors, which use various contact sensors to monitor vital parameters like heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature in real time. Considering the large number of injured people urgently needing physiological monitoring during large-scale infectious disease outbreaks or major accidents, contact-based physiological parameter monitoring equipment is not only inefficient and has limited availability, but is also affected by the specific location of the injuries, and increases the risk of cross-infection between medical staff and patients.

[0003] The proposed physiological parameter detection method utilizes wavelet transform to decompose 3D video data frame by frame, suppressing noise while preserving important information such as remote photoplethysmography (PPG) pulse. Based on the wavelet-transformed data, a spatiotemporal feature spectrum is constructed. This spectrum is then used as input data to a CNN for feature analysis to extract the corresponding heart rate value.

[0004] However, this method does not fully utilize the important information contained in the original RGB color space of the video data, making it difficult to extract subtle physiological signals and reducing the accuracy of detection. Summary of the Invention

[0005] To address the shortcomings of existing technologies that fail to fully utilize the important information contained in the original RGB color space of video data, making it difficult to extract subtle physiological signals, this invention proposes a non-contact visual perception method, system, and device for detecting physiological parameters. By combining data from different spectra and color spaces, it enhances the ability to extract subtle physiological signals, improves the accuracy and reliability of detection, and thus solves the problems existing in the prior art.

[0006] A non-contact visual perception physiological parameter detection method includes the following steps:

[0007] Collect facial video data of the target under different bands and extract the region of interest (ROI) of the face; transform the facial video data of different bands corresponding to the ROI to a multi-color space and extract the corresponding spatiotemporal feature spectra.

[0008] Each spatiotemporal feature spectrum is input into the physiological detection network. A compressed excitation residual initial convolution module is used to initially extract feature maps for each spatiotemporal feature spectrum. Each feature map is then input into a multi-scale dense visual state space module. A downsampling unit simplifies each input feature map to obtain feature maps that can represent physiological features. Each feature map that can represent physiological features is then input into four densely connected multi-scale visual state space (MVSS) units to extract the corresponding feature spectrum. The MSTM-VMF multi-spatiotemporal visual mamba fusion module is used to linearly transform and concatenate each feature spectrum. The MVSS module is then used to extract features from the concatenated feature spectrum. The extracted features are then subjected to global average pooling to obtain the output features.

[0009] The physiological parameters of the target are detected based on the output features.

[0010] Furthermore, the KLT facial unit recognition algorithm is used to automatically mark multiple marker points on the target face. Based on each marker point, the facial location region from the glabella to both cheeks and then to the jaw is automatically extracted, thereby obtaining the facial region of interest (ROI).

[0011] Furthermore, the step of transforming the facial video data of different bands corresponding to the ROI to a multi-color space and extracting the corresponding spatiotemporal feature spectra specifically includes the following steps:

[0012] The ROI was transformed from the original RGB color space to the YUV and YCrCb color spaces, which reflect skin color.

[0013]

[0014] The ROIs of natural light, red light, blue light, and thermal infrared data are divided into multiple sub-ROIs. i ;

[0015] From each subROI i The pixel mean values ​​from the RGB, YUV, and YCrCb color spaces are extracted along the time axis to form a spatiotemporal feature value signal line.

[0016] The spatiotemporal characteristic spectral signal lines of different band video data of the facial ROI in RGB, YUV and YCrCb color spaces are calculated sequentially and arranged in rows to generate the spatiotemporal characteristic spectrum of multispectral data in RGB, YUV and YCrCb color spaces.

[0017] Furthermore, the LeakyHardSwish activation function is expressed as:

[0018]

[0019] Where x represents the input feature.

[0020] Furthermore, it also includes calculating the mutual information loss mLoss between each feature spectrum after performing a linear transformation on each feature spectrum, and calculating the total information loss L. M , respectively represented as:

[0021]

[0022] Where, patches represents the number of blocks in the feature spectrum in MSTM-VMF, F' RGB 、F' YUV 、F' YCrCb These are the characteristic spectra after linear transformation.

[0023] Furthermore, the total loss function L of the physiological detection network all Represented as:

[0024]

[0025] Where λ is a hyperparameter; Let L represent the mean squared error loss function for predicting the k-th physiological parameter, where k∈1:n, and n is the total number of categories of the physiological indicator to be measured; M L represents the total information loss. triplet This represents the triple loss function.

[0026] Furthermore, a multilayer perceptron is used to perform regression analysis on the output features to obtain the detection results of the physiological parameters of the target.

[0027] The present invention also includes a non-contact visual perception physiological parameter detection system, comprising:

[0028] The acquisition module is used to acquire facial video data of the target under test in different bands and extract the region of interest (ROI) of the face; the facial video data of different bands corresponding to the ROI are transformed into a multi-color space and the corresponding spatiotemporal feature spectra are extracted respectively.

[0029] The computation module inputs each spatiotemporal feature spectrum into the physiological detection network, and initially extracts feature maps of each spatiotemporal feature spectrum through a compressed excitation residual initial convolution module. Each feature map is then input into a multi-scale dense visual state space module, where a downsampling unit extracts feature maps that can represent physiological features. Each feature map that can represent physiological features is then input into four densely connected multi-scale visual state space (MVSS) units to extract the corresponding feature spectrum. The MSTM-VMF module is used to perform linear transformation on each feature spectrum and then concatenate them. The MVSS module is used to extract features from the concatenated feature spectrum, and the extracted features are then subjected to global average pooling to obtain the output features.

[0030] The detection module is used to detect the physiological parameters of the target based on the output features.

[0031] The present invention also includes a computer device for detecting physiological parameters through non-contact visual perception, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the non-contact visual perception method for detecting physiological parameters.

[0032] The present invention also includes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of the non-contact visual perception physiological parameter detection method.

[0033] This invention provides a non-contact visual perception method, system, and device for detecting physiological parameters, which has the following beneficial effects:

[0034] This invention innovatively utilizes visual perception technology, avoiding direct contact with subjects and improving the comfort and convenience of monitoring. By converting facial video data from different bands of the facial ROI into multi-color spaces, corresponding spatiotemporal feature spectra are extracted, fully characterizing the dynamic changes in facial color information. A multi-scale dense visual state space module is designed to extract and fuse important information from the generated spatiotemporal feature spectra of the multi-color spaces. This module, through multi-scale analysis, can capture visual features at different scales, enhancing the network's sensitivity to subtle physiological signals. Then, a multi-spatiotemporal visual Mamba fusion module integrates feature information from various color spaces, achieving predictive output through deep fusion. This method combines data from different spectra and color spaces, enhancing the ability to extract subtle physiological signals and improving the accuracy and reliability of detection. Simultaneously, the multi-scale dense visual state space module allows for more comprehensive capture and analysis of physiological features, and the fusion prediction module effectively integrates information from multiple spatiotemporal feature spectra, enabling simultaneous detection of multiple physiological parameters and improving monitoring efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart of a non-contact visual perception physiological parameter detection method in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of spatiotemporal feature spectrum generation in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of the SER module, MVSS module and MSTM-VMF module in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0039] This invention proposes a non-contact visual perception method for detecting physiological parameters such as heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature. First, a multispectral facial region automatic extraction module is designed to automatically extract facial regions from natural light, red light, blue light, and thermal infrared video stream data. Next, a multi-color space-based spatiotemporal feature spectrum generation module is designed to extract spatiotemporal feature spectra from the multispectral video data in three typical color spaces. Then, an innovative physiological measurement network is designed, including a multi-scale dense visual state space module and a fusion prediction module, to fully extract and organically fuse the important information contained in the three spatiotemporal feature spectra, achieving non-contact detection of multiple physiological parameters. Figure 1As shown, the specific steps include:

[0040] S1. Multispectral Data Acquisition: Based on a multispectral video sensor, natural light, red light, blue light, and thermal infrared video stream data of the face are acquired to provide data support for subsequent analysis. To reduce the interference of head movement, light changes, and other factors caused by long-term acquisition on the data and the impact on the measurement results, each set of data is acquired for only 2 minutes.

[0041] S2. Multispectral Facial Region Automatic Extraction Module: Based on the Kanade-Lucas-Tomasi (KLT) facial unit recognition algorithm, it automatically detects and labels 68 key points of the face. Based on this, it automatically extracts a rectangular Region of Interest (ROI) extending from the eyebrows to both cheeks to the bottom of the jaw, preserving information from important facial areas to the maximum extent while discarding redundant information outside the ROI. Specifically, it includes the following steps:

[0042] (1) The KLT facial unit recognition algorithm was used to automatically label 68 marker points in the face video, covering the facial region from the brow bone to both cheeks and then to the jaw. Based on the spatial horizontal and vertical coordinate relationships of these 68 marker points, the position coordinates of the upper left corner of the ROI (X) were automatically selected. c ,Y c The dimensions of the ROI (W×H) are shown in formula (1):

[0043]

[0044] Among them, X n and Y n These represent the vertical and horizontal coordinates of the 68 facial markers in each frame of the video.

[0045] (2) The facial ROI is transformed from the original RGB color space to the YUV and YCrCb color spaces that can effectively reflect the skin color state, as shown in formulas (2) and (3):

[0046]

[0047] (3) The facial ROIs under natural light, red light, blue light, and thermal infrared data are divided into 20×16 sub-regions (subROIs). i (i∈20×16), so as to calculate the spatiotemporal feature spectrum later.

[0048] S3. Spatiotemporal Feature Spectrum Generation Module Based on Multi-Color Space: First, the natural light, red light, blue light, and thermal infrared video data of the facial ROI are converted to RGB, YUV, and YCrCb color spaces respectively. Then, the corresponding spatiotemporal feature spectra are extracted from them to fully characterize the dynamic changes of facial color information. Specifically, this includes the following steps:

[0049] (1) By dividing the facial ROI of each light source into several sub-regions, from each sub-ROI i The pixel mean values ​​from the RGB, YUV, and YCrCb color spaces are extracted along their temporal axis to form a spatiotemporal feature signal line. Taking the natural light subROI1 data in the RGB color space as an example, the pixel mean values ​​of subROI1 and its temporal axis in the R, G, and B channels are extracted to form three spatiotemporal feature signal lines, which are arranged sequentially in the first three rows of the RGB spatiotemporal feature spectrum. Then, the above steps are repeated until all natural light subROIs are traversed. i The spatiotemporal characteristic spectrum of natural light in the RGB color space is calculated and generated.

[0050] (2) Calculate the spatiotemporal characteristic spectral lines of red light, blue light and thermal infrared data in the RGB color space in sequence and arrange them in rows to finally generate the spatiotemporal characteristic spectrum of the above multispectral data in the RGB color space.

[0051] (3) Based on the spatiotemporal characteristic spectrum generation method of multispectral data in RGB color space, calculate the spatiotemporal characteristic spectrum of YUV and YCrCb color spaces in turn.

[0052] S4, the physiological measurement network, mainly comprises two modules: a multi-scale dense visual state space module and a fusion prediction module. The network employs a multi-scale dense visual state space module designed to extract and fuse important information from the spatiotemporal feature spectra of the generated RGB, YUV, and YCrCb color spaces. Through multi-scale analysis, this module captures visual features at different scales, enhancing the network's sensitivity to subtle physiological signals. The fusion prediction module integrates feature information from various color spaces, achieving predictive output through deep fusion. Several innovative loss functions are proposed, including mutual information loss (mLoss) and total information loss (L). M Total loss function L all This is used to update network weights, accelerate convergence, and improve detection accuracy. Based on this physiological measurement network, the system ultimately detects and outputs multiple physiological parameters such as heart rate, respiration, blood pressure, blood oxygen saturation, and body temperature. Specifically, it includes the following steps:

[0053] First, the spatiotemporal feature spectra of RGB, YUV, and YCrCb are respectively input into a squeeze-excitation residual (SER) initial convolution module to initially extract the effective information of each spatiotemporal feature spectrum and remove redundant information.

[0054] Secondly, the feature maps output by the SER module are further input into the multi-scale dense visual state space module for information extraction and analysis, such as... Figure 2As shown. This module includes a downsampling unit with a stride of 2, and four densely connected multiscale visual space (MVSS) units. Each MVSS contains the following main nodes: Figure 3 As shown. First, the downsampling unit is used to simplify the feature map from the SER module, retaining important features. Then, the feature map passes through the MVSS unit for further separation and extraction of more effective deep features. Here, the present invention innovatively designs an MVSS network based on dense connections to preserve and continuously extract important features across different MVSS units, avoiding the marginalization of important features in multiple MVSS operations and affecting the accuracy of physiological parameter estimation. At the same time, the LeakyHardSwish activation function is innovatively proposed, as shown in Equation (4), to solve the problem of vanishing gradients in the network and enhance convergence efficiency.

[0055]

[0056] The three-channel feature spectrum F output from the RGB, YUV, and YCrCb color spaces via the multi-scale dense state space module. RGB F YUV F YCrCb The feature spectra are fully fused using the Multiple Spatial-Temporal Map Visual Mamba Fusion (MSTM-VMF) module. Specifically, the three feature spectra are first linearly transformed to obtain a new feature spectrum F'. RGB 、F' YUV 、F' YCrCb ,like Figure 3 As shown. Next, the mutual information loss mLoss among the above feature spectra is calculated, and based on this, the total information loss L is further calculated. M As shown in formulas (5) and (6), these are used to adjust network parameters and control convergence.

[0057]

[0058] Here, patches represents the number of blocks in this module for the above feature spectrum.

[0059] The characteristic spectrum F' RGB 、F' YUV 、F' YCrCb Perform a concatenate operation to concatenate the elements into F' all Then, an MVSS module is used for further feature extraction and global average pooling to finally obtain the output feature f'. outAnd set a triple loss function L triplet It is used to effectively control overfitting caused by class imbalance.

[0060] Feature f' out Regression analysis was performed using a multi-layer perceptron (MLP) to predict multiple physiological parameters. The loss function was the mean squared error (MSE) loss function. k∈1:n, where k represents the mean squared error loss function for predicting the k-th physiological parameter, and L represents the total loss function of the network. all As shown in formula (7). Here, λ is a hyperparameter used to balance the total information loss L. M :

[0061]

[0062] System Integration and Testing: Integrate all modules into a unified system platform. Conduct system-level testing to verify the accuracy of data acquisition, the effectiveness of information processing, and the real-time performance and accuracy of physiological parameter detection. Optimize system performance based on test results to improve detection stability and response speed.

[0063] Based on the same inventive concept, this invention also proposes a non-contact visual perception physiological parameter detection system, comprising:

[0064] The acquisition module is used to acquire facial video data of the target under test in different bands and extract the region of interest (ROI) of the face; the facial video data of different bands corresponding to the ROI are transformed into a multi-color space and the corresponding spatiotemporal feature spectra are extracted respectively.

[0065] The computation module is used to input each spatiotemporal feature spectrum into the physiological detection network. A compressed excitation residual initial convolution module is used to initially extract feature maps for each spatiotemporal feature spectrum. Each feature map is then input into a multi-scale dense visual state space module, where a downsampling unit extracts features from each input feature map to obtain feature maps that represent physiological characteristics. Each feature map representing a physiological characteristic is then input into four densely connected multi-scale visual state space (MVSS) units to extract the corresponding feature spectrum. The MSTM-VMF multi-spatiotemporal visual Mamba fusion module is used to linearly transform and concatenate each feature spectrum. The MVSS module is then used to extract features from the concatenated feature spectrum. Finally, the extracted features are subjected to global average pooling to obtain the output features.

[0066] The detection module is used to detect the physiological parameters of the target based on the output features.

[0067] The present invention also proposes a computer device for detecting physiological parameters through non-contact visual perception, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for detecting physiological parameters through non-contact visual perception.

[0068] The present invention also proposes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform steps of a non-contact visual perception physiological parameter detection method.

[0069] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting physiological parameters using non-contact visual perception, characterized in that, Includes the following steps: Collect facial video data of the target under different bands and extract the region of interest (ROI) of the face; transform the facial video data of different bands corresponding to the ROI to a multi-color space and extract the corresponding spatiotemporal feature spectra. Each spatiotemporal feature spectrum is input into the physiological detection network, and the feature map of each spatiotemporal feature spectrum is initially extracted through the compressed excitation residual initial convolution module; Each feature map is input into a multi-scale dense visual state space module. A downsampling unit extracts each input feature map to obtain a feature map representing physiological characteristics. Each feature map representing physiological characteristics is then input into four densely connected multi-scale visual state space (MVSS) modules to extract the corresponding feature spectrum. The Multi-Spatiotemporal Visual Mamba Fusion (MSTM-VMF) module performs a linear transformation on each feature spectrum and then concatenates them. The MVSS module connected to the MSTM-VMF module then extracts features from the concatenated feature spectrum. The extracted features are then subjected to global average pooling to obtain the output features. The physiological parameters of the target are detected based on the output features.

2. The method for detecting physiological parameters of non-contact visual perception according to claim 1, characterized in that, The KLT facial unit recognition algorithm is used to automatically mark multiple marker points on the target face. Based on each marker point, the facial location area from the glabella to the cheeks and then to the jaw is automatically extracted, thereby obtaining the facial region of interest (ROI).

3. The method for detecting physiological parameters of non-contact visual perception according to claim 1, characterized in that, The process of transforming the facial video data of different bands corresponding to the ROI to a multi-color space and extracting the corresponding spatiotemporal feature spectra includes the following steps: The ROI was transformed from the original RGB color space to the YUV and YCrCb color spaces, which reflect skin color. The ROIs of natural light, red light, blue light, and thermal infrared data were divided into multiple sub-regions. ; From each sub-region subROIi, the pixel mean values ​​from the RGB, YUV, and YCrCb color spaces are extracted along its time axis to form a spatiotemporal feature value signal line. The spatiotemporal characteristic spectral signal lines of different band video data of the facial ROI in RGB, YUV and YCrCb color spaces are calculated sequentially and arranged in rows to generate the spatiotemporal characteristic spectrum of multispectral data in RGB, YUV and YCrCb color spaces.

4. The method for detecting physiological parameters of non-contact visual perception according to claim 1, characterized in that, The LeakyHardSwish activation function in the MVSS module is expressed as follows: in, x This represents the input features; the input features are the features output after LayerNorm transformation in the MVSS module.

5. The method for detecting physiological parameters of non-contact visual perception according to claim 3, characterized in that, This also includes calculating the mutual information loss between each feature spectrum after performing a linear transformation on each feature spectrum. mLoss And calculate the total information loss. , respectively represented as: in, patches This indicates the number of blocks in the characteristic spectrum within the MSTM-VMF. , , These are the characteristic spectra after linear transformation.

6. The method for detecting physiological parameters of non-contact visual perception according to claim 5, characterized in that, The total loss function of the physiological detection network Represented as: in, It is a hyperparameter; Indicates the first The mean squared error loss function for predicting various physiological parameters. , This represents the total number of categories of the physiological indicators to be measured. Indicates the total information loss. This represents the triple loss function.

7. The method for detecting physiological parameters of non-contact visual perception according to claim 1, characterized in that, Multilayer perceptron was used to perform regression analysis on the output features to obtain the detection results of the physiological parameters of the target.

8. A non-contact visual perception physiological parameter detection system, characterized in that, include: The acquisition module is used to acquire facial video data of the target under test in different bands and extract the region of interest (ROI) of the face. The facial video data of different bands corresponding to the ROI are transformed into multiple color spaces, and the corresponding spatiotemporal feature spectra are extracted respectively. The computation module is used to input each spatiotemporal feature spectrum into the physiological detection network, and to initially extract the feature map of each spatiotemporal feature spectrum through the compressed excitation residual initial convolution module; Each feature map is input into a multi-scale dense visual state space module. A downsampling unit extracts each input feature map to obtain a feature map representing physiological characteristics. Each feature map representing physiological characteristics is then input into four densely connected multi-scale visual state space (MVSS) modules to extract the corresponding feature spectrum. The Multi-Spatiotemporal Visual Mamba Fusion (MSTM-VMF) module performs a linear transformation on each feature spectrum and then concatenates them. The MVSS module connected to the MSTM-VMF module then extracts features from the concatenated feature spectrum. The extracted features are then subjected to global average pooling to obtain the output features. The detection module is used to detect the physiological parameters of the target based on the output features.

9. A non-contact visual perception physiological parameter detection computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the non-contact visual perception physiological parameter detection method according to any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, are used to perform the steps of the non-contact visual perception physiological parameter detection method according to any one of claims 1-7.

Images