Non-contact visual perception physiological parameter detection method, system and device

Abstract

The invention discloses a non-contact visual perception physiological parameter detection method, system and device, and relates to the technical field of visual physiological index measurement, and the method comprises the steps: collecting face video data under different wavebands, and extracting a face region of interest ROI; converting the video data of different wave bands of the face ROI into a multi-color space, and extracting corresponding spatio-temporal characteristic spectrums; inputting each spatio-temporal characteristic spectrum into the physiological detection network, and preliminarily extracting a characteristic pattern of each spatio-temporal characteristic spectrum; inputting each feature map into a multi-scale dense visual state space module, and outputting a corresponding feature spectrum; fusing each characteristic spectrum by using a multiple space-time atlas vision Mangbar fusion module (MSTM-VMF) to obtain an output characteristic; according to the method, information of various spatio-temporal characteristic spectrums is effectively integrated through the fusion prediction module, simultaneous detection of various physiological parameters is achieved, and the monitoring efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual physiological index measurement, and in particular to a physiological parameter detection method, system and device based on non-contact visual perception. Background Art

[0002] Physiological parameter testing is a crucial medical practice in emergency treatment and chronic disease monitoring for the injured and sick. However, conventional physiological testing often relies on contact-based testing methods, such as multi-parameter physiological monitors, which use various contact sensors to measure a patient's heart rate, blood pressure, blood oxygen saturation, respiratory rate, body temperature, and other vital parameters in real time. Considering the large number of injured individuals who urgently require physiological testing during large-scale infectious disease outbreaks or major casualty accidents, contact-based physiological parameter testing equipment is not only inefficient and limited in stock, but also affected by the specific location of the injury, increasing the risk of cross-infection between medical staff and patients.

[0003] The proposed physiological parameter detection method uses wavelet transforms to decompose 3D video data frame by frame, suppressing noise while preserving important information such as remote photoplethysmography. Based on the wavelet-transformed data, a spatiotemporal signature is constructed. This signature is then fed into a CNN for feature analysis, extracting the corresponding heart rate value.

[0004] However, this method does not fully utilize the important information contained in the original RGB color space of the video data, which makes it difficult to extract subtle physiological signals and reduces the accuracy of detection. Summary of the Invention

[0005] In view of the deficiency of the existing technology that the important information contained in the original RGB color space of video data is not fully utilized, resulting in difficulty in extracting subtle physiological signals, the present invention proposes a physiological parameter detection method, system and device based on non-contact visual perception. By combining data from different spectra and color spaces, the ability to extract subtle physiological signals is enhanced, the accuracy and reliability of detection are improved, and thus the problems existing in the existing technology are solved.

[0006] A method for detecting physiological parameters by non-contact visual perception comprises the following steps:

[0007] Collect facial video data of the target under test in different bands and extract the facial region of interest (ROI); transform the facial video data of different bands corresponding to the ROI into multi-color space and extract the corresponding spatiotemporal feature spectra;

[0008] Each spatiotemporal feature spectrum is input into the physiological detection network, and the feature map of each spatiotemporal feature spectrum is initially extracted through the compressed excitation residual initial convolution module; each feature map is input into the multi-scale dense visual state space module, and each input feature map is simplified through the downsampling unit to obtain a feature map that can represent the physiological characteristics, and each feature map that can represent the physiological characteristics is input into four densely connected multi-scale visual state space MVSS units to extract the corresponding feature spectrum; each feature spectrum is linearly transformed and then spliced ​​using the multiple spatiotemporal map visual mamba fusion module MSTM-VMF, and the spliced ​​feature spectrum is extracted using the MVSS module. The extracted features are subjected to global average pooling to obtain the output features;

[0009] The physiological parameters of the target to be measured are detected according to the output characteristics.

[0010] Furthermore, the KLT facial unit recognition algorithm is used to automatically mark multiple marker points on the target face. Based on each marker point, the facial position area from the eyebrows to the cheeks and then to the mandible is automatically extracted to obtain the facial region of interest ROI.

[0011] Furthermore, the facial video data of different bands corresponding to the ROI are transformed into multi-color spaces respectively, and the corresponding spatiotemporal feature spectra are extracted respectively, which specifically includes the following steps:

[0012] Convert the ROI from the original RGB color space to the YUV and YCrCb color spaces that can reflect the skin color state:

[0013]

[0014] Divide the ROI of natural light, red light, blue light, and thermal infrared data into multiple sub-regions subROI i ;

[0015] From each sub-region subROI i The pixel mean values ​​from RGB, YUV, and YCrCb color spaces are extracted in the time longitudinal direction to form a spatiotemporal eigenvalue signal line;

[0016] The spatiotemporal characteristic spectrum signal lines of different bands of video data of facial ROI in RGB, YUV and YCrCb color spaces are calculated in sequence and arranged in rows to generate the spatiotemporal characteristic spectrum of multispectral data in RGB, YUV and YCrCb color spaces.

[0017] Furthermore, the LeakyHardSwish activation function is expressed as:

[0018]

[0019] Where x represents the input feature.

[0020] Furthermore, after each feature spectrum is linearly transformed, the mutual information loss mLoss between each feature spectrum is calculated, and the total information loss L is calculated. M , respectively expressed as:

[0021]

[0022] Among them, patches represents the number of blocks of the characteristic spectrum in MSTM-VMF, F' RGB 、F' YUV 、F' YCrCb are the characteristic spectra after linear transformation.

[0023] Furthermore, the total loss function L of the physiological detection network is all Expressed as:

[0024]

[0025] Among them, λ is a hyperparameter; represents the mean square error loss function for the prediction of the kth physiological parameter, k∈1:n, n is the total number of categories of the physiological indicators to be measured; L M represents the total information loss, L triplet represents the triplet loss function.

[0026] Furthermore, a multi-layer perceptron is used to perform regression analysis on the output features to obtain the detection results of the physiological parameters of the target to be measured.

[0027] The present invention also includes a non-contact visual perception physiological parameter detection system, comprising:

[0028] The acquisition module is used to collect facial video data of the target under test in different bands and extract the facial region of interest (ROI); transform the facial video data of different bands corresponding to the ROI into multi-color space and extract the corresponding spatiotemporal feature spectrum;

[0029] The computing module is used to input each spatiotemporal feature spectrum into the physiological detection network, and preliminarily extract the feature map of each spatiotemporal feature spectrum through the compressed excitation residual initial convolution module; input each feature map into the multi-scale dense visual state space module, extract each input feature map through the downsampling unit to obtain a feature map that can represent the physiological characteristics, and input each feature map that can represent the physiological characteristics into four densely connected multi-scale visual state space MVSS units to extract the corresponding feature spectrum; use the multiple spatiotemporal map visual mamba fusion module MSTM-VMF to splice each feature spectrum after linear transformation, use the MVSS module to extract features from the spliced ​​feature spectrum, and obtain the output feature after global average pooling of the extracted features;

[0030] The detection module is used to detect the physiological parameters of the target to be detected based on the output characteristics.

[0031] The present invention also includes a computer device for detecting physiological parameters using non-contact visual perception, comprising: a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the steps of the method for detecting physiological parameters using non-contact visual perception are implemented.

[0032] The present invention also includes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the non-contact visual perception physiological parameter detection method.

[0033] The present invention provides a method, system, and device for detecting physiological parameters using non-contact visual perception, which have the following beneficial effects:

[0034] This invention innovatively uses visual perception technology to avoid direct contact with the subject, improving the comfort and convenience of monitoring. By converting facial video data from different bands of the facial ROI into multi-color spaces, the corresponding spatiotemporal feature spectra are extracted from them, fully characterizing the dynamic changes in facial color information. A multi-scale dense visual state space module is designed to extract and fuse important information from the generated spatiotemporal feature spectra of the multi-color space. This module, through multi-scale analysis, can capture visual features at different scales and enhance the network's sensitivity to subtle physiological signals. The multi-scale spatiotemporal map visual mamba fusion module then integrates feature information from each color space, achieving a prediction output through deep fusion. This method combines data from different spectra and color spaces, enhancing the ability to extract subtle physiological signals and improving detection accuracy and reliability. Simultaneously, the multi-scale dense visual state space module can more comprehensively capture and analyze physiological features. Furthermore, the fusion prediction module can effectively integrate information from multiple spatiotemporal feature spectra, enabling simultaneous detection of multiple physiological parameters and improving monitoring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for detecting physiological parameters using non-contact visual perception according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of generating a spatiotemporal characteristic spectrum according to an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the structure of the SER module, MVSS module and MSTM-VMF module in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0039] The present invention proposes a non-contact visual perception physiological parameter detection method for non-contact detection of human physiological parameters such as heart rate, blood pressure, blood oxygen saturation, respiratory rate, body temperature, etc. First, a multi-spectral facial area automatic extraction module is designed to automatically extract facial areas from natural light, red light, blue light, and thermal infrared video stream data; then, a spatiotemporal feature spectrum generation module based on multi-color space is designed to extract spatiotemporal feature spectra of the above multi-spectral video data in three typical color spaces; then, an innovative physiological measurement network is designed, including a multi-scale dense visual state space module and a fusion prediction module, which fully extracts the important information contained in the above three spatiotemporal feature spectra and organically integrates them to achieve non-contact detection of multiple physiological parameters. Figure 1As shown, the specific steps include:

[0040] S1. Multispectral data acquisition: Based on a multispectral video sensor, natural light, red light, blue light, and thermal infrared video stream data of the human face are collected to provide data support for subsequent analysis. To reduce the interference of factors such as head movement and light changes caused by long-term collection and the impact on the measurement results, each set of data is only collected for 2 minutes.

[0041] S2, Multispectral Facial Region Automatic Extraction Module: Based on the Kanade-Lucas-Tomasi (KLT) facial unit recognition algorithm, it automatically detects and marks 68 key points on the face. Based on this, it automatically extracts a rectangular ROI region from the eyebrows to the cheeks and the bottom of the jaw, preserving the most important facial area information and discarding redundant information outside the ROI. The specific steps include:

[0042] (1) Using the KLT facial unit recognition algorithm, 68 markers are automatically marked in the face video, including the facial position area from the eyebrows to the cheeks and then to the mandible. Based on the spatial horizontal and vertical coordinate position relationship of these 68 markers, the position coordinates of the upper left corner of the ROI (X c ,Y c ), and the width and height of ROI (W×H), as shown in formula (1):

[0043]

[0044] Among them, X n and Y n They represent the vertical and horizontal coordinate positions of the 68 facial landmarks in each frame of video.

[0045] (2) The facial ROI is transformed from the original RGB color space to the YUV and YCrCb color spaces that can effectively reflect the skin color state, as shown in formulas (2) and (3):

[0046]

[0047] (3) Divide the facial ROI of natural light, red light, blue light, and thermal infrared data into 20×16 sub-regions (subROI i ,i∈20×16) for the subsequent calculation of the spatiotemporal feature spectrum.

[0048] S3, a module for generating spatiotemporal signatures based on multiple color spaces: First, the natural light, red light, blue light, and thermal infrared video data of the facial ROI are converted into RGB, YUV, and YCrCb color spaces, respectively. Then, the corresponding spatiotemporal signatures are extracted from these colors to fully characterize the dynamic changes in facial color information. Specifically, the following steps are involved:

[0049] (1) By dividing the facial ROI of each light source into several sub-regions, i The pixel mean values ​​from RGB, YUV, and YCrCb color spaces are extracted in the time longitudinal direction to form a spatiotemporal eigenvalue signal line. Taking the data of natural light subROI1 in RGB color space as an example, the pixel mean values ​​of subROI1 and its time longitudinal direction in R, G, and B channels are first extracted to form three spatiotemporal eigenvalue signal lines, which are arranged in the first three rows of the RGB spatiotemporal feature spectrum. After that, the above steps are repeated until all natural light subROIs are traversed. i , calculate and generate the spatiotemporal characteristic spectrum of natural light in RGB color space.

[0050] (2) The spatiotemporal characteristic spectra of red light, blue light, and thermal infrared data in the RGB color space are calculated in sequence and arranged in rows, and finally the spatiotemporal characteristic spectra of the above multispectral data in the RGB color space are generated.

[0051] (3) Based on the method of generating the spatiotemporal characteristic spectrum of multispectral data in RGB color space, the spatiotemporal characteristic spectrum of YUV and YCrCb color space is calculated in turn.

[0052] S4, physiological measurement network; mainly includes two main modules: multi-scale dense visual state space module and fusion prediction module. The network designs a multi-scale dense visual state space module, which aims to extract and fuse important information from the spatiotemporal feature spectra of the generated RGB, YUV, and YCrCb color spaces. Through multi-scale analysis, the module can capture visual features at different scales and enhance the network's sensitivity to subtle physiological signals. The fusion prediction module integrates feature information from each color space and realizes prediction output through deep fusion. Among them, a variety of loss functions are innovatively proposed, including mutual information loss mLoss, total information loss L M , total loss function L all , used to update network weights, accelerate convergence, and improve detection accuracy. Based on this physiological measurement network, multiple physiological parameter detection outputs such as heart rate, respiration, blood pressure, blood oxygen saturation, and body temperature are ultimately achieved. Specifically, the following steps are included:

[0053] First, the RGB, YUV, and YCrCb spatiotemporal feature spectra are respectively input into a Squeeze-Excitation Residual (SER) initial convolution module to preliminarily extract the effective information of each spatiotemporal feature spectrum and remove redundant information.

[0054] Secondly, the feature map output by the SER module is further input into the multi-scale dense visual state space module for information extraction and analysis. Figure 2The module includes a downsampling unit with a stride of 2 and four multiscale visual space state (MVSS) units based on dense connections. The main nodes of each MVSS are as follows: Figure 3 As shown in Figure 4. First, the downsampling unit is used to simplify the feature map from the SER module and retain important features. Afterwards, the feature map passes through the MVSS unit to further separate and extract more effective deep features. Here, the present invention innovatively designs a densely connected MVSS network for the retention and continuity extraction of important features between different MVSS units, avoiding the marginalization of important features in multiple MVSS operations, which affects the accuracy of the estimation of physiological parameters. At the same time, the innovative LeakyHardSwish activation function is proposed, as shown in formula (4), to solve the problem of network gradient disappearance and enhance convergence efficiency.

[0055]

[0056] The 3-way feature spectrum F output by the multi-scale dense state space module of RGB, YUV, and YCrCb color spaces RGB 、F YUV 、F YCrCb , and then fully fused by the Multiple Spatial-Temporal Map Visual Mamba Fusion (MSTM-VMF) module. Among them, the above three feature spectra are first linearly transformed to obtain a new feature spectrum F' RGB 、F' YUV 、F' YCrCb ,like Figure 3 As shown. Afterwards, the mutual information loss mLoss between the above feature spectra is calculated, and the total information loss L is further calculated based on this. M , as shown in formula (5) and formula (6), is used to adjust network parameters and control convergence.

[0057]

[0058] Among them, patches represents the number of blocks of the above feature spectrum in this module.

[0059] The characteristic spectrum F' RGB 、F' YUV 、F' YCrCb Perform the Concatenate operation to form F' all , followed by an MVSS module for further feature extraction, and global averagepooling operation, and finally the output feature f' out, and set a triple loss function L triplet , which is used to effectively control the overfitting phenomenon caused by category imbalance.

[0060] Feature f' out A multi-layer perceptron (MLP) is used for regression analysis to achieve multi-physiological parameter prediction. The loss function is the mean square error (MSE), i.e. k∈1:n, represents the mean square error loss function of the kth physiological parameter prediction, and the total loss function of the network L all As shown in Formula (7). Where λ is a hyperparameter used to balance the total information loss L M :

[0061]

[0062] System Integration and Testing: Integrate all modules into a unified system platform. Conduct system-level testing to verify the accuracy of data acquisition, the effectiveness of information processing, and the real-time and accuracy of physiological parameter detection. Based on test results, optimize system performance and improve detection stability and response speed.

[0063] Based on the same inventive concept, the present invention also proposes a non-contact visual perception physiological parameter detection system, comprising:

[0064] The acquisition module is used to collect facial video data of the target under test in different bands and extract the facial region of interest ROI; the facial video data of different bands corresponding to the ROI are transformed into multi-color space respectively, and the corresponding spatiotemporal feature spectra are extracted respectively.

[0065] The computing module is used to input each spatiotemporal feature spectrum into the physiological detection network, and preliminarily extract the feature map of each spatiotemporal feature spectrum through the compressed excitation residual initial convolution module; input each feature map into the multi-scale dense visual state space module, extract each input feature map through the downsampling unit to obtain a feature map that can represent the physiological characteristics, and input each feature map that can represent the physiological characteristics into four densely connected multi-scale visual state space MVSS units to extract the corresponding feature spectrum; use the multiple spatiotemporal map visual mamba fusion module MSTM-VMF to splice each feature spectrum after linear transformation, use the MVSS module to extract features from the spliced ​​feature spectrum, and obtain the output features after global average pooling of the extracted features.

[0066] The detection module is used to detect the physiological parameters of the target to be detected based on the output characteristics.

[0067] The present invention also proposes a non-contact visual perception physiological parameter detection computer device, including: a memory, a processor and a computer program stored in the memory, and when the processor executes the computer program, the steps of the non-contact visual perception physiological parameter detection method are implemented.

[0068] The present invention also proposes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the physiological parameter detection method based on non-contact visual perception.

[0069] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A non-contact visual perception physiological parameter detection method, characterized in that: The following steps are involved: Collect facial video data of the target under test in different bands and extract the facial region of interest (ROI); transform the facial video data of different bands corresponding to the ROI into multi-color space and extract the corresponding spatiotemporal feature spectra; Each spatiotemporal feature spectrum is input into the physiological detection network, and the feature map of each spatiotemporal feature spectrum is initially extracted through the compressed excitation residual initial convolution module; Each feature map is input into a multi-scale dense visual state space module, and each input feature map is extracted through a downsampling unit to obtain a feature map that can represent physiological features. Each feature map that can represent physiological features is input into four densely connected multi-scale visual state space MVSS units to extract the corresponding feature spectrum. The multiple spatiotemporal map visual mamba fusion module MSTM-VMF is used to splice each feature spectrum after linear transformation. The MVSS module is used to extract features from the spliced ​​feature spectrum. The extracted features are subjected to global average pooling to obtain the output features. The physiological parameters of the target to be measured are detected according to the output characteristics.

2. The method for detecting physiological parameters by non-contact visual perception according to claim 1, characterized in that: The KLT facial unit recognition algorithm is used to automatically mark multiple marker points on the target face. Based on each marker point, the facial position area from the eyebrows to the cheeks and then to the mandible is automatically extracted to obtain the facial region of interest (ROI).

3. The method for detecting physiological parameters by non-contact visual perception according to claim 1, characterized in that: The facial video data of different bands corresponding to the ROI are transformed into multi-color spaces respectively, and the corresponding spatiotemporal feature spectra are extracted respectively, which specifically includes the following steps: Convert the ROI from the original RGB color space to the YUV and YCrCb color spaces that can reflect the skin color state: Divide the ROI of natural light, red light, blue light, and thermal infrared data into multiple sub-regions subROI i ; From each sub-region subROI i The pixel mean values ​​from RGB, YUV, and YCrCb color spaces are extracted in the time longitudinal direction to form a spatiotemporal eigenvalue signal line; The spatiotemporal characteristic spectrum signal lines of different bands of video data of facial ROI in RGB, YUV and YCrCb color spaces are calculated in sequence and arranged in rows to generate the spatiotemporal characteristic spectrum of multispectral data in RGB, YUV and YCrCb color spaces.

4. The method for detecting physiological parameters by non-contact visual perception according to claim 1, characterized in that: The LeakyHardSwish activation function is expressed as: Where x represents the input feature.

5. The method for detecting physiological parameters by non-contact visual perception according to claim 3, characterized in that: It also includes calculating the mutual information loss mLoss between each feature spectrum after linear transformation of each feature spectrum, and calculating the total information loss L M , respectively expressed as: Among them, patches represents the number of blocks of the characteristic spectrum in MSTM-VMF, F' RGB 、F' YUV 、F' YCrCb are the characteristic spectra after linear transformation.

6. The method for detecting physiological parameters by non-contact visual perception according to claim 5, characterized in that: The total loss function L of the physiological detection network all Expressed as: Among them, λ is a hyperparameter; represents the mean square error loss function for the prediction of the kth physiological parameter, k∈1:n, n is the total number of categories of the physiological indicators to be measured; L M represents the total information loss, L triplet represents the triplet loss function.

7. The method for detecting physiological parameters by non-contact visual perception according to claim 1, characterized in that: A multi-layer perceptron is used to perform regression analysis on the output features to obtain the detection results of the physiological parameters of the target to be measured.

8. A non-contact visual perception physiological parameter detection system, characterized in that: include: The acquisition module is used to collect facial video data of the target under test in different bands and extract the facial region of interest (ROI); The facial video data of different bands corresponding to the ROI are transformed into multi-color space respectively, and the corresponding spatiotemporal feature spectra are extracted respectively; A calculation module is used to input each spatiotemporal feature spectrum into the physiological detection network and preliminarily extract the feature map of each spatiotemporal feature spectrum through the compressed excitation residual initial convolution module; Each feature map is input into a multi-scale dense visual state space module, and each input feature map is extracted through a downsampling unit to obtain a feature map that can represent physiological features. Each feature map that can represent physiological features is input into four densely connected multi-scale visual state space MVSS units to extract the corresponding feature spectrum. The multiple spatiotemporal map visual mamba fusion module MSTM-VMF is used to splice each feature spectrum after linear transformation. The MVSS module is used to extract features from the spliced ​​feature spectrum. The extracted features are subjected to global average pooling to obtain the output features. The detection module is used to detect the physiological parameters of the target to be detected based on the output characteristics.

9. A non-contact visual perception physiological parameter detection computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the steps of the physiological parameter detection method based on non-contact visual perception according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the non-contact visual perception physiological parameter detection method according to any one of claims 1 to 7.

Images