A non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion

Abstract

This invention belongs to the field of computer vision and intelligent health monitoring technology, and specifically provides a non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion. The method includes: acquiring RGB and near-infrared videos containing the human face region, and simultaneously acquiring heart rate and blood oxygen saturation reference signals; constructing a standardized dataset after preprocessing; building a fusion-based dual-path physiological signal estimation network to extract spatiotemporal features from the RGB and near-infrared face sequences; introducing a gating enhancement module to perform weighted modulation of the two feature streams; setting a multimodal feature fusion module to obtain a joint feature representation; constructing a physiological parameter estimation branch to achieve pulse wave signal reconstruction and joint prediction of heart rate and blood oxygen saturation; designing a joint loss function, performing end-to-end training and optimization of the network, and outputting the corresponding heart rate and blood oxygen saturation estimation results. This invention can effectively improve the accuracy and robustness of non-contact physiological parameter estimation in complex environments.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent health monitoring technology, specifically to a non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion. Background Technology

[0002] Non-contact physiological parameter monitoring technology analyzes subtle changes in human body surface images or videos to remotely estimate physiological parameters such as heart rate and blood oxygen saturation. It offers advantages such as being contactless, convenient to use, and suitable for long-term monitoring, and has received widespread attention in the field of intelligent health monitoring in recent years. Video-based remote photoplethysmography can extract physiological information from the temporal changes in facial regions caused by blood pulsation, providing a new technical approach to traditional contact-based measurement methods.

[0003] Most existing non-contact physiological parameter estimation methods use a single RGB video as input, estimating heart rate or blood oxygen saturation by modeling the spatiotemporal features of the facial region. However, in practical applications, RGB videos are easily affected by factors such as changes in ambient lighting, facial movement, local occlusion, and skin color differences, leading to unstable physiological signal extraction and affecting the accuracy and robustness of heart rate and blood oxygen saturation estimation. Especially in daily monitoring environments, the noise interference in the input video is more complex, making it difficult for single-modal methods to reliably obtain high-quality physiological features. Therefore, we propose a non-contact physiological parameter estimation method that can simultaneously utilize RGB and near-infrared video information. Through dual-path feature extraction, gating enhancement, multimodal feature fusion, and joint loss function optimization, we achieve stable joint estimation of heart rate and blood oxygen saturation, addressing a pressing technical problem in this field. Summary of the Invention

[0004] To address the technical problem that existing non-contact physiological parameter estimation methods are easily affected by changes in lighting, facial movements, and local occlusion in complex environments, resulting in insufficient accuracy and stability in heart rate and blood oxygen saturation estimation, this invention provides a non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion includes the following steps: S1: Acquire RGB and near-infrared videos containing the human face region, and simultaneously acquire corresponding heart rate and blood oxygen saturation reference signals. Preprocess the acquired data to construct a standardized dataset for model training and testing. S2: Construct a fusion dual-path physiological signal estimation network, which receives preprocessed RGB facial sequences and near-infrared facial sequences respectively, extracts spatiotemporal features from the two input data, and obtains the corresponding multimodal physiological signal feature representations; S3: A gated enhancement module is introduced into the fusion dual-path physiological signal estimation network to perform weighted modulation on the features extracted from the RGB branch and the near-infrared branch, highlighting the effective feature responses related to physiological signals and suppressing interference information; S4: Set up a multimodal feature fusion module to fuse the gated enhanced RGB features and near-infrared features to obtain a joint feature representation for physiological parameter estimation; S5: Based on the joint feature representation, construct a physiological parameter estimation branch, reconstruct the pulse wave signal, and combine it with the blood oxygen saturation estimation branch to achieve joint prediction of heart rate and blood oxygen saturation; S6: Design a joint loss function that includes a pulse wave signal constraint term and a blood oxygen saturation estimation error term, perform end-to-end training and optimization on the fusion dual-path physiological signal estimation network, and input the RGB video and near-infrared video to be tested into the trained model to output the corresponding heart rate and blood oxygen saturation estimation results.

[0006] This invention enhances the representation of facial temporal physiological information by simultaneously introducing RGB video and near-infrared video as input; improves the effective feature extraction capability of multimodal modes and suppresses noise interference by setting a fusion dual-channel coding structure and a gating enhancement module; realizes joint modeling of two modalities through a multimodal feature fusion module; and improves the stability of model training and the accuracy and robustness of heart rate and blood oxygen saturation estimation by using a joint loss function composed of pulse wave signal constraint term and blood oxygen saturation estimation error term. Attached Figure Description

[0007] Figure 1 This is an overall flowchart of the non-contact physiological parameter estimation method of the present invention; Figure 2 This is a schematic diagram of the fusion dual-path physiological signal estimation network and physiological parameter estimation branch structure of the present invention; Figure 3 This is a schematic diagram of the RGB branch encoder structure of the present invention; Figure 4 This is a schematic diagram of the near-infrared branch encoder structure of the present invention; Figure 5 This is a schematic diagram of the gate control enhancement module structure of the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0009] like Figure 1 As shown, a non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion includes the following steps: S1: Acquire RGB and near-infrared videos containing the human face region, and simultaneously acquire corresponding heart rate and blood oxygen saturation reference signals. Preprocess the acquired data to construct a standardized dataset for model training and testing. First, RGB and near-infrared videos containing the human facial region are acquired, along with corresponding heart rate and blood oxygen saturation reference signals, forming raw multimodal physiological parameter data. To ensure sample diversity and model generalization ability, data acquisition can be performed under different lighting conditions, background environments, and poses. After acquisition, face detection and facial region cropping are performed on the RGB and near-infrared videos to obtain RGB and near-infrared facial video sequences. Further, the facial video sequences are subjected to sequence segmentation, uniform size transformation, differential normalization, and standardization. The processed RGB and near-infrared facial video sequences, along with the corresponding heart rate and blood oxygen saturation reference signals, are then uniformly formatted, linked, and stored to construct a standardized dataset for model training and testing.

[0010] S2: Construct a fusion dual-path physiological signal estimation network, which receives preprocessed RGB facial sequences and near-infrared facial sequences respectively, extracts spatiotemporal features from the two input data, and obtains the corresponding multimodal physiological signal feature representations; The preprocessed RGB facial sequences and near-infrared facial sequences are respectively input into a fusion dual-path physiological signal estimation network for feature extraction. The fusion dual-path physiological signal estimation network is as follows: Figure 2 As shown, a dual-path parallel structure is used to independently encode the two modal inputs. Specifically, the network includes an RGB branch and a near-infrared branch, where the RGB branch is as follows: Figure 3The diagram shows an encoding structure consisting of ConvBlock (convolutional module), MaxPoolSpa (spatial pooling module), and MaxPoolSpaTem (spatiotemporal pooling module) to extract features layer by layer from RGB facial sequences. It progressively expands the receptive field through downsampling operations in both spatial and spatiotemporal dimensions, obtaining multi-level spatiotemporal feature representations of RGB facial sequences while preserving temporal variation information. The near-infrared branch, as shown... Figure 4 The diagram illustrates an encoding structure consisting of Conv3D (3D convolutional layers), ReLU activation function, SEBlock (attention module), and MaxPool3D (pooling layers) to extract features layer by layer from near-infrared facial sequences. This structure enhances the expressive power of near-infrared temporal features through repeated stacking, resulting in a spatiotemporal representation of the near-infrared facial sequences. After dual-path encoding, basic features corresponding to the RGB and near-infrared modalities are obtained, providing input for subsequent gating enhancement and multimodal feature fusion.

[0011] S3: A gated enhancement module is introduced into the fusion dual-path physiological signal estimation network to perform weighted modulation on the features extracted from the RGB branch and the near-infrared branch, highlighting the effective feature responses related to physiological signals and suppressing interference information; A gated enhancement module is set after the dual-channel encoding of the fusion dual-channel physiological signal estimation network to perform gated enhancement processing on the spatiotemporal features output from the RGB branch and the near-infrared branch, respectively. Specifically, each gated unit first performs pooling compression on the input features to aggregate global response information in the feature map; then, it combines Conv3D (three-dimensional convolutional layer) and ReLU activation function to generate corresponding feature weights; finally, it maps the feature weights to gate coefficients through the Sigmoid activation function, and then performs weighted modulation on the gate coefficients and the original input features to enhance the effective features related to physiological signal changes, while suppressing interference information introduced by illumination changes, facial movements, and local occlusion. After gated enhancement, the enhanced RGB features and enhanced near-infrared features are obtained, and then fed into the subsequent multimodal feature fusion module.

[0012] S4: Set up a multimodal feature fusion module to fuse the gated enhanced RGB features and near-infrared features to obtain a joint feature representation for physiological parameter estimation; The gated and enhanced RGB and near-infrared features are jointly input into the multimodal feature fusion module for joint modeling. This module employs a fusion structure composed of Conv3D (a 3D convolutional layer), BatchNorm3D (a normalization layer), and ReLU activation function. It first integrates the two input features, then further refines the fused features in the channel and spatiotemporal dimensions through convolutional operations, thereby enhancing the correlation expression between different modalities and obtaining a joint feature representation for subsequent physiological parameter estimation. This joint feature representation can be expressed as: in, This represents the joint feature representation after fusion. This indicates the enhanced RGB features after gating. This indicates the near-infrared characteristics after gating enhancement. This represents the mapping function of the multimodal feature fusion module.

[0013] The multimodal feature fusion module can simultaneously retain effective physiological information in both the RGB and near-infrared modes, providing a unified fusion representation for subsequent pulse wave signal reconstruction and blood oxygen saturation estimation.

[0014] S5: Construct a physiological parameter estimation branch based on the joint feature representation, reconstruct the pulse wave signal, and combine it with the blood oxygen saturation estimation branch to achieve joint prediction of heart rate and blood oxygen saturation; The joint feature representation output by the multimodal feature fusion module is input into the physiological parameter estimation branch. This physiological parameter estimation branch includes a pulse wave signal reconstruction branch and a blood oxygen saturation estimation branch. The pulse wave signal reconstruction branch models the periodic changes in the fused features using temporal convolution operations, outputting the corresponding pulse wave signal; the pulse wave signal can be represented as: in, This represents the reconstructed pulse wave signal. This represents the joint feature representation after fusion. The mapping function represents the reconstructed branch of the pulse wave signal.

[0015] After obtaining the pulse wave signal, the heart rate is estimated by extracting its temporal periodic features. The blood oxygen saturation estimation branch then performs a one-dimensional convolution mapping on the temporal signal output from the pulse wave signal reconstruction branch to obtain the blood oxygen saturation estimation result. Let the predicted blood oxygen saturation value be... Then it can be expressed as: in, This represents a one-dimensional convolution mapping operation. It represents the reconstructed pulse wave signal, and finally outputs the heart rate estimation result and blood oxygen saturation estimation result, realizing the joint prediction of non-contact physiological parameters.

[0016] S6: Design a joint loss function that includes a pulse wave signal constraint term and a blood oxygen saturation estimation error term, perform end-to-end training and optimization on the fusion dual-path physiological signal estimation network, and input the RGB video and near-infrared video to be tested into the trained model to output the corresponding heart rate and blood oxygen saturation estimation results.

[0017] To achieve end-to-end training of the fusion-based dual-path physiological signal estimation network, a joint loss function is constructed, comprising a pulse wave signal constraint term and a blood oxygen saturation estimation error term. Specifically, the error between the predicted pulse wave signal output from the pulse wave signal reconstruction branch and the corresponding real pulse wave reference signal is calculated to obtain the pulse wave signal constraint term; the error between the predicted blood oxygen saturation value output from the blood oxygen saturation estimation branch and the corresponding real blood oxygen saturation reference value is calculated to obtain the blood oxygen saturation estimation error term. The joint loss function is expressed as follows: in, Denotes the joint loss function. This represents the pulse wave signal constraint term. This represents the error term in blood oxygen saturation estimation. This represents the weighting coefficient of the blood oxygen saturation estimation error term. This represents the mean square error function. This represents the reconstructed pulse wave signal. This represents the true pulse wave reference signal. This represents the predicted blood oxygen saturation value. This represents the true blood oxygen saturation reference value. The entire network is optimized end-to-end based on the joint loss function, enabling the network to learn the temporal variation characteristics of the pulse wave signal while improving the accuracy and stability of blood oxygen saturation estimation. Preferably, the weight coefficients... It can be set according to the weight allocation of training tasks.

[0018] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion, characterized in that, Includes the following steps: S1: Acquire RGB and near-infrared videos containing the human face region, and simultaneously acquire corresponding heart rate and blood oxygen saturation reference signals. Preprocess the acquired data to construct a standardized dataset for model training and testing. S2: Construct a fusion dual-path physiological signal estimation network, which receives preprocessed RGB facial sequences and near-infrared facial sequences respectively, extracts spatiotemporal features from the two input data, and obtains the corresponding multimodal physiological signal feature representations; S3: A gated enhancement module is introduced into the fusion dual-path physiological signal estimation network to perform weighted modulation on the features extracted from the RGB branch and the near-infrared branch, highlighting the effective feature responses related to physiological signals and suppressing interference information; S4: Set up a multimodal feature fusion module to fuse the gated enhanced RGB features and near-infrared features to obtain a joint feature representation for physiological parameter estimation; S5: Based on the joint feature representation, construct a physiological parameter estimation branch, reconstruct the pulse wave signal, and combine it with the blood oxygen saturation estimation branch to achieve joint prediction of heart rate and blood oxygen saturation; S6: Design a joint loss function that includes a pulse wave signal constraint term and a blood oxygen saturation estimation error term, perform end-to-end training and optimization on the fusion dual-path physiological signal estimation network, and input the RGB video and near-infrared video to be tested into the trained model to output the corresponding heart rate and blood oxygen saturation estimation results.

2. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S1, RGB and near-infrared videos containing the human face region are acquired and a standardized dataset is established; RGB and near-infrared videos containing the human face region are acquired, and corresponding heart rate and blood oxygen saturation reference signals are acquired simultaneously to form raw multimodal physiological parameter data; face detection and facial region cropping are performed on the acquired RGB and near-infrared videos to obtain RGB and near-infrared facial video sequences; the RGB and near-infrared facial video sequences are subjected to sequence segmentation, uniform size transformation, differential normalization, and standardization to obtain preprocessed data suitable for network input. The preprocessed RGB facial video sequences, near-infrared facial video sequences, and corresponding heart rate and blood oxygen saturation reference signals are arranged and stored in a unified format to construct a standardized dataset for model training and testing.

3. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S2, a fusion dual-path physiological signal estimation network is constructed. The preprocessed RGB facial sequence and near-infrared facial sequence are respectively input into the RGB branch and near-infrared branch of the fusion dual-path physiological signal estimation network for feature extraction. The fusion dual-path physiological signal estimation network adopts a dual-path parallel structure to independently encode the two modal inputs. In the RGB branch, an encoding structure composed of ConvBlock (convolutional module), MaxPoolSpa spatial pooling module, and MaxPoolSpaTem spatiotemporal pooling module is used to extract features from the RGB facial sequence layer by layer. The spatiotemporal feature representation of the RGB facial sequence is obtained through downsampling operations in the spatial and spatiotemporal dimensions. In the near-infrared branch, an encoding structure consisting of Conv3D (three-dimensional convolutional layer), ReLU activation function, SEBlock attention module and MaxPool3D pooling layer is used to extract features layer by layer from the near-infrared facial sequence. The ability to express near-infrared temporal features is enhanced by repeated stacking, and the spatiotemporal feature representation of the near-infrared facial sequence is obtained. The spatiotemporal features output from the RGB branch and the near-infrared branch are used as inputs to the subsequent gating enhancement module and multimodal feature fusion module to form multimodal basic features for physiological parameter estimation.

4. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S3, a gated enhancement module is introduced into the fusion dual-path physiological signal estimation network. The spatiotemporal features output from the RGB branch and the near-infrared branch are respectively input into the corresponding gated units, and the two features are subjected to independent gated enhancement processing. The gated unit first compresses the input features using an AvgPool pooling layer to aggregate global response information in the feature map. Then, it generates corresponding feature weights by combining Conv3D and ReLU activation functions, and maps the feature weights to gate coefficients through a Sigmoid activation function. The gate coefficients are weighted and modulated with the input features of the corresponding modality to enhance the effective features related to physiological signal changes, while suppressing interference information introduced by illumination changes, facial movements, and local occlusion, thereby obtaining the gated enhanced RGB features and near-infrared features. The gated enhanced RGB features and near-infrared features are output to the subsequent multimodal feature fusion module as input features for fusion processing.

5. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S4, a multimodal feature fusion module is set up to fuse the gated enhanced RGB features and near-infrared features to obtain a joint feature representation for physiological parameter estimation. The gated enhanced RGB features and near-infrared features are then input into the multimodal feature fusion module for joint modeling. The multimodal feature fusion module employs a fusion structure composed of Conv3D, BatchNorm3D normalization layers, and ReLU activation functions to perform joint feature extraction on the two input features in both channel and spatiotemporal dimensions, thereby enhancing the correlation expression between different modalities. Let the gated enhanced RGB features be... The near-infrared feature after gating enhancement is Then the joint feature representation output by the multimodal feature fusion module can be expressed as: in, This represents the joint feature representation after fusion. This represents the RGB features after gating enhancement. This indicates the near-infrared signature after gating enhancement. This represents the mapping function for the multimodal feature fusion module.

6. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S5, a physiological parameter estimation branch is constructed based on the joint feature representation to reconstruct the pulse wave signal. The blood oxygen saturation estimation result is obtained by performing a one-dimensional convolution mapping on the pulse wave signal, achieving joint prediction of heart rate and blood oxygen saturation. The joint feature representation output by the multimodal feature fusion module is input into the pulse wave signal reconstruction branch. The periodic variation information in the fused features is modeled through temporal convolution operations, and the corresponding pulse wave signal is output. If the joint feature representation output by the multimodal feature fusion module is given, then the pulse wave signal output by the pulse wave signal reconstruction branch can be represented as follows: Then the pulse wave signal output by the pulse wave signal reconstruction branch can be expressed as: in, This represents the reconstructed pulse wave signal. The mapping function representing the reconstructed branch of the pulse wave signal. This represents the joint feature representation output by the multimodal feature fusion module; Based on the temporal periodic features of the pulse wave signal, the corresponding heart rate estimation result is extracted, and the heart rate estimation result can be expressed as: in, This represents the predicted heart rate value. This represents the reconstructed pulse wave signal. This represents the mapping function from the pulse wave signal to the heart rate; The pulse wave signal is input into the blood oxygen saturation estimation branch, and the pulse wave signal is mapped using Conv1D (one-dimensional convolution) to output the blood oxygen saturation estimation result, which can be expressed as: in, This represents the predicted blood oxygen saturation value. This represents the joint feature representation output by the multimodal feature fusion module. This represents the reconstructed pulse wave signal. The mapping function representing the blood oxygen saturation estimation branch; The heart rate estimation results and blood oxygen saturation estimation results are used as non-contact physiological parameter estimation outputs to achieve joint prediction of heart rate and blood oxygen saturation.

7. The non-contact physiological parameter estimation method based on RGB and near-infrared multimodal fusion according to claim 1, characterized in that, In step S6, a joint loss function comprising a pulse wave signal constraint term and a blood oxygen saturation estimation error term is designed to perform end-to-end training and optimization on the fusion dual-path physiological signal estimation network. The error between the predicted pulse wave signal output from the pulse wave signal reconstruction branch and the corresponding real pulse wave reference signal is calculated to construct a pulse wave signal constraint term, which is used to constrain the network to learn the real periodic physiological signal change patterns. Let the predicted pulse wave signal be... The true pulse wave reference signal is Then the pulse wave signal constraint term is expressed as: The error between the predicted blood oxygen saturation value output by the blood oxygen saturation estimation branch and the corresponding true blood oxygen saturation reference value is calculated to construct a blood oxygen saturation estimation error term, which is used to improve the accuracy of blood oxygen saturation prediction. Let the predicted blood oxygen saturation value be... The true reference value for blood oxygen saturation is Then the blood oxygen saturation estimation error term is expressed as: The pulse wave signal constraint term and the blood oxygen saturation estimation error term are weighted and combined to construct a joint loss function for network training. The joint loss function is expressed as follows: in, Denotes the joint loss function. This represents the pulse wave signal constraint term. This represents the error term in blood oxygen saturation estimation. This represents the weighting coefficient of the blood oxygen saturation estimation error term.

Images