Remote physiological signal detection method and system based on state space and double-path interaction

Abstract

The present application relates to the technical field of non-contact physiological signal detection, and discloses a remote physiological signal detection method and system based on state space and double-path interaction. The method comprises: obtaining a sample set; performing frame difference preprocessing on an original face video to extract time sequence features; modeling the extracted time sequence features in time sequence dependence through a state space model and a state space duality mechanism; enhancing the time sequence features in a frequency domain through a frequency domain feedforward network; inputting the time sequence features enhanced in the frequency domain into a self-attention path and a cross-attention path respectively, and then fusing the time sequence features processed in the two paths to take the fused data as a predicted rPPG signal; performing model training by minimizing the difference between the predicted rPPG signal and an actual rPPG signal; and inputting a face video of a person to be detected into the trained prediction model to output a corresponding predicted rPPG signal. The detection method has good accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of non-contact physiological signal detection technology, specifically a remote physiological signal detection method and system based on state space and dual-path interaction. Background Technology

[0002] Remote photoplethysmography (rPPG) is a technique that uses cameras to capture minute changes in facial blood flow to extract physiological indicators such as heart rate and blood oxygen saturation without contact. This technology requires no sensors and has been widely applied in scenarios such as psychological state monitoring, telemedicine, and anti-counterfeiting. In recent years, with the development of deep learning and computer vision, methods based on convolutional neural networks and attention mechanisms (such as PhysFormer) have significantly improved the accuracy of rPPG in static or slightly moving scenes. Furthermore, multispectral imaging (such as the GREEN method) effectively mitigates the impact of skin pigmentation differences and changes in ambient light on signal quality by selecting specific illumination bands and using spatiotemporal fusion techniques. An emerging research direction is a dual-path interaction architecture combining state-space models and attention mechanisms, achieving collaborative modeling of long- and short-term dependencies.

[0003] However, existing rPPG methods based on deep CNNs or self-attention Transformers still struggle to balance efficiency and robustness in practical applications: they rely on O(T) 2 Complex attention mechanisms or multi-layer convolutional networks result in a quadratic increase in the number of parameters and multiply-accumulate operations with the length of the video, making real-time monitoring impossible on resource-constrained embedded devices. Furthermore, most models are trained only within a fixed frame length, exhibiting poor adaptability to excessively short or long video segments, and their performance is easily limited by GPU memory and computational latency. In addition, these methods often focus on unidirectional enhancement in the time or frequency domains, lacking frame-wide pooling and dual-domain interactive designs, failing to simultaneously capture the long-term temporal dependence and key frequency components of physiological signals such as heart rate. Moreover, the limited information interaction capabilities of single-path or shallow fusion structures mean that signal loss and noise residue remain significant problems in real-world scenarios with large movements, complex lighting, and diverse skin tones. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a remote physiological signal detection method and system based on state space and dual-path interaction. The physiological signal detection method of this invention has good accuracy and robustness, while reducing computational complexity.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention discloses a remote physiological signal detection method based on state space and dual-path interaction, comprising:

[0007] S1. Obtain a sample set, which includes face videos and the corresponding real rPPG signals;

[0008] S2. Perform frame difference preprocessing on the original face video to extract temporal features;

[0009] S3. Through state-space model and state-space duality mechanism, the extracted temporal features are modeled for temporal dependence;

[0010] S4. Enhance the temporal features using a frequency domain feedforward network;

[0011] S5. Input the enhanced temporal features of the frequency domain features into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths, and use the fused data as the predicted rPPG signal;

[0012] S6. By minimizing the difference between the predicted rPPG signal and the real rPPG signal, the model is trained to obtain the trained prediction model;

[0013] S7. Input the face video of the person to be detected into the trained prediction model, and output the corresponding predicted rPPG signal according to the processing method of steps S2 to S5.

[0014] As a further improvement to the above scheme, the expression for frame differential preprocessing in step S2 is as follows:

[0015] X Stem =Stem(X) origin )+Stem(X origin +X diff )

[0016] In the formula, X origin Raw face video data; X diff The difference data between adjacent frames; X Stem This is the output feature map; Stem(·) represents the Frame Stem function.

[0017] As a further improvement to the above scheme, in step S3, the expression of the state-space model is as follows:

[0018] h t =A h t-1 +B x t

[0019]

[0020] In the formula, A, B, and C are learnable parameter matrices; h t and h t-1Let x be the hidden state vectors at time t and t-1, respectively; t The input feature at time t is X Stem ;y t Let T be the output feature vector at time t; T is the transpose sign.

[0021] As a further improvement to the above scheme, in step S3, the expression for the state-space duality mechanism SSD is as follows:

[0022]

[0023] In the formula, Q, K, and V are the query, key, and value matrices, respectively; L is the structured mask matrix; This is the symbol for element-wise multiplication.

[0024] As a further improvement to the above scheme, in step S4, the processing method of the frequency domain feedforward network is as follows:

[0025] First, perform a Fast Fourier Transform on the time-series features, as shown in the following expression:

[0026]

[0027] In the formula, f represents the time series feature; H(f) represents the time series feature in the frequency domain; h(t) represents the time series feature in the time domain; dt is the time integration operator; e is the natural constant; and j is the imaginary unit.

[0028] The enhanced signal in the frequency domain is then recovered by inverse Fourier transform, as shown in the following expression:

[0029]

[0030] As a further improvement to the above scheme, the expression for self-attention path processing in step S5 is as follows:

[0031]

[0032] The expression for cross-attention path processing is as follows:

[0033]

[0034] The concatenation expression of the two path features along the channel dimension is as follows:

[0035] X fusion =Concat(FSA,FCA)

[0036] In the formula, X SA The output features of the self-attention path; L S Q is the structured mask matrix for the self-attention path; S K S and VS These are the query, key, and value matrices corresponding to the self-attention path; X is the element-wise multiplication symbol; T is the transpose symbol; X is the element-wise multiplication symbol. CA The output features of the cross-attention path; L C The structured mask matrix for the cross-attention path; Q C K C and V C These are the query, key, and value matrices corresponding to the cross-attention path; X fusion is the fused feature of the output features of the self-attention path and the cross-attention path after concatenation in the channel dimension; FSA is the output feature of the self-attention path after enhancement by the frequency domain feedforward network; FCA is the output feature of the cross-attention path after enhancement by the frequency domain feedforward network; Concat(·) is the concatenation process.

[0037] As a further improvement to the above scheme, in step S6, the overall loss function expression during model training is as follows:

[0038] L overall =L Time +L Freq

[0039] In the formula, L overall For the overall loss, L Time Used to measure the degree of matching between the predicted rPPG signal and the actual rPPG signal in the time domain; L Freq Used to measure the similarity between an rPPG signal and a real rPPG signal in the frequency domain.

[0040] This invention also discloses a remote physiological signal detection system based on state space and dual-path interaction, applying the remote physiological signal detection method based on state space and dual-path interaction as described above; the detection system includes:

[0041] The frame difference preprocessing module is used to acquire a sample set, which includes face videos and the corresponding real rPPG signals; the frame difference preprocessing module is also used to perform frame difference preprocessing on the original face videos to extract temporal features.

[0042] The temporal dependency modeling module is used to perform temporal dependency modeling on the extracted temporal features through the state-space model and the state-space duality mechanism.

[0043] The frequency domain feature enhancement module is used to enhance the time-series features through a frequency domain feedforward network.

[0044] The prediction module is used to input the enhanced temporal features of the frequency domain features into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths, and use the fused data as the predicted rPPG signal; wherein, by minimizing the difference between the predicted rPPG signal and the real rPPG signal, the model is trained to obtain the trained prediction model.

[0045] The sampling module is used to acquire facial videos of the person to be detected and input them into the trained prediction model; the corresponding predicted rPPG signal is output in accordance with the processing method of the above module.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] 1. This invention extracts spatiotemporal features from raw facial videos through frame differencing preprocessing, and then uses a state-space model to model the temporal information. The state-space model can capture long-term dependencies and adapt to subtle and continuous dynamic changes in physiological signals. Simultaneously, a state-space duality mechanism is introduced to weight features from different time segments and spatial regions, enhancing the capture of key physiological signals. Furthermore, a frequency-domain feedforward network is used to perform a Fast Fourier Transform (FFT) on the temporally modeled signal to extract important components in the frequency domain, and then the enhanced signal is recovered through an Inverse Fourier Transform (IFFT). This dual-track processing method in both the time and frequency domains not only accurately models the temporal evolution of the signal globally but also highlights frequency components related to physiological indicators such as heart rate and respiration, thereby improving the accuracy and robustness of monitoring.

[0048] 2. The prediction model of this invention is based on the principles of cooperative state-space duality and two-path interaction. It is an end-to-end architecture that, through multi-timescale cooperative state-space duality module learning and two-path interaction, can effectively extract weak rPPG signals from videos containing significant spatiotemporal redundancy and understand the periodic patterns of their long-range context. Furthermore, it maintains stable performance for video inputs of arbitrary length. This detection method has broad application prospects in fields such as healthcare, affective computing, and fraud prevention.

[0049] 3. The dual-path interaction structure designed in this invention processes temporal features through a self-attention path and a cross-attention path, respectively, and then fuses the information obtained from the two paths along the channel dimension. The self-attention path focuses on capturing the internal relationships and local consistency of the input data, while the cross-attention path focuses on exploring complementary information and global dependencies between different features. This design allows the model to fully mine and integrate spatiotemporal information at different levels, avoiding information loss that may occur with single-path processing, and suppressing noise and interference, thereby greatly enhancing the model's sensitivity and stability to remote physiological signals in complex environments.

[0050] 4. The State-Space Model (SSM) used in this invention has a computational complexity of O(T), while the traditional attention-based Transformer has a complexity of O(HWT). 2 In this model, thanks to the linear complexity of the cooperative state-space duality mechanism and frame-wide average pooling, its computational cost increases linearly with video length. In terms of the number of parameters and the number of multiply-accumulate operations (MACs), this model (1.065M parameters, 12.943 GMACs) is significantly lower than other methods, such as DeepPhys with 7.504M parameters and 120.000 GMACs, enabling this invention to run efficiently even on resource-constrained devices. Attached Figure Description

[0051] Figure 1 This is a flowchart of the remote physiological signal detection method based on state space and dual-path interaction according to Embodiment 1 of the present invention.

[0052] Figure 2 This is a framework diagram of the remote physiological signal detection system based on state space and dual-path interaction according to Embodiment 2 of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] Please see Figure 1 This embodiment provides a remote physiological signal detection method based on state space and dual-path interaction. First, the model is trained by inputting the collected sample data during the training process. Then, the trained model is used to monitor remote physiological signals. During the monitoring process, the input is the face video of the person to be detected, and the output is the corresponding predicted rPPG signal.

[0056] The training process is as follows:

[0057] S1. Obtain a sample set, which includes face videos and the corresponding real rPPG signals. This step ensures data diversity, providing sufficient and realistic labeled data for subsequent model training.

[0058] S2. Perform frame differencing preprocessing on the original face video to extract temporal features. The expression for frame differencing preprocessing is as follows:

[0059] X Stem =Stem(X) origin )+Stem(X origin +X diff )

[0060] In the formula, X origin Raw face video data; X diff The difference data between adjacent frames; X Stem This is for outputting feature maps; Stem(·) represents the Frame Stem module function.

[0061] The Stem(·) function is described below:

[0062] First, calculate the difference between adjacent frames to highlight motion information:

[0063] ΔX t =X t+1 -X t ,t=1,…,D-1

[0064] Next, two "Stem" modules are used to extract static textures and motion features respectively:

[0065] Y raw =Stem 11 (X t )

[0066] Y diff =Stem 12 (concat(ΔX t ))

[0067] Finally, the output is obtained by fusing the learnable weights α and β at two levels and downsampling:

[0068] F1=αY raw +βY diff

[0069] Y out =αStem 21 (F1)+βStem 22 (Y diff ).

[0070] In the above formula, Let ΔX be the raw image data of frame t; t The difference between frame t and frame t+1 highlights the temporal variation; Stem 11 and Stem 12 All of them contain a feature extraction module that includes Conv-BN-ReLU-Pool (i.e., convolutional layer-batch normalization-ReLU-pooling); Y rawFor static texture feature output, Y diff The output is the differential motion feature, where α and β are the learnable weights for fusing the two feature paths; Stem 21 and Stem 22 This is the second-level down-collection and feature processing module; F1 is the intermediate feature after first-level fusion; Y out For the final output features, the data has been sampled down to H / 8×W / 8 and the number of channels is the set dimension.

[0071] S3. Through state-space model and state-space duality mechanism, the extracted temporal features are modeled for temporal dependencies.

[0072] In step S3, the expression for the state-space model is as follows:

[0073] h t =A h t-1 +B x t

[0074] y t =C T h t

[0075] In the formula, A, B, and C are learnable parameter matrices; h t and h t-1 Let x be the hidden state vectors at time t and t-1, respectively; t The input feature at time t is X Stem ;y t Let be the output feature vector at time t; This is the transpose symbol.

[0076] The expression for the State-Space Duality (SSD) mechanism is as follows:

[0077]

[0078] In the formula, Q, K, and V are the query, key, and value matrices, respectively; L is the structured mask matrix; This is the symbol for element-wise multiplication.

[0079] S4. Enhance the temporal features using a frequency domain feedforward network (FDF).

[0080] In step S4, the frequency domain feedforward network processes the data as follows:

[0081] First, perform a Fast Fourier Transform on the time-series features, as shown in the following expression:

[0082]

[0083] In the formula, f represents the time series feature; H(f) represents the time series feature in the frequency domain; h(t) represents the time series feature in the time domain; dt is the time integration operator; e is the natural constant; and j is the imaginary unit.

[0084] The enhanced signal in the frequency domain is then recovered by inverse Fourier transform, as shown in the following expression:

[0085]

[0086] S5. Input the enhanced temporal features from the frequency domain into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths. Use the fused data as the predicted rPPG signal.

[0087] In step S5, the expression for self-attention path processing is as follows:

[0088]

[0089] The expression for cross-attention path processing is as follows:

[0090]

[0091] The concatenation expression of the two path features along the channel dimension is as follows:

[0092] X fusion =Concat(FSA,FCA)

[0093] In the formula, X SA The output features of the self-attention path; L S Q is the structured mask matrix for the self-attention path; S K S and V S These represent the query, key, and value matrices corresponding to the self-attention path; ° represents the element-wise multiplication symbol; X CA The output features of the cross-attention path; L C The structured mask matrix for the cross-attention path; Q C L C and V C These are the query, key, and value matrices corresponding to the cross-attention path; X fusion is the fused feature of the output features of the self-attention path and the cross-attention path after concatenation in the channel dimension; FSA is the output feature of the self-attention path after enhancement by the frequency domain feedforward network; FCA is the output feature of the cross-attention path after enhancement by the frequency domain feedforward network; Concat(·) is the concatenation process.

[0094] S6. By minimizing the difference between the predicted rPPG signal and the real rPPG signal, the model is trained to obtain the trained prediction model.

[0095] In step S6, the overall loss function expression during model training is as follows:

[0096] L overall =L Time +L Freq

[0097] In the formula, L overall For the overall loss, L Time Used to measure the degree of matching between the predicted rPPG signal and the actual rPPG signal in the time domain; L Freq Used to measure the similarity between an rPPG signal and a real rPPG signal in the frequency domain.

[0098] The monitoring process is as follows:

[0099] S7. Input the face video of the person to be detected into the trained prediction model, and output the corresponding predicted rPPG signal according to the processing method of steps S2 to S5. Specifically, the face video of the person to be detected is subjected to frame difference preprocessing to extract spatiotemporal features; the temporal features are subjected to temporal dependency modeling and frequency domain feature enhancement; the temporal features are processed through dual paths to obtain fused data; and the fused data processed by the model is output as the final rPPG signal.

[0100] This invention constructs a complete end-to-end system from preprocessing, temporal modeling, frequency domain enhancement to final signal prediction. Each stage is collaboratively optimized during the overall training process, reducing redundancy and error accumulation in information transmission while ensuring high levels of real-time performance and resource efficiency, making deployment and practical application more convenient. Frame differencing is employed in the preprocessing stage to effectively suppress static background noise and highlight dynamic changes between frames. This method allows the subsequently extracted spatiotemporal features to more accurately reflect the subtle dynamics of human physiological activities, providing richer foundational information for the model to capture key physiological signals.

[0101] Unlike previous remote physiological signal detection methods that often employ network structures such as RNNs, LSTMs, or CNNs to model temporal dependencies, this invention introduces a state-space model, enabling the capture of long-term temporal dependencies with lower computational complexity. Furthermore, the introduction of the state-space duality mechanism allows for dynamic weighting of features across different time periods and spatial regions. This enables the model to not only adaptively adjust the importance of each feature but also effectively suppress noise interference in signal prediction, thereby significantly improving the accuracy and stability of detection.

[0102] When focusing on time-domain processing, important frequency information related to specific physiological parameters in physiological signals may be overlooked. This invention, after time-series modeling, transforms the signal to the frequency domain using Fast Fourier Transform (FFT), then utilizes a frequency-domain feedforward network to enhance key frequency components, and finally recovers the enhanced signal using Inverse Fourier Transform (IFFT). This strategy combining time and frequency domains can more comprehensively extract and enhance the features of physiological signals, especially in environments with significant interference, significantly improving signal robustness and recognition accuracy.

[0103] Traditionally, single-path attention mechanisms are commonly used in the feature fusion stage, which may struggle to balance local details with global information. The model in this invention employs two paths: self-attention and cross-attention, extracting information from the perspectives of internal data consistency and cross-feature complementarity, respectively. After processing by each path, the data is fused along the channel dimension, resulting in a richer feature representation. This multi-dimensional information fusion strategy enables the model to more comprehensively understand spatiotemporal dynamics, further improving the accuracy of remote physiological signal prediction.

[0104] Example 2

[0105] This embodiment provides a remote physiological signal detection system based on state space and dual-path interaction, applying the remote physiological signal detection method based on state space and dual-path interaction as described in Embodiment 1; the detection system includes: a frame difference preprocessing module, a temporal dependency modeling module, a frequency domain feature enhancement module, a prediction module, and a sampling module; the working process of the detection system is divided into a training process and an estimation process;

[0106] During the training process:

[0107] The frame difference preprocessing module is used to acquire a sample set, which includes face videos and the corresponding real rPPG signals; the frame difference preprocessing module is also used to perform frame difference preprocessing on the original face videos to extract temporal features.

[0108] The temporal dependency modeling module is used to perform temporal dependency modeling on the extracted temporal features through the state-space model and the state-space duality mechanism;

[0109] The frequency domain feature enhancement module is used to enhance the frequency domain features of time series features through a frequency domain feedforward network;

[0110] The prediction module is used to input the enhanced temporal features of the frequency domain features into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths, and use the fused data as the predicted rPPG signal.

[0111] Specifically, the model is trained by minimizing the difference between the predicted rPPG signal and the real rPPG signal to obtain the trained prediction model.

[0112] In the estimation process:

[0113] The sampling module is used to acquire the face video to be estimated and input it into the trained prediction model; the corresponding predicted rPPG signal is output in accordance with the processing method of the above modules.

[0114] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A remote physiological signal detection method based on state space and dual-path interaction, characterized in that, include: S1. Obtain a sample set, which includes face videos and the corresponding real rPPG signals; S2. Perform frame difference preprocessing on the original face video to extract temporal features; The expression for frame difference preprocessing is as follows: In the formula, This is raw facial video data; This refers to the difference data between adjacent frames; Output feature map; Represents the Frame Stem function; S3. Temporal dependency modeling is performed on the extracted temporal features using a state-space model and a state-space duality mechanism; the expression of the state-space model is as follows: In the formula, A , B and C The learnable parameter matrix; and Let be the hidden state vectors at time t and t-1, respectively; The input features at time t are... ; Let be the output feature vector at time t; Here is the transpose symbol; the expression for the State-Space Duality Mechanism (SSD) is as follows: In the formula, Q , K and V These are the query, key, and value matrices, respectively. L For structured mask matrix; The symbol for element-wise multiplication; S4. Enhance the time-series features using a frequency-domain feedforward network; the processing method of the frequency-domain feedforward network is as follows: First, perform a Fast Fourier Transform on the time-series features, as shown in the following expression: In the formula, f It is a time-series feature; This represents the temporal characteristics in the frequency domain. dt represents the temporal characteristics in the time domain; dt is the time integral operator. e It is a natural constant; j The imaginary unit; The enhanced signal in the frequency domain is then recovered by inverse Fourier transform, as shown in the following expression: S5. Input the enhanced temporal features from the frequency domain into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths. Use the fused data as the predicted rPPG signal; the expression for the self-attention path processing is as follows: The expression for cross-attention path processing is as follows: The concatenation expression of the two path features along the channel dimension is as follows: In the formula, The output features of the self-attention path; This is the structured mask matrix for the self-attention path; , and These are the query, key, and value matrices corresponding to the self-attention path; The symbol for element-wise multiplication; It is the transpose symbol; The output features of the cross-attention path; The structured mask matrix for cross-attention paths; , and These are the query, key, and value matrices corresponding to the cross-attention paths; This is the fused feature of the output features of the self-attention path and the cross-attention path concatenated along the channel dimension. The output features of the self-attention path are enhanced by the frequency domain feedforward network. The cross-attention path output features are enhanced by a frequency domain feedforward network. For splicing processing; S6. By minimizing the difference between the predicted rPPG signal and the real rPPG signal, the model is trained to obtain the trained prediction model; S7. Input the face video of the person to be detected into the trained prediction model, and output the corresponding predicted rPPG signal according to the processing method of steps S2 to S5.

2. The remote physiological signal detection method based on state space and dual-path interaction according to claim 1, characterized in that, In step S6, the overall loss function expression during model training is as follows: In the formula, For the overall loss, Used to measure the degree of matching between the predicted rPPG signal and the real rPPG signal in the time domain; Used to measure the similarity between an rPPG signal and a real rPPG signal in the frequency domain.

3. A remote physiological signal detection system based on state space and dual-path interaction, characterized in that, The method for detecting remote physiological signals based on state space and dual-path interaction as described in any one of claims 1 to 2 is applied; the detection system comprises: The frame difference preprocessing module is used to acquire a sample set, which includes face videos and the corresponding real rPPG signals; the frame difference preprocessing module is also used to perform frame difference preprocessing on the original face videos to extract temporal features. The temporal dependency modeling module is used to perform temporal dependency modeling on the extracted temporal features through the state-space model and the state-space duality mechanism. The frequency domain feature enhancement module is used to enhance the time-series features through a frequency domain feedforward network. The prediction module is used to input the enhanced temporal features of the frequency domain features into the self-attention path and the cross-attention path respectively, and then fuse the temporal features processed by the two paths, and use the fused data as the predicted rPPG signal; wherein, by minimizing the difference between the predicted rPPG signal and the real rPPG signal, the model is trained to obtain the trained prediction model. The sampling module is used to acquire facial videos of the person to be detected and input them into the trained prediction model; the corresponding predicted rPPG signal is output in accordance with the processing method of the above module.

Images