Cross-feature interaction remote physiological signal measurement framework based on comparative learning

Abstract

The invention provides a cross-feature interaction remote physiological signal measurement framework based on comparative learning. The framework is composed of a learnable frequency enhancement module, a feature extraction and prediction module and a cross-feature interaction module. The method aims at solving the problem that an unsupervised remote physiological signal measuring method is limited by noise interference such as motion artifacts and illumination, and consequently performance is poor.

Description

TECHNICAL FIELD

[0001] The present application relates to innovations in contrastive learning, attention mechanisms, and non-invasive remote physiological signal measurement. BACKGROUND

[0002] The heart, as one of the most important organs of the human body, the monitoring of physiological parameters related to it (such as heart rate, blood pressure, blood oxygen saturation, etc.) has great significance for health management, disease prevention, etc. Traditional measurement methods are mainly contact PPG (photoplethysmography), such as finger clip oximeters and contact devices such as wristbands. However, these methods have the defects of inconvenience in certain scenarios, and may cause discomfort to the wearer. In this case, remote photoplethysmography (rPPG) analyzes the changes in light intensity caused by the periodic changes in blood volume on the surface of the skin to infer physiological signals, avoiding direct contact and having low deployment costs, and has become the mainstream direction of current research.

[0003] In the early years, researchers used blind source separation or traditional algorithms based on skin reflection models to directly extract signals from human faces, but the anti-interference ability was limited in real-world environments. With the development of machine learning, rPPG algorithms based on deep neural networks have become a research hotspot, and many excellent supervised algorithms have emerged, achieving good results. However, supervised learning usually relies on a large amount of labeled data sets for training, and has limitations in cases where data resources are scarce and labeling costs are high. Therefore, researchers have turned their attention to unsupervised learning.

[0004] Unsupervised learning is a learning mechanism derived from its proxy task rather than directly from labels, and it has gradually attracted more attention by breaking away from the dependence on a large amount of labeled data. However, the current unsupervised learning method has a significant gap in effectiveness compared to the mainstream supervised learning method, and still needs more in-depth research and innovation.

[0005] In addition, most current deep learning-based methods use frame difference features for input, trying to focus on local subtle color changes to extract features, but they have inherent defects. The difference operation will amplify both useful physiological changes and useless noise, and in the absence of noise filtering mechanisms, deep features can also be easily overwhelmed; in addition, the difference signal cannot cover the entire heartbeat cycle, resulting in incomplete signals and a lack of long-term correlation modeling. SUMMARY

[0006] The present application aims to solve the problem that the unsupervised remote psychological measurement method is limited by motion artifacts, illumination and other noise interference, resulting in poor performance, uses a time normalization module to replace the traditional frame difference signal for input, effectively suppresses global noise while establishing long-term correlation, uses a learnable frequency enhancement module to generate more accurate negative sample pairs, introduces a self-similar graph attention signal periodicity, performs cross-feature interaction learning in the attention mechanism, and proposes a cross-feature interaction remote psychological signal measurement framework based on contrast learning.

[0007] In a short time window (10s-20s), the drift caused by light changes, camera auto exposure and slow head movement is approximately linear over time, and can be excluded after detrending. Physiological signals are periodic, while detrended noise is random. After normalization, the relative amplitude of periodic signals will be amplified, and the relative amplitude of random noise will be suppressed. Therefore, the root mean square is calculated in the time dimension and standardized after detrending. The time normalization module based on the two steps is used to replace the currently widely used frame difference feature for input, which amplifies the useful signal and eliminates part of the noise, while solving the problem of small receptive field of the frame difference signal. At the same time, to solve the problem of single negative sample in current contrast learning, the present application uses a learnable frequency enhancement module to control the sampling rate more accurately without changing the visual appearance of the original image, improving the sample utilization rate. Through the pyramid multi-scale structure, the original video features are extracted and down-sampled to obtain features of multiple scales. Then, the frequency modulation block is used to realize nonlinear frequency transformation. The modulated multi-scale features are then up-sampled, spliced and reconstructed into a video. Thus, the predicted rPPG signal negative sample pair is obtained through the feature extraction module and the prediction module. At the same time, the original signal and the original signal mask signal after time normalization are also subjected to the feature extraction and prediction module to obtain the positive sample pair of the predicted rPPG signal, i.e. to constitute a contrast learning framework, and to calculate the contrast learning loss.

[0008] In addition, the present application designs another path based on the time domain self-similarity of the rPPG signal, which is only used during training, to supplement the fine-grained lack of the contrastive learning branch in the time dimension. The original signal and the original signal mask signal after time normalization processing are input into the cross-feature interaction module after passing through the feature extraction module to ensure further learning of multi-scale and long-distance features while improving the understanding of the relationship between different features. After the input time sequence features are converted into tokens and projected into QKV matrices, the long-distance context semantic association is enhanced through the calculation of multi-head dot product attention, and finally converted into a self-similarity graph, thereby capturing the self-similarity in the time dimension. In this process, the mask signal uses the Q matrix generated by the original signal, while preserving the integrity of the original signal QKV, to realize the directional alignment and information complementation of the damaged features and the original features, and complete the cross-feature interaction. Thus, the dependence relationship of the features in the time sequence is captured, and the time dimension robustness that is lacking in contrast learning only relying on sample comparison is made up. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The overall model framework diagram proposed by the present application is shown in the figure;

[0010] Figure 2 The structure diagram of the learnable frequency enhancement module is shown in the figure;

[0011] Figure 3 The structure diagram of the feature extraction and prediction module is shown in the figure;

[0012] Figure 4 The structure diagram of the time domain similarity module is shown in the figure; DETAILED DESCRIPTION

[0013] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in detail below in combination with the figures and specific examples.

[0014] As shown in the figure, the main structure of the network is divided into two parts: Figure 1

[0015] ​The first part involves performing face bounding boxes on the input video and dividing it into training and testing sets to obtain video images with reshaped bounding boxes. The input video is divided into segments of approximately 30 frames. A linear trend is fitted to each pixel (i, j) in the segment along the time dimension, and then this trend is subtracted from the original signal for detrending. The root mean square of the detrended signal is calculated along the time dimension and standardized. Finally, the segments are stitched together to complete the time normalization of the input signal. Features from the original video are extracted using one 3D convolutional layer and two 3D residual blocks, and then downsampled to obtain features at three scales. For each scale feature, the spatial dimension is compressed using 3D global average pooling, and the channel dimension is compressed using 1D convolution to obtain a coarse rPPG signal. The vector obtained by copying the random sampling frequency ratio representing the target frequency ratio T times is concatenated with the rPPG signal and then passed through a 1D residual block and a bidirectional LSTM to generate a modulation vector, which is then multiplied element-wise with the original scale features to modulate the frequency. Finally, the modulated multi-scale features are upsampled and stitched together, and the video is reconstructed using a 1×1×1 3D convolution to obtain the frequency-enhanced signal. After passing through four spatiotemporal convolutional modules, the signal is fed into the rPPG prediction module to obtain negative samples for contrastive learning. The original signal and the masked original signal are then time-normalized and directly passed through the feature extraction and rPPG prediction modules to obtain positive samples for contrastive learning. The contrastive learning loss for the positive and negative sample pairs is then calculated.

[0016] The second part involves inputting the time-normalized original signal and the masked original signal into four spatiotemporal convolutional blocks of the feature extraction module, then passing them through a global average pooling layer before feeding them into the cross-feature interaction module. The input sequence is tokenized to form tokens, projected into a QKV matrix, and multi-head dot product attention is calculated. Subsequently, another linear layer is used for projection to calculate the cosine similarity between each pair of tokens, generating a self-similarity map. The masked input signal does not generate a Q-matrix but uses the Q-matrix of the original input to achieve cross-feature interaction.

[0017] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only to describe an example more clearly. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0018] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-feature interactive remote physiological signal measurement framework based on contrastive learning, characterized in that, Includes the following steps: Step 1: Use a time normalization module to replace the traditional frame difference signal as input; Step 2: Establish a learnable frequency enhancement module; Step 3: Establish the feature extraction module; Step 4: Establish the rPPG signal prediction module; Step 5: Establish a cross-feature interaction module; Step 6: Establish a comparative learning framework.

2. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, In step 1, the time normalization module is designed with two core steps based on the Shafer reflection model: detrending and time dimension normalization. Detrending is achieved using least squares linear regression, as shown in the following formula: in This represents the slope of the (i, j)th pixel. This indicates the intercept of the pixel. This represents the detrended value. The trend is subtracted from the original signal, and the root mean square of the detrended signal is calculated over time and then normalized, as shown in the following formula: Where ε is a small constant.

3. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, Step 2 uses a learnable frequency enhancement module to generate negative samples for contrastive learning. A pyramid multi-scale structure is employed, with the original video as input and a randomly sampled frequency ratio representing the target frequency ratio. After feature extraction and frequency modulation, the frequency ratio consistency loss is calculated as follows: in This represents the ratio of the dominant frequencies of the negative sample signal to the positive sample signal. If its ratio to the target frequency r... i If they are the same, then frequency modulation is successful. Finally, feature reconstruction is performed to ensure that the original video x... a With reconstructing negative sample videos The color difference will not be too large; calculate the video reconstruction loss:

4. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, In step 3, the feature extraction module consists of four spatiotemporal convolutional modules cascaded together, and finally passes through a global average pooling layer to obtain preliminary spatiotemporal features.

5. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, In step 4, the rPPG prediction module is connected after the last spatiotemporal convolutional block of the feature extraction module to further process the preliminary spatiotemporal features for waveform construction.

6. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, The cross-feature interaction module in step 5 consists of two temporal similarity pyramids, one for inputting the original signal features and the other for inputting the masked signal features. Each temporal similarity pyramid is composed of multiple cascaded temporal similarity sub-modules, each corresponding to a time scale. When mapping the input temporal tokens to a QKV matrix to calculate dot product attention, the masked input branch does not generate a Q matrix but uses the Q matrix of the original input branch. Finally, the cosine similarity between tokens is calculated to convert it into a self-similarity map for subsequent contrastive learning, thus achieving cross-feature interaction. The MSE loss is calculated as follows:

7. The cross-feature interactive remote physiological signal measurement framework based on contrastive learning according to claim 1, characterized in that, Step 6: The time-normalized original signal and the original signal mask signal are processed through the feature extraction and prediction module to obtain positive sample pairs for the predicted rPPG signal. The Pearson loss between the positive sample pairs is: The original signal after time normalization is resampled by the learnable frequency enhancement module and then passed through the feature extraction and prediction module to generate the predicted rPPG signal, which constitutes the negative sample pair for contrastive learning. The contrastive learning loss is calculated using the positive and negative sample pairs to form the contrastive learning framework.

Images