Remote physiological signal detection method and device, electronic equipment and storage medium

Abstract

The invention provides a remote physiological signal detection method and device, electronic equipment and a storage medium, and relates to the technical field of computers. Preprocessing the obtained original face video data to obtain a plurality of standardized video frames; inputting the standardized video frame into a pre-trained detection model, and performing feature extraction on the standardized video frame by using a video encoder to obtain an intermediate layer feature and an initial prediction signal; performing feature alignment on the interlayer features of different source domains by using an optimal transportation alignment module to obtain aligned features; performing fusion processing on the alignment features and the initial prediction signal by using a multi-modal fusion module to obtain a fusion feature token; and utilizing a predictor to predict the physiological signal based on the fused feature token. The optimal transport alignment module is introduced into the detection model to perform feature alignment on the interlayer features from different source domains, so that the generalization ability of the detection model during domain migration is enhanced, and the accuracy and robustness of physiological signal detection are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method, apparatus, electronic device, and storage medium for remote physiological signal detection. Background Technology

[0002] In the field of computer vision, vision-based remote photoplethysmography (rPPG) is an effective, non-contact technique for detecting physiological signals. This method estimates physiological parameters (such as heart rate and respiratory rate) by analyzing optical information in facial videos.

[0003] Early rPPG methods primarily relied on analyzing subtle changes in skin color, typically reflected mainly in the green channel within the RGB color space. These methods often employed color subspace transformations based on chromaticity and orthogonal projection planes of skin to enhance signal discriminability. However, these methods generally depended on manually selecting regions of interest and predefined filtering, making them ill-suited for complex lighting environments and susceptible to head movements, shadow variations, and skin color differences, resulting in insufficient robustness in signal extraction.

[0004] In recent years, the rise of deep learning technology has provided new directions for the development of rPPG research. In particular, end-to-end models can directly learn layered features from raw video frames and capture richer and more subtle physiological patterns. However, this also makes them more susceptible to performance degradation when domain shifts occur due to imaging conditions (such as illumination and motion artifacts). This makes it imperative for end-to-end methods to conduct specialized domain generalization (DG) research because: (1) they cannot benefit from the robustness of engineering methods such as STMap (spatiotemporal graph); (2) the features they learn are more prone to entanglement with domain-specific artifacts.

[0005] While domain generalization has been extensively studied in image classification and other computer vision tasks, its application in end-to-end rPPG learning remains largely unexplored. Existing rPPG generalization research either relies on data-level preprocessing (such as STMap construction) or focuses on architectural innovation in single-source settings. These methods fail to effectively address the generalization problem of a complete end-to-end rPPG pipeline that directly maps raw video to physiological signals. Their generalization ability is severely limited when faced with domain shifts (such as differences in illumination, camera sensor, skin color, or motion patterns), making it impossible to accurately estimate physiological signals. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a remote physiological signal detection method, device, electronic device and storage medium to solve the problem that the existing end-to-end rPPG model has a severely limited generalization ability when facing domain offset and cannot accurately estimate physiological signals.

[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a method for remote physiological signal detection, the method comprising: The acquired raw facial video data is preprocessed to obtain multiple standardized video frames; The standardized video frames are input into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor. The video encoder is used to extract features from the standardized video frames to obtain intermediate layer features and initial prediction signals; The optimal transport alignment module is used to align the intermediate layer features of different source domains to obtain aligned features; The alignment features and the initial prediction signal are fused using the multimodal fusion module to obtain a fused feature token; Physiological signals are predicted using the predictor based on the fused feature tokens.

[0008] In an optional implementation, the multimodal fusion module includes an initial fusion module and a large language model; the step of fusing the alignment features and the initial prediction signal using the multimodal fusion module to obtain a fused feature token includes: The initial fusion module is used to perform shallow feature fusion on the alignment features and the initial prediction signal to obtain a visual token and a signal token. The visual token and the signal token are subjected to deep semantic fusion using the large language model to obtain a fused feature token.

[0009] In an optional implementation, the step of using the optimal transport alignment module to align the intermediate layer features of different source domains to obtain aligned features includes: The intermediate layer features from different source domains are mapped to a unified feature representation space using the transport mapping function in the optimal transport alignment module, thereby obtaining aligned features.

[0010] In an optional implementation, the optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function maps source domain features to a unified feature representation space, and the dual potential function is used during model training to evaluate the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function. The training process of the detection model includes: For each iteration of training of the detection model, based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features, the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function are calculated. The parameters of the video encoder, multimodal fusion module, predictor, and transport mapping function in the detection model are updated by gradient descent according to the total loss function, and the parameters of the dual potential function are updated by gradient ascent according to the dual loss function; wherein the parameter updates of the dual potential function and the transport mapping function are performed alternately.

[0011] In an optional implementation, the step of calculating the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features includes: Calculate the prediction loss based on the physiological signal label and the predicted physiological signal value; The alignment loss is calculated based on the mapped features and the transport mapping potential function in the dual potential function. The total loss function is obtained based on the alignment loss and the prediction loss; The dual loss function is calculated based on the source domain features, the mapped features, the source domain potential function and the transport mapping potential function in the dual potential function.

[0012] In a second aspect, the present invention provides a remote physiological signal detection device, the device comprising: The preprocessing module is used to preprocess the acquired raw facial video data to obtain multiple standardized video frames; An input module is used to input the standardized video frames into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor; The detection module is used to extract features from the standardized video frames using the video encoder to obtain intermediate layer features and an initial prediction signal; to align intermediate layer features from different source domains using the optimal transport alignment module to obtain aligned features; to fuse the aligned features and the initial prediction signal using the multimodal fusion module to obtain a fused feature token; and to predict physiological signals based on the fused feature token using the predictor.

[0013] In an optional implementation, the multimodal fusion module includes an initial fusion module and a large language model; the detection module is specifically used to perform shallow feature fusion on the alignment features and the initial prediction signal using the initial fusion module to obtain a visual token and a signal token; and to perform deep semantic fusion on the visual token and the signal token using the large language model to obtain a fused feature token. The detection module is also specifically used to map the intermediate layer features of different source domains to a unified feature representation space using the transport mapping function in the optimal transport alignment module, thereby obtaining aligned features.

[0014] In an optional implementation, the optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function maps source domain features to a unified feature representation space, and the dual potential function is used during model training to evaluate the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function. The device further includes: The model training module is used for each iteration of training the detection model. Based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features, it calculates the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function. It then updates the parameters of the video encoder, multimodal fusion module, predictor, and transfer mapping function in the detection model using gradient descent based on the total loss function, and updates the parameters of the dual potential function using gradient ascent based on the dual loss function. The parameter updates of the dual potential function and the transfer mapping function are performed alternately.

[0015] Thirdly, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the remote physiological signal detection method as described in any of the foregoing embodiments.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the remote physiological signal detection method as described in any of the foregoing embodiments.

[0017] This invention provides a remote physiological signal detection method, apparatus, electronic device, and storage medium. The method includes: preprocessing acquired raw facial video data to obtain multiple standardized video frames; inputting the standardized video frames into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor; using the video encoder to extract features from the standardized video frames to obtain intermediate layer features and an initial prediction signal; using the optimal transport alignment module to align the intermediate layer features from different source domains to obtain aligned features; using the multimodal fusion module to fuse the aligned features and the initial prediction signal to obtain a fused feature token; and using the predictor to predict the physiological signal based on the fused feature token. By introducing an optimal transport alignment module into the detection model to align intermediate layer features from different source domains, the generalization ability of the detection model in the face of domain shift is enhanced, thereby improving the accuracy and robustness of physiological signal detection. To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This invention provides a schematic flowchart of a remote physiological signal detection method according to an embodiment of the present invention. Figure 2 A schematic diagram of one model structure for the detection model is shown; Figure 3 A schematic diagram of another model structure for the detection model is shown; Figure 4 This diagram illustrates another flowchart of the remote physiological signal detection method provided in an embodiment of the present invention. Figure 5 This diagram illustrates a functional block diagram of a remote physiological signal detection device provided in an embodiment of the present invention. Figure 6Another functional block diagram of the remote physiological signal detection device provided in an embodiment of the present invention is shown; Figure 7 A block diagram of an electronic device provided in an embodiment of the present invention is shown.

[0020] Icons: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication module; 600 - Remote physiological signal detection device; 610 - Preprocessing module; 620 - Input module; 630 - Detection module; 640 - Model training module. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] Current research on rPPG generalization mainly focuses on data-level preprocessing optimization (such as STMap construction) or architectural improvements under single data source conditions. These methods fail to address the generalization problem of a complete end-to-end pipeline from raw video to physiological signals and lack theoretical understanding of how to align or unify representations from different source domains. Furthermore, the application of large language models (LLMs) in biomedical signal processing is receiving increasing attention, particularly in tasks such as structured data interpretation, medical report generation, and cross-modal understanding. However, the integration of LLMs with remote physiological measurement techniques (rPPG) remains underexplored.

[0025] Based on this, the embodiments of the present invention mainly address the problem that existing end-to-end rPPG models have severely limited generalization ability when facing domain offsets (such as differences in illumination, camera sensor, skin color, or motion patterns). Specifically, this includes the following main problems: 1. Performance degradation due to domain offset: Existing end-to-end rPPG models experience significant performance degradation when applied to new domains with different lighting, camera sensors, skin colors, or motion patterns; 2. Domain generalization methods lacking theoretical guidance: Existing rPPG domain generalization methods usually lack a theoretical understanding of the principles, making it difficult to effectively align or unify the representations of different source domains, thus limiting their generalization ability across tasks and architectures.

[0026] 3. Insufficient end-to-end pipeline generalization: Existing methods have failed to effectively solve the problem of generalizing a complete end-to-end rPPG pipeline that directly maps raw video to physiological signals.

[0027] To address the aforementioned issues, embodiments of the present invention provide a remote physiological signal detection method, apparatus, electronic device, and storage medium. By introducing an optimal transport alignment module into the detection model to align intermediate layer features from different source domains, the invention enhances the detection model's generalization ability in the face of domain shifts, thereby improving the accuracy and robustness of physiological signal detection. Furthermore, theoretical derivation provides guiding principles for model design and further explores integration with large language models to improve the model's adaptability and stability in complex real-world environments.

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] Please refer to Figure 1Figure 1 is a schematic flowchart illustrating a remote physiological signal detection method provided in an embodiment of the present invention. It should be noted that the remote physiological signal detection method of the present invention is not limited to the specific order shown in Figure 1 and the following figures. It should be understood that in other embodiments, the order of some steps in the remote physiological signal detection method of the present invention can be interchanged according to actual needs, or some steps can be omitted or deleted. This remote physiological signal detection method can be applied to electronic devices such as laptops, tablets, PCs (Personal Computers), and servers. The specific flowchart shown in Figure 1 will be described in detail below.

[0030] Step S101: Preprocess the acquired raw facial video data to obtain multiple standardized video frames.

[0031] In this embodiment, preprocessing operations on the raw facial video data may include detecting and cropping the face region to focus on the region of interest; resampling the video frame rate (e.g., adjusting it to 30 frames per second); and adjusting the size of the video frames (e.g., adjusting it to 128×128 pixels) to accommodate the input requirements of the subsequent neural network. This results in multiple standardized video frames.

[0032] Step S102: Input the standardized video frames into the pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor.

[0033] In this embodiment, after multiple standardized video frames, these standardized video frames are input into a pre-trained detection model, which can be a deep learning model. Figure 2 The diagram shown is a schematic of one model structure of the detection model. The detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor. After the standardized video frame is input into the detection model, it will be processed sequentially by the video encoder, the optimal transport alignment module, the multimodal fusion module, and the predictor to finally obtain the predicted physiological signal.

[0034] Step S103: Use a video encoder to extract features from standardized video frames to obtain intermediate layer features and initial prediction signals.

[0035] In this embodiment, the video encoder can be understood as an end-to-end rPPG backbone network module, which is mainly used to extract features from standardized video frames. Typically, deep neural networks (e.g., PhysNet, PhysFormer, or RhythmFormer) directly learn complex spatiotemporal features from the video sequence.

[0036] In this embodiment, the intermediate layer features extracted by the video encoder are raw feature representations that may contain domain-specific biases. In addition to outputting the extracted intermediate layer features, the video encoder also uses its built-in predictor to perform coarse rPPG signal prediction based on these features, thereby outputting an initial prediction signal (i.e., a coarse rPPG signal). Therefore, the output of the video encoder includes two parts: intermediate layer features and the initial prediction signal. The intermediate layer features are input to the optimal transport alignment module for multi-scale feature alignment, while the initial prediction signal is input to the multimodal fusion module for corresponding processing.

[0037] Step S104: Use the optimal transport alignment module to align the intermediate layer features of different source domains to obtain aligned features.

[0038] In this embodiment, the optimal transport alignment module utilizes optimal transport (OT) theory to explicitly align feature distributions from multiple source domains, thereby obtaining aligned features with domain invariance. These features can better generalize to unseen domains, providing a more reliable foundation for subsequent physiological signal prediction. This eliminates the influence of environmental differences (such as lighting, camera type, skin color, etc.) on the features, enabling the model to more accurately identify real physiological signals.

[0039] Step S105: Use the multimodal fusion module to fuse the alignment features and the initial prediction signal to obtain the fused feature token.

[0040] In this embodiment, a token can be understood as a vector representation with specific semantics and structure. By fusing the aligned features and the initial prediction signal through a multimodal fusion module, the final prediction depends not only on the aligned features but also on the initial prediction signal, effectively mitigating prediction bias and improving prediction stability.

[0041] Step S106: Use the predictor to predict physiological signals based on fused feature tokens.

[0042] In this embodiment, the predictor maps the input fused feature tokens to specific physiological signals and outputs the predicted physiological signals, such as heart rate (HR) values ​​or blood volume pulse (BVP) waveforms.

[0043] As can be seen, the remote physiological signal detection method provided in this embodiment of the invention includes: preprocessing the acquired raw facial video data to obtain multiple standardized video frames; inputting the standardized video frames into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor; using the video encoder to extract features from the standardized video frames to obtain intermediate layer features and an initial prediction signal; using the optimal transport alignment module to align the intermediate layer features from different source domains to obtain aligned features; using the multimodal fusion module to fuse the aligned features and the initial prediction signal to obtain a fused feature token; and using the predictor to predict the physiological signal based on the fused feature token. By introducing the optimal transport alignment module into the detection model to align the intermediate layer features from different source domains, the generalization ability of the detection model in the face of domain shift is enhanced, thereby improving the accuracy and robustness of physiological signal detection. In one embodiment, step S104 specifically includes: using the transport mapping function in the optimal transport alignment module to map the intermediate layer features of different source domains to a unified feature representation space, thereby obtaining aligned features.

[0044] In this embodiment, the transfer mapping function T This can be represented as X→Z, through this transfer mapping function. T The intermediate layer features from different source domains (original feature space X) can be mapped to a unified feature representation space (i.e., shared latent space Z), so that the intermediate layer features from different source domains form a more consistent distribution in the shared latent space Z, resulting in aligned features.

[0045] In one implementation, to further enhance the generalization performance of the detection model, the aforementioned multimodal fusion module can be implemented based on a large language model. For example... Figure 3 As shown, the multimodal fusion module includes an initial fusion module and a large language model. Step S105 specifically includes: The initial fusion module is used to perform shallow feature fusion on the alignment features and the initial prediction signal to obtain visual tokens and signal tokens; the large language model is used to perform deep semantic fusion on the visual tokens and signal tokens to obtain fused feature tokens.

[0046] In this embodiment, the initial fusion module performs shallow feature fusion (i.e., preliminary attention fusion) on the alignment features and the initial prediction signal, ultimately obtaining visual tokens and signal tokens that the large language model can understand. These visual tokens and signal tokens are then input into the large language model for deep semantic fusion. Leveraging the semantic understanding and context modeling capabilities of the large language model, the final output is a fused feature token for the predictor to predict physiological signals. It is understood that in this embodiment, the output of the large language model is the fused feature token obtained from the final processing layer before deriving the inference result, and is not the physiological signal obtained based on the fused feature token.

[0047] In one implementation, the optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function maps source domain features to a unified feature representation space, and the dual potential function is used during model training to evaluate the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function. Please refer to [reference needed]. Figure 4 The training process of the detection model includes: Step S401: For each iteration of training of the detection model, based on the physiological signal labels corresponding to the training samples, the predicted values ​​of the physiological signals output by the detection model, the source domain features, and the mapped features, calculate the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function.

[0048] Specifically, step S401 includes: calculating the prediction loss based on the physiological signal label and the predicted physiological signal value; calculating the alignment loss based on the mapped features and the transfer mapping potential function in the dual potential function; obtaining the total loss function based on the alignment loss and the prediction loss; and calculating the dual loss function based on the source domain features, the mapped features, the source domain potential function in the dual potential function, and the transfer mapping potential function.

[0049] Step S402: The parameters of the video encoder, multimodal fusion module, predictor and transport mapping function in the detection model are updated by gradient descent according to the total loss function, and the parameters of the dual potential function are updated by gradient ascent according to the dual loss function; wherein, the parameter updates of the dual potential function and the transport mapping function are performed alternately.

[0050] In this embodiment, the optimal transport alignment module is a lightweight, modular OT alignment module named OTAdapter. This module can be seamlessly integrated into various end-to-end rPPG architectures.

[0051] Specifically, OTAdapter explicitly aligns feature distributions from multiple source domains by utilizing Wasserstein distance to measure distribution divergence, thereby reducing inter-domain differences in the feature space. This alignment mechanism not only preserves the flexibility of existing rPPG models but also introduces a principled approach to learning domain-invariant representations.

[0052] In OTAdapter, the dual potential function (including the source domain potential function) from the Kantorovich dual form is introduced. and transport mapping potential function Source domain potential function Acting on the original feature space (source domain), the transfer mapping potential function It operates on the feature space after the transport mapping. This is achieved by optimizing the dual loss function that contains these dual potential functions. This achieves the alignment of source domain features to the shared latent space. Specifically, the source domain potential function... and transport mapping potential function As a "discriminator," it guides the transfer mapping function by capturing the differences between source domain features and mapped features. T The learning process. The dual loss function can be expressed as: ; in, Represents source domain characteristics, This indicates the data distribution of the source domain. It is through the transfer mapping function T Transformed features (i.e., mapped features). It is the point-to-point transfer cost between source domain features and mapped features, where E represents the mathematical expectation. Represents the mapped features, This represents the data distribution within the target domain. Maximizing this dual loss function helps in finding the optimal dual potential function. and This allows for better quantification and minimization of the Wasserstein distance between source domain features and mapped features.

[0053] In this embodiment, a unified feature alignment framework is proposed to capture domain-invariant physiological features. This framework learns a transfer mapping function. Each source domain feature Mapping to a common potential space By minimizing transmission costs, OTAdapter encourages samples from all source domains to form a more consistent distribution in the shared latent space, thereby constructing a more generalizable shared representation. For example, if the input features are face video features from different lighting conditions (domain A and domain B), OTAdapter will learn a transmission mapping function. , so that after The transformed features from domain A and domain B are made as close as possible to each other in the shared latent space, thus reducing the impact of illumination differences. Finally, a predictor is used... Prediction is made based on the features mapped from these transports (mapped features).

[0054] In this embodiment, the total loss function of the detection model It can be represented as: ; in, Indicates alignment loss. Indicates the predicted loss. This represents the true value (physiological signal label). The alignment loss originates from optimal transport theory, which encourages the features after transport mapping to be closely aligned with the implicit centroid target distribution in the shared latent space; minimizing its negative value encourages samples from all source domains to be mapped to a more consistent distribution in the shared latent space, thereby constructing a shared representation that is more generalizable to unseen target domains.

[0055] In this embodiment, a min-max optimization algorithm is used to train a detection model that generalizes across domains. This algorithm updates the dual potential function ( and ) and optimized transport mapping function T The process alternates between time periods. Specifically: During the initialization phase, the video encoder, initial fusion module, large language model, predictor g, and transport mapping function are randomly initialized. T and the dual potential function ( and The parameters of ).

[0056] Iterative training phase: In each training iteration, mini-batch data is sampled from each source domain. If the previous iteration updated the transfer mapping function... T Then, the transfer mapping function is maintained in this iteration of training. T With the parameters unchanged, the dual potential function ( and Update the parameters of ); calculate the transfer mapping function. T The transformed features, i.e. the mapped features Then based on the dual loss function Dual potential function ( and The parameters of the function are updated using gradient ascent to maximize the dual loss function. Simultaneously, calculate the total loss function. According to the total loss function The parameters of the video encoder, initial fusion module, large language model, and predictor are updated using gradient descent to minimize the total loss function. Similarly, if the previous iteration of training updated the dual potential function ( and Then, the dual potential function is maintained in this iteration of training. and The parameters of the transfer mapping function remain unchanged. T The parameters are updated based on the calculated total loss function. The parameters of the video encoder, initial fusion module, large language model, predictor, and transport mapping function are updated using gradient descent to minimize the total loss function. .

[0057] The training method described above ensures that the detection model learns robust representations, thereby solving the generalization problem by aligning the source domain and optimizing physiological signal prediction.

[0058] It should be noted that the embodiments of the present invention are not limited to empirical design, but also derive a novel generalization upper bound based on OT differences. This upper bound theoretically correlates the divergence between source domains with the expected error on the unseen target domain. Specifically, it includes: Theorem 1 (Reducing Domain Gap Through Joint Hybrid Alignment): Under specific assumptions (such as bounded transmission distances between the target domain and all source domains, the joint source distribution after transmission being close to the reference distribution, and the reference distribution approximating the target domain), the transmission distances between the aligned joint source distribution and the aligned target distribution satisfy: ; in, , , It is a parameter that measures the similarity of distributions. It is a transfer mapping function T The Lipschitz constant, W c This represents the Wasserstein distance (i.e., the optimal transmission distance based on the cost function c). Indicates the transfer mapping function T The resulting source domain distribution (aligned distribution). This indicates the effect of the transfer mapping function. T Aligning the source domain, the domain difference is compressed to Significantly lower than the original gap That is, the embodiments of the present invention effectively reduce the domain gap.

[0059] Theorem 2 (Interpretable Generalization Boundary with Joint Hybrid Transmission): Assume the loss function yes Continuous, and transfer mapping function T yes If the generalization error is continuous, then the generalization error over the target domain is bounded: ; in, Representation Model h (This can be understood as the generalization error of the aforementioned predictor g) in the target domain; Representation Model h The expected loss of source domain data after transmission Represents the loss function of Constants. This provides a formal proof for the alignment strategy in the embodiments of the present invention and reveals that domain generalization is more challenging in time series regression tasks such as rPPG.

[0060] Theorem 3 (The joint hybrid generalization boundary is tighter): The joint hybrid generalization boundary proposed in this invention is tighter than the classical domain generalization boundary.

[0061] ; Representation Model h Expected loss in the source domain.

[0062] This demonstrates that optimal transmission alignment not only compresses the distribution distance but also improves the model's performance in the source domain. These theoretical guarantees ensure that the embodiments of the present invention can reduce the domain gap of rPPG, thereby achieving robust generalization to unseen physiological signal distributions.

[0063] As can be seen, this invention proposes an innovative framework in the field of remote photoplethysmography (rPPG)—a multi-source domain generalization framework based on optimal transmission (OT)—aiming to overcome the limitation of traditional methods experiencing significant performance degradation under complex and variable real-world imaging conditions. Compared with existing methods, this invention demonstrates significant advantages in model generalization ability, theoretical support, and technological integration. Traditional rPPG models, especially end-to-end models, suffer from severely limited generalization ability and cannot maintain high measurement accuracy when faced with domain offsets such as illumination, motion artifacts, skin color, and camera sensor variations. To address these issues, this invention has made optimizations and improvements in several aspects.

[0064] First, this invention introduces a plug-and-play optimal transport alignment module to address the neighborhood offset problem by explicitly aligning the feature distributions of multiple source domains. This module leverages the mathematical rigor and distribution alignment capabilities of the Wasserstein distance to encourage samples from different source domains to form a more consistent distribution in the latent space, thereby constructing a more generalizable shared physiological feature representation. This design not only retains the flexibility of existing rPPG models but also introduces a fundamental mechanism for neighborhood-invariant representation learning.

[0065] Secondly, this invention theoretically derives new generalization bounds, providing principled guidance for model design. These theories ensure that the proposed framework can effectively reduce the domain gap in rPPG. Through a series of assumptions and theorems (such as Theorem 1, Theorem 2, and Theorem 3), this invention proves that the transmission distance between aligned source domain features and target domain features can be effectively reduced, and that the proposed joint hybrid generalization bound is tighter than the classical domain generalization bound. This provides a solid theoretical foundation for domain generalization in time series regression tasks such as rPPG measurements.

[0066] Furthermore, this invention is the first to attempt to integrate a large language model with end-to-end rPPG technology, further enhancing the model's adaptability and stability in complex real-world environments. By combining visual-language alignment, temporal signal modeling, and the feature coordination capabilities of OTAdapter, a novel LLM-guided signal generation pipeline is constructed. This cross-modal and multimodal fusion approach not only enhances the model's semantic understanding capabilities but also provides new ideas for generative modeling and multimodal analysis in future health monitoring systems.

[0067] Experimental results demonstrate that the embodiments of this invention achieve superior performance in cross-domain tasks, exhibiting strong adaptability and robustness, especially in complex environments. Large-scale experiments on various datasets, including UBFC-rPPG, PURE, BUAA-MIHR, and MMPD, validated the effectiveness of the embodiments of this invention in cross-modal remote physiological signal sensing tasks. Even when faced with interference factors such as illumination variations and motion artifacts, it maintains high-precision rPPG signal extraction. Compared to traditional methods and existing domain generalization baselines, the embodiments of this invention demonstrate stronger generalization capabilities on different datasets, particularly exhibiting stability and accuracy on cross-domain data, further highlighting its advantages in real-world complex scenarios. Simultaneously, ablation studies have also confirmed the effectiveness of the various components of the OTAdapter (such as the dual potential function). and The study demonstrated the significant contribution of transmission alignment loss to overall performance and verified the universality and effectiveness of OTAdapter on various rPPG backbone models.

[0068] In summary, this invention significantly improves the accuracy and robustness of end-to-end remote physiological signal sensing by introducing innovative applications of optimal transmission theory, rigorous derivation of theoretical generalization limits, and pioneering integration of LLMs and rPPG, providing new ideas and solutions for the widespread application of rPPG technology in complex environments.

[0069] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of a remote physiological signal detection device is given below. Please refer to... Figure 5 This is a functional block diagram of a remote physiological signal detection device 600 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the remote physiological signal detection device 600 provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The remote physiological signal detection device 600 includes: a preprocessing module 610, an input module 620, and a detection module 630.

[0070] The preprocessing module 610 is used to preprocess the acquired raw facial video data to obtain multiple standardized video frames.

[0071] It is understandable that the preprocessing module 610 can perform the above step S101.

[0072] The input module 620 is used to input standardized video frames into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor.

[0073] It is understandable that the input module 620 can perform the above step S102.

[0074] The detection module 630 is used to extract features from standardized video frames using a video encoder to obtain intermediate layer features and an initial prediction signal; to align intermediate layer features from different source domains using an optimal transport alignment module to obtain aligned features; to fuse the aligned features and the initial prediction signal using a multimodal fusion module to obtain a fused feature token; and to predict physiological signals based on the fused feature token using a predictor.

[0075] It is understood that the detection module 630 can perform the above steps S103 to S106.

[0076] Optionally, the multimodal fusion module includes an initial fusion module and a large language model; the detection module 630 is specifically used to perform shallow feature fusion on the alignment features and the initial prediction signal using the initial fusion module to obtain a visual token and a signal token; and to perform deep semantic fusion on the visual token and the signal token using the large language model to obtain a fused feature token.

[0077] Optionally, the detection module 630 is further configured to use the transport mapping function in the optimal transport alignment module to map the intermediate layer features of different source domains to a unified feature representation space, thereby obtaining aligned features.

[0078] Optionally, the optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function maps source domain features to a unified feature representation space, while the dual potential function evaluates the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function during model training. Please refer to [reference needed]. Figure 6 The remote physiological signal detection device 600 provided in this embodiment of the invention may further include a model training module 640, which is used for each iteration of training of the detection model. Based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features, the module calculates the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function. The module updates the parameters of the video encoder, multimodal fusion module, predictor, and transfer mapping function in the detection model using gradient descent according to the total loss function, and updates the parameters of the dual potential function using gradient ascent according to the dual loss function. The parameter updates of the dual potential function and the transfer mapping function are performed alternately.

[0079] Optionally, the model training module 640 is specifically used to calculate the prediction loss based on the physiological signal label and the predicted value of the physiological signal; calculate the alignment loss based on the mapped features and the transfer mapping potential function in the dual potential function; obtain the total loss function based on the alignment loss and the prediction loss; and calculate the dual loss function based on the source domain features, the mapped features, the source domain potential function and the transfer mapping potential function in the dual potential function.

[0080] It is understood that the model training module 640 can execute the above steps S401 and S402.

[0081] Please refer to Figure 7 This is a block diagram of an electronic device 100 provided in an embodiment of the present invention. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0082] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0083] The processor 120 is used to read / write data or programs stored in the memory 110 and to perform corresponding functions. For example, when a computer program stored in the memory 110 is executed by the processor 120, the remote physiological signal detection method disclosed in the above embodiments can be implemented.

[0084] The communication module 130 is used to establish a communication connection between the electronic device 100 and other devices via a network, and to send and receive data via the network.

[0085] It should be understood that, Figure 7 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0086] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor 120, implements the remote physiological signal detection method disclosed in the above embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0088] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0089] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for remote physiological signal detection, characterized in that, The method includes: The acquired raw facial video data is preprocessed to obtain multiple standardized video frames; The standardized video frames are input into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor. The video encoder is used to extract features from the standardized video frames to obtain intermediate layer features and initial prediction signals; The optimal transport alignment module is used to align the intermediate layer features of different source domains to obtain aligned features; The alignment features and the initial prediction signal are fused using the multimodal fusion module to obtain a fused feature token; Physiological signals are predicted using the predictor based on the fused feature tokens.

2. The remote physiological signal detection method according to claim 1, characterized in that, The multimodal fusion module includes an initial fusion module and a large language model; the process of fusing the alignment features and the initial prediction signal using the multimodal fusion module to obtain a fused feature token includes: The initial fusion module is used to perform shallow feature fusion on the alignment features and the initial prediction signal to obtain a visual token and a signal token. The visual token and the signal token are subjected to deep semantic fusion using the large language model to obtain a fused feature token.

3. The remote physiological signal detection method according to claim 1, characterized in that, The step of using the optimal transport alignment module to align intermediate layer features from different source domains to obtain aligned features includes: The intermediate layer features from different source domains are mapped to a unified feature representation space using the transport mapping function in the optimal transport alignment module, thereby obtaining aligned features.

4. The remote physiological signal detection method according to any one of claims 1-3, characterized in that, The optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function is used to map source domain features to a unified feature representation space. The dual potential function is used to evaluate the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function during model training. The training process of the detection model includes: For each iteration of training of the detection model, based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features, the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function are calculated. The parameters of the video encoder, multimodal fusion module, predictor, and transport mapping function in the detection model are updated by gradient descent according to the total loss function, and the parameters of the dual potential function are updated by gradient ascent according to the dual loss function; wherein the parameter updates of the dual potential function and the transport mapping function are performed alternately.

5. The remote physiological signal detection method according to claim 4, characterized in that, The calculation of the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features includes: Calculate the prediction loss based on the physiological signal label and the predicted physiological signal value; The alignment loss is calculated based on the mapped features and the transport mapping potential function in the dual potential function. The total loss function is obtained based on the alignment loss and the prediction loss; The dual loss function is calculated based on the source domain features, the mapped features, the source domain potential function and the transport mapping potential function in the dual potential function.

6. A remote physiological signal detection device, characterized in that, The device includes: The preprocessing module is used to preprocess the acquired raw facial video data to obtain multiple standardized video frames; An input module is used to input the standardized video frames into a pre-trained detection model; the detection model includes a video encoder, an optimal transport alignment module, a multimodal fusion module, and a predictor; The detection module is used to extract features from the standardized video frames using the video encoder to obtain intermediate layer features and an initial prediction signal; to align intermediate layer features from different source domains using the optimal transport alignment module to obtain aligned features; to fuse the aligned features and the initial prediction signal using the multimodal fusion module to obtain a fused feature token; and to predict physiological signals based on the fused feature token using the predictor.

7. The remote physiological signal detection device according to claim 6, characterized in that, The multimodal fusion module includes an initial fusion module and a large language model; the detection module is specifically used to perform shallow feature fusion on the alignment features and the initial prediction signal using the initial fusion module to obtain a visual token and a signal token; and to perform deep semantic fusion on the visual token and the signal token using the large language model to obtain a fused feature token. The detection module is also specifically used to map the intermediate layer features of different source domains to a unified feature representation space using the transport mapping function in the optimal transport alignment module, thereby obtaining aligned features.

8. The remote physiological signal detection device according to claim 6 or 7, characterized in that, The optimal transport alignment module includes a dual potential function and a transport mapping function. The transport mapping function maps source domain features to a unified feature representation space. The dual potential function is used during model training to evaluate the difference between the source domain features input to the transport mapping function and the mapped features output by the transport mapping function. The device further includes: The model training module is used for each iteration of training the detection model. Based on the physiological signal labels corresponding to the training samples, the predicted physiological signal values ​​output by the detection model, the source domain features, and the mapped features, it calculates the total loss function corresponding to the detection model and the dual loss function corresponding to the dual potential function. It then updates the parameters of the video encoder, multimodal fusion module, predictor, and transfer mapping function in the detection model using gradient descent based on the total loss function, and updates the parameters of the dual potential function using gradient ascent based on the dual loss function. The parameter updates of the dual potential function and the transfer mapping function are performed alternately.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the remote physiological signal detection method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the remote physiological signal detection method as described in any one of claims 1-5.

Images