Multi-part rPPG fusion physiological signal monitoring method and device and storage medium

Abstract

The invention discloses a multi-part rPPG fusion physiological signal monitoring method and device and a storage medium. The method comprises the following steps: acquiring a multi-part skin video of a patient and preprocessing; editing physiological part prompt information; multi-part skin videos and physiological part prompt information are input into the model, the model can accurately extract spatial features in the skin videos of different parts under the guidance of a dynamic convolution mechanism, an asynchronous distillation technology and the physiological part prompt information, the rPPG pulse waveform features of all the parts are obtained through time sequence analysis and fusion, and then the rPPG pulse waveform features of all the parts are obtained through weighted fusion. A comprehensive rPPG waveform signal is generated; and finally, post-processing and calculating the comprehensive rPPG waveform signal to obtain a plurality of physiological signal values, and outputting a comprehensive report. The physiological state of the patient is comprehensively monitored through extraction and fusion of the rPPG signals of multiple parts such as the four limbs, and the problem that the rPPG signals cannot be extracted due to the fact that the face and the upper body are shielded is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer deep learning technology, and specifically relates to a multi-site rPPG fusion physiological signal monitoring method, device and storage medium. Background Art

[0002] Remote photoplethysmography (rPPG) can accurately capture changes in blood volume by monitoring tiny fluctuations in light reflected from the skin, thereby providing physiological indicators such as heart rate, respiratory rate, and blood oxygen saturation. However, the traditional method usually extracts rPPG signals from the patient's face, which is easily affected by the patient's facial expressions, head movements, and lighting conditions, and therefore has limitations. Existing rPPG technology has the following defects: 1. Dependence on facial signals: When a patient enters a medical device such as a CT scan, the patient's head and upper body are covered. Since traditional rPPG technology relies on facial signals, it cannot work properly in such scenarios.

[0003] 2. Impact of light changes: The lighting environment in medical equipment is complex. Traditional rPPG technology is prone to noise under strong or weak light conditions, which affects the accuracy of the rPPG signal.

[0004] 3. Insufficient single-site monitoring: Traditional rPPG technology relies on the extraction of rPPG signals from a single site (such as the face) and cannot obtain physiological signals from multiple sites at the same time, which also limits the comprehensive monitoring of the overall physiological state. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a multi-site rPPG fusion physiological signal monitoring method, device and storage medium.

[0006] In order to solve the above technical problems and achieve the above technical effects, the present invention is implemented through the following technical solutions: A multi-site rPPG fusion physiological signal monitoring method, comprising: Step 1) Collect skin videos of multiple parts of the current patient; Step 2) preprocessing the collected skin videos of multiple parts to eliminate environmental noise and interference; Step 3) editing the physiological region prompt information (BioRegion Prompt) for indicating the source region of the collected multi-region skin video signals; Step 4) The pre-processed multi-site skin videos and the edited physiological site prompt information are used as input data and simultaneously sent to a multi-site rPPG fusion physiological signal extraction model trained based on an improved hyper-convolutional neural network and using asynchronous distillation technology for processing; The multi-site rPPG fusion physiological signal extraction model adjusts the attention weights of skin videos of different sites through the input physiological site prompt information, dynamically optimizes the feature extraction capability of specific areas, extracts the rPPG pulse waveform features of the skin video space of each site, and combines time series analysis to ensure the spatiotemporal integrity and stability of the rPPG pulse waveform features, and finally performs weighted fusion on the rPPG pulse waveform features extracted from each site to generate a comprehensive rPPG waveform signal; Step 5) The extracted comprehensive rPPG pulse waveform signal is post-processed, and then the physiological signal values ​​including heart rate and respiratory rate are calculated through frequency domain analysis methods (such as fast Fourier transform FFT), and a comprehensive report containing these physiological data is output.

[0007] Furthermore, the multi-site skin video is a skin video obtained by collecting data from multiple different parts of the patient (such as limbs or other skin areas) using a camera device including a near-infrared camera or a high-frame rate RGB camera. The multi-site skin video can reflect subtle changes in light reflected from the skin of multiple different parts of the patient so as to capture blood flow information.

[0008] Furthermore, the preprocessing focuses on the segmentation and enhancement of skin areas in multi-site skin videos to ensure the stability and accuracy of subsequent rPPG pulse waveform feature extraction.

[0009] The skin region segmentation process is to extract a mask including the face and limbs.

[0010] Skin area enhancement processing is to improve the signal quality of multiple skin parts videos by filtering and removing artifacts through the use of image processing technology; including using a bandpass filter to remove high-frequency noise and low-frequency noise and retain the frequency range related to the pulse; or using smoothing filtering and artifact removal technology to eliminate motion artifacts to improve signal reliability.

[0011] Furthermore, the physiological part prompt information is a physiological part code edited by using one-hot encoding to edit the physiological part prompt word input by the user, or is a vector corresponding to each of the multiple parts visual features converted by the CLIP model into the physiological part prompt word input by the user; The physiological part prompt words are prompt words with part keywords (such as face, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.) manually input by the user according to the characteristics of different body parts in the multi-part skin video collected; Each of the physiological part codes or the vectors corresponds to a separate or combined physiological part, and is used to indicate the source parts of the multi-part skin video signals and to guide the attention weighting of these parts.

[0012] Furthermore, the architecture of the multi-site rPPG fusion physiological signal extraction model includes an input layer, a feature extraction module, a physiological site prompt module, a feature weighting module and a dense layer.

[0013] The input layer is responsible for receiving the collected skin videos of multiple parts and performing preliminary standardization processing. The skin videos of multiple parts are skin videos obtained by shooting multiple different parts of the patient using a camera device; it is also responsible for receiving the physiological part prompt words input by the user.

[0014] The feature extraction module consists of multiple convolutional layers, super-convolution modules, time translation modules and dynamic attention layers. It is responsible for extracting the rPPG pulse waveform features of each part from the skin videos of multiple parts, and obtaining a comprehensive rPPG waveform signal after fusion and weighting for subsequent physiological signal analysis.

[0015] The multi-layer convolution layer (Conv Layer) is the core of the model, responsible for extracting spatial features from multi-part skin videos, and the spatial features reflect the local changes of light reflected by the skin; each convolution layer scans the multi-part skin videos through multiple convolution kernels to extract low-level features including edges and textures; through the superposition of multiple layers of convolution, the model can gradually capture high-level features including characteristic changes in pulse waveforms in multi-part skin videos; The hyper-convolution module is responsible for dynamically adjusting the size and receptive field of the convolution kernel according to the body part sources of the multi-part skin videos, so as to improve the model's ability to capture the details of the multi-part skin videos during the feature extraction process, and ensure that the model can efficiently extract the unique signal features of each part of the skin video.

[0016] The time shift module (TSM) is responsible for injecting the time series of multi-part skin videos into the multi-layer convolutional layers, so that the model can capture the time features in the multi-part skin videos, and is used to capture the dynamic fluctuations of the multi-part skin videos, and realize the time series analysis of the spatial features, so as to obtain stable and accurate spatiotemporal features, namely, the rPPG pulse waveform features; the introduction of the time series ensures that the model has better processing capabilities for the time-dependent characteristics of the multi-part skin videos, so that the model can better capture the changing patterns of the pulse waveform at different time points.

[0017] The dynamic attention layer is responsible for automatically adjusting the model's attention to different areas in the multi-part skin video according to the characteristics of the multi-part skin video, thereby enhancing the model's attention to high-quality signal areas and reducing the weights of areas with larger noise.

[0018] The BioRegion Prompt Module is responsible for converting the physiological region prompt words (such as left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.) input by the user into vectors matching the visual features of multiple parts through the CLIP model, or editing the physiological region prompt words input by the user into physiological region codes by using one-hot encoding, and passing the vectors or the physiological region codes as input to the super-convolution module, so as to provide a basis for the super-convolution module to identify the body region sources of the input multi-region skin videos, and help the super-convolution module dynamically adjust the convolution kernel size and receptive field, thereby enhancing the model's attention to specific body parts.

[0019] The feature weighting module is responsible for fusing and weighting the rPPG pulse waveform features of various parts extracted by the feature extraction module according to the weighting mechanism through the weighted gate layer to ensure that the model can effectively fuse the rPPG pulse waveform features of multiple parts and adjust the weighted proportion according to the quality of the rPPG pulse waveform features.

[0020] The weighting mechanism weights the rPPG pulse waveform features of each part according to the quality and time series of the rPPG pulse waveform features of each part, that is, rPPG pulse waveform features that are relatively stable and have higher signal quality will occupy a larger proportion in the model output, while rPPG pulse waveform features that are noisy or unstable will occupy a smaller proportion in the model output.

[0021] The dense layer is responsible for weighting and nonlinearly mapping the feature vector output by the feature weighting module, integrating the multi-dimensional features into a compact representation, namely, a comprehensive rPPG waveform signal.

[0022] Furthermore, the training method of the multi-site rPPG fusion physiological signal extraction model is: 1) Collect a certain number of skin videos of multiple parts of patients in clinical environments, and synchronously obtain the corresponding PPG pulse waveform signal true value and physiological parameter true value including heart rate, blood oxygen, and respiratory rate, to form an rPPG dataset that contains time-space aligned skin videos of multiple parts, PPG pulse waveform signal true value and physiological parameter true value; 2) Using the constructed rPPG signal dataset, the asynchronous distillation technology is adopted in the distillation network to train the multi-part rPPG fusion physiological signal extraction model. The training process of the model adopts a staged guidance method, and the goal is to enhance the model's ability to extract multi-part skin video features. During training, multi-part skin videos are used as input, the true value of the PPG pulse waveform signal is used as the primary label, and the true value of the physiological parameter is used as the supplementary label. The model learns the mapping relationship between the video frames of the multi-part skin video and the corresponding PPG pulse waveform true value features, so that the model can accurately extract the rPPG pulse waveform features of each part according to the input multi-part rPPG signal, and output the comprehensive rPPG waveform signal after fusion weighting; 3) Use a specific loss function to optimize the model parameters.

[0023] Furthermore, the asynchronous distillation technology dynamically adjusts the learning weights of the signals of each part by decoupling the optimization process of information from different parts, so that the signal features of each part can be efficiently extracted; the importance of the signals of each part is considered to be uniform and equal, and the model adjusts the optimization process according to the part prompted by the physiological part prompt information to ensure that the signal features of all parts are fully learned, while providing support for the independent prediction and fusion of multi-part signals; facial signals provide more guidance in model training due to their strong reflective characteristics and large amount of data, but since facial signals are blocked in scenes such as CT equipment, the asynchronous distillation technology enhances the learning and optimization of signals of parts other than the face, so that in the absence of facial signals, the model can still independently and accurately predict the physiological signal features of parts other than the face; in the process of multi-part signal fusion, the model dynamically adjusts the weighting method of the signals of each part to ensure that the physiological signal features output during the fusion process are consistent or synchronized with the facial signals in terms of timing and amplitude; the features of the "non-target class" parts are asynchronously optimized with the weights of another cycle, so that the student model can maintain a certain understanding of the features of the non-target parts without interfering with the main target features, thereby achieving efficient separation and optimization of multi-part information.

[0024] Furthermore, the loss function includes mean absolute error (MAE), root mean square error (RMSE) and signal-to-noise ratio (SNR); wherein, The mean absolute error is used to evaluate the mean absolute difference between the predicted value and the true value of the rPPG signal; The root mean square error is used to evaluate the root mean square of the square error between the predicted value and the true value of the rPPG signal; The signal-to-noise ratio is used to evaluate the ratio of signal intensity to noise intensity in the rPPG signal; In a CT equipment environment, the mean absolute error, the root mean square error and the signal-to-noise ratio are used to calculate the error for signals of specific parts other than the face, and finally the total error is weighted to improve the generalization ability of the overall model and the ability to adapt to complex environments.

[0025] Furthermore, an adaptive algorithm based on illumination changes is used to post-process the integrated rPPG pulse waveform signal to correct the signal distortion caused by changes in light intensity; specifically, according to the extracted brightness value, cross-attention is performed on the real and imaginary parts respectively, and finally an inverse fast Fourier transform is performed to convert it into a brightness-adapted rPPG waveform signal output.

[0026] A computer device comprises: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform operations corresponding to the above-mentioned multi-site rPPG fusion physiological signal monitoring method.

[0027] A computer-readable storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned multi-site rPPG fusion physiological signal monitoring method.

[0028] The beneficial effects of the present invention are: The present invention proposes a physiological signal monitoring technology of multi-site rPPG signal fusion. The technology is based on remote photoplethysmography (rPPG) signal detection, and aims to achieve comprehensive physiological monitoring through the collection and fusion of rPPG signals from multiple sites such as limbs. It solves the problem that rPPG signal extraction cannot be performed due to occlusion of the face and upper body, and supplements real-time multi-site physiological signal monitoring.

[0029] The present invention adopts a multi-site rPPG fusion method. When the patient's face or upper body is partially covered (such as when the patient is in medical equipment such as CT), rPPG signals are extracted from exposed limbs (such as left and right arms and left and right legs, etc.), and the rPPG signals of multiple parts are fused to obtain comprehensive physiological signals (such as heart rate, respiratory rate) data, ensuring high-precision physiological monitoring when the patient's head or upper body is covered.

[0030] The present invention combines hyperconvolution technology with asynchronous distillation technology, and combines it with embedded physiological site prompt words, so that the model has the ability to dynamically adapt to rPPG signals of limbs, thereby realizing feature extraction of rPPG signals of multiple parts, enhancing the robustness of the model to complex signal environments, and effectively reducing computational costs.

[0031] The asynchronous distillation technology of the present invention enables the model to effectively distinguish the feature learning of the target area and the non-target area, ensuring that high-quality physiological signal output can still be obtained in a complex signal environment, enhancing the anti-interference ability of the model, ensuring the stable output of physiological signals, and being suitable for real-time physiological monitoring. The asynchronous distillation technology transfers features between the teacher model and the student model in different time periods to achieve more stable and accurate model training, further optimizing the learning effect of the model and reducing the computational complexity of the model.

[0032] The experimental results show that the present invention can provide accurate physiological signal monitoring in multi-site rPPG signal detection. This technology is expected to provide important support for the widespread clinical application of physiological signal monitoring, while improving the accuracy and efficiency of real-time monitoring.

[0033] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a flowchart of the steps of the multi-site rPPG fusion physiological signal monitoring method of the present invention; Figure 2 Schematic diagram of rPPG pulse waveform feature collection and fusion of the multi-site rPPG fusion physiological signal monitoring method of the present invention; Figure 3 It is a training flow chart of the multi-site rPPG fusion physiological signal extraction model of the present invention under the distillation network framework; Figure 4 It is a schematic diagram of the structure of the hyperconvolution module in the multi-site rPPG fusion physiological signal extraction model of the present invention; Figure 5 It is a schematic diagram of the process of embedding physiological part prompt words into a super-convolution module in the multi-part rPPG fusion physiological signal extraction model of the present invention; Figure 6 A sample diagram of a data set collected in an embodiment of the present invention; Figure 7 It is a schematic diagram of the technical route of the multi-site rPPG fusion physiological signal extraction model of the present invention; Figure 8 Schematic diagram of multi-site signal extraction weights in the distillation network framework of the present invention. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0036] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0037] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0038] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0039] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0040] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] In order to address the shortcomings of traditional physiological signal monitoring when the face and upper body are partially covered, as well as the accuracy and robustness of signal acquisition in complex environments, this paper proposes a comprehensive physiological monitoring method based on deep learning for multi-site rPPG signal fusion. When the patient's head is covered or occluded, this method extracts rPPG signals from the patient's limbs (such as arms and legs), and combines dynamic convolution and asynchronous knowledge distillation technology to ensure that high-precision and high-robust physiological signal monitoring can still be achieved under complex occlusion conditions, avoiding the limitation that facial rPPG signals cannot be obtained.

[0042] See also Figure 1 As shown, a multi-site rPPG fusion physiological signal monitoring method specifically includes the following: Step 1) Collect skin videos of multiple parts of the current patient; Step 2) preprocessing the collected skin videos of multiple parts to eliminate environmental noise and interference; Step 3) editing the physiological region prompt information (BioRegion Prompt) for indicating the source region of the collected multi-region skin video signals; Step 4) The pre-processed multi-site skin videos and the edited physiological site prompt information are used as input data and sent to the multi-site rPPG fusion physiological signal extraction model (BRPDNet) based on the improved hyper-convolutional neural network and trained with asynchronous distillation technology for processing; The BRPDNet model adjusts the attention weights of skin videos of different parts through the input physiological part prompt information, dynamically optimizes the feature extraction capabilities of specific areas, extracts the rPPG pulse waveform features of the skin video space of each part, and combines time series analysis to ensure the spatiotemporal integrity and stability of the rPPG pulse waveform features. Finally, the rPPG pulse waveform features extracted from each part are weighted and fused to generate a comprehensive rPPG waveform signal. Step 5) The extracted comprehensive rPPG pulse waveform signal is post-processed, and the physiological signal values ​​including heart rate and respiratory rate are calculated through frequency domain analysis methods (such as fast Fourier transform FFT), and a comprehensive report containing these physiological data is output. These physiological signal data can be fed back to the medical equipment operator in real time for clinical monitoring and decision support.

[0043] See also Figure 2 As shown in the figure, the method of the present invention can still provide high-precision multi-site rPPG signal detection in the scenario where the patient enters medical equipment such as CT and the head or upper body skin is covered, which can greatly enhance the real-time and reliability of physiological signal monitoring. This technology will provide strong technical support for a wide range of medical applications such as clinical diagnosis, surgical monitoring and long-term health management.

[0044] As a preferred embodiment of the present invention, in step 1), the method for collecting rPPG signals at multiple locations is as follows: When a patient enters a large medical device such as a CT scanner or the like, the face and upper body are usually covered, which makes the traditional physiological signal monitoring method that relies on facial rPPG signals unable to work properly. In order to solve this problem, the present invention adopts an rPPG signal acquisition scheme based on the limbs.

[0045] Specifically, a camera device (such as a near-infrared camera or a high-frame-rate RGB camera) is used to shoot the patient's exposed limbs to obtain video images of the patient's limb skin. These limb skin video images can reflect the subtle changes in light reflected by the patient's limb skin, so as to capture blood flow information (pulse waveform characteristics or blood pressure volume pulse signals), thereby completing the rPPG signal acquisition of the limbs, ensuring that even if the face is covered, effective physiological signals can be extracted from the limbs.

[0046] At this stage, it is crucial to ensure the stability of camera equipment configuration and signal acquisition, especially in medical environments, where the calibration of camera equipment and environmental light management will directly affect the quality of rPPG signal acquisition.

[0047] As a preferred embodiment of the present invention, in step 2), since light changes, equipment movement or slight displacement of the patient in the medical environment may interfere with the signal, the rPPG signal needs to be preprocessed after acquisition. The preprocessing focuses on the segmentation and enhancement of the skin area in the multi-site skin video to ensure the stability and accuracy of the subsequent rPPG pulse waveform feature extraction.

[0048] The skin region segmentation process is to extract a mask including the face and limbs.

[0049] Skin area enhancement processing is to filter and remove artifacts of multiple skin parts videos by using image processing technology to eliminate environmental noise and interference, ensuring that the multiple skin parts videos input into the BRPDNet model have sufficient signal quality.

[0050] The preprocessing method can use a variety of filtering techniques, such as using a bandpass filter to remove high-frequency and low-frequency noise and retain the frequency range related to the pulse. In this step, motion artifacts can also be eliminated through smoothing filtering and artifact removal techniques to improve the reliability of the signal.

[0051] As a preferred embodiment of the present invention, in step 3), the editing method of the physiological part prompt information is as follows: The physiological part prompt information is a physiological part code edited by using One-Hot Encoding to edit the physiological part prompt words input by the user, or is a vector corresponding to each of the multiple part visual features converted by the CLIP model from the physiological part prompt words input by the user.

[0052] Physiological part prompt words are prompt texts with part keywords (such as face, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.) manually entered by users based on the characteristics of different body parts in the collected multi-part skin videos.

[0053] Each of the physiological part codes or the vectors corresponds to a single or combined physiological part (such as face, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.), which is used to indicate the source parts of multi-part skin video signals and guide the attention of these parts by weighting.

[0054] As a preferred embodiment of the present invention, in step 4), see Figure 3 As shown, the present invention designs and builds a multi-region rPPG fusion physiological signal extraction model (BRPDNet) with the ability to dynamically adapt to multi-region rPPG signals. The architecture of the BRPDNet model mainly consists of an input layer, a multi-layer convolution layer (Conv Layer), a hyper-convolution module (Hyper-convModule), a physiological region prompt module (BioRegion Prompt Module), a time shift module (TSM), a dynamic attention layer (Dynamic Attention Layer), a multi-layer feature weighting module, a dense layer (Dense) and an output layer.

[0055] Among them, the multi-layer convolution layer is used as an encoder, and together with the super-convolution module, time translation module, and dynamic attention layer, it constitutes the feature extraction module of the BRPDNet model, and the multi-layer feature weighting module is used as a decoder. Their functions are as follows: The input layer is responsible for receiving the collected multi-part skin videos and performing preliminary standardization processing. Multi-part skin videos are optical sensor signals from different parts of the patient's body (such as arms and legs). The input size of each signal will vary depending on the resolution and sampling frequency of the camera device. Therefore, the preliminary standardization process can ensure that the multi-part skin videos collected from different parts can be processed in the same feature space. The multi-part skin video is a skin video obtained by shooting multiple different parts of the patient using a camera device. At the same time, it is responsible for receiving the physiological part prompt words input by the user, such as face, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.

[0056] The multi-layer convolution layer (Conv Layer) is responsible for extracting spatial features from the input multi-part skin video, which reflects the local changes in the light reflected by the skin. The convolution layer is the core of the BRPDNet model. Each layer scans the input multi-part skin video through multiple convolution kernels to extract low-level features such as edges and textures. Through the superposition of multiple layers of convolution, the BRPDNet model can gradually capture high-level features in multi-part skin videos, such as characteristic changes in pulse waveforms.

[0057] The Hyper-conv Module is responsible for dynamically adjusting the size and receptive field of the convolution kernel according to the body parts of the multi-part skin video, so as to improve the BRPDNet model's ability to capture the details of the multi-part skin video during the feature extraction process, and ensure that the BRPDNet model can efficiently extract the unique signal features of each part of the skin video. Figure 4 As shown in the figure, super convolution is an improved convolution operation that allows the convolution kernel to dynamically adjust its size and receptive field at runtime to adapt to skin videos from different body parts. By adjusting the size and receptive field of the convolution kernel, super convolution can improve the BRPDNet model's ability to capture signal details during feature extraction, especially in the extraction of differentiated signals between different parts.

[0058] The time shift module (TSM) is responsible for injecting the time series of multi-part skin videos into the multi-layer convolutional layers, so that the BRPDNet model can capture the time features in the multi-part skin videos, and is used to capture the dynamic fluctuations of the multi-part skin videos, and realize the time series analysis of the spatial features, so as to obtain stable and accurate spatiotemporal features, namely, the rPPG pulse waveform features; the multi-part skin videos are time series data, so the time series is introduced to ensure that the BRPDNet model has better processing capabilities for the time-dependent characteristics of the multi-part skin videos. For example, the BRPDNet model can better capture the changing patterns of the pulse waveform at different time points.

[0059] The Dynamic Attention Layer is responsible for automatically adjusting the BRPDNet model's attention to different areas in the multi-part skin video based on the characteristics of the multi-part skin video, thereby enhancing the BRPDNet model's attention to high-quality signal areas and reducing the weights of noisy areas. In multi-part skin videos, the signal quality of different parts may be different. For example, the signal of some parts is greatly affected by motion artifacts or light changes, while the signal of other parts is relatively stable. The dynamic attention layer can enhance the attention to high-quality signal areas and reduce the weights of noisy areas.

[0060] BioRegion Prompt Module, see Figure 5As shown, it is responsible for converting the physiological part prompt words (such as left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot, etc.) input by the user into vectors matching the visual features of multiple parts through the CLIP model, or using one-hot encoding to edit the physiological part prompt words input by the user into physiological part codes, and passing the vectors or the physiological part codes as input to the super convolution module, so as to provide a basis for the super convolution module to identify the body part sources of the input multi-part skin video, and help the super convolution module dynamically adjust the convolution kernel size and receptive field, thereby enhancing the BRPDNet model's attention to specific body parts.

[0061] These physiological part cues not only affect the hyperconvolution module, but also can be combined with asynchronous distillation technology in the BRPDNet model training to further optimize feature extraction in the distillation network. The distillation network learns effective feature extraction methods from the teacher model, and the physiological part cues enhance the student model (BRPDNet model)'s feature learning effect on key parts, allowing the BRPDNet model to assign higher weights to certain parts during signal processing. The introduction of asynchronous distillation ensures the efficient learning of the BRPDNet model in complex environments, reduces computational complexity, and improves the accuracy and robustness of multi-part signal extraction.

[0062] The multi-layer feature weighting module is responsible for fusing and weighting the rPPG pulse waveform features of each part extracted by the feature extraction module according to the weighted mechanism through the weighted gate layer, so as to ensure that the BRPDNet model can effectively fuse the rPPG pulse waveform features of multiple parts, and can appropriately adjust the weighting proportion according to the quality of the rPPG pulse waveform features. The weighting mechanism weights the rPPG pulse waveform features of each part according to the quality and time series of the rPPG pulse waveform features of each part, that is, the rPPG pulse waveform features with relatively stable and high signal quality will occupy a larger proportion in the model output, while the rPPG pulse waveform features with large noise or instability will occupy a smaller proportion in the model output. This weighting mechanism can significantly improve the accuracy of the final physiological parameter estimation.

[0063] The dense layer (Dense) is responsible for weighting and nonlinear mapping the feature vector output by the feature weighting module. This process essentially integrates multi-dimensional features into a compact representation, namely the comprehensive rPPG waveform signal, so that the BRPDNet model can convert the spatiotemporal features of the input image into one-dimensional continuous signal features, and finally output a waveform signal related to the target physiological signal (such as heart rate or respiratory rate) calculated later. In the BRPDNet model, the convolution features of multiple parts may form a high-dimensional vector. The dense layer can compress these high-dimensional features into a low-dimensional global representation H, which reduces the amount of calculation and improves the robustness of the BRPDNet model. Under the action of the dense layer, the BRPDNet model can remove redundant information and retain only the most valuable features for heart rate or respiratory rate estimation.

[0064] The feature of each part is represented as a tensor Fi, where i represents different parts (such as face, arm, etc.). Each Fi is a high-dimensional feature vector, representing the spatiotemporal information extracted from the part. The function of the dense layer is to integrate these features into the final global feature vector H. Assuming that the weight matrix of the dense layer is W, the operation of the dense layer can be expressed as: (1); In formula (1), σ is a nonlinear activation function (such as ReLU), which is used to introduce nonlinear features.

[0065] The output layer is responsible for outputting a one-dimensional visualized comprehensive rPPG waveform signal.

[0066] As a preferred embodiment of the present invention, in step 4), the training method of the BRPDNet model of the present invention is as follows: 1) Construction of dataset; A certain number of skin videos of multiple parts (such as arms and legs) of past patients in clinical environments are collected, and the corresponding PPG pulse waveform signal true values ​​and physiological parameter true values ​​including heart rate, blood oxygen, and respiratory rate are obtained synchronously, forming an rPPG dataset that contains multi-part skin videos aligned in time and space, paired with PPG pulse waveform signal true values ​​and physiological parameter true values.

[0067] In order to enhance the stability and robustness of the signal, the present invention designs and collects rPPG signal data sets covering multiple parts and states, such as face, arms and legs, in order to enhance the stability and robustness of the signal. The data covers subjects of different ages, skin colors and health conditions, so as to ensure that stable signal monitoring can be provided in a complex medical equipment environment. In particular, the rPPG signal of the leg is less affected by movement and light, has higher stability and reliability, and can be used as a stable source of physiological signals.

[0068] During the research of this invention, we cooperated with the Fifth Affiliated Hospital of Sun Yat-sen University to collect new datasets and integrated public facial datasets such as PURE, UBFC-rPPG, VIPL-HR, and UCLA-rPPG. Figure 6 As shown, Figure 6 Figure a is a sample of the PURE dataset, and figure b is a sample of the UBFC-rPPG dataset. At the same time, we also introduced multi-site PPG and respiratory monitoring data of 30 adults and newborns from the Kiang Wu Hospital in Macau, as well as 30 clinical case data from the Zhuhai People's Hospital.

[0069] It should be noted that the facial dataset can improve the generalization of the facial rPPG model for the teacher model on the one hand, and can also be used as a soft label for distillation to provide rPPG signal label guidance for multiple parts of the student model on the other hand. After adding RTT, the teacher's pred is equivalent to the student's label.

[0070] 2) Model training; In order to further improve the efficiency and accuracy of signal extraction, the present invention introduces an innovative physiological region prompt distillation network framework during model training. This framework dynamically adapts to the extraction of rPPG signal features in different parts by combining physiological region prompts (BioRegion Prompt) and hyperconvolution technology. As a result, the method of the present invention can focus on extracting signals from the limbs in scenes that are easily affected by facial expressions, head movements and lighting conditions, significantly improving the adaptability of the model and the accuracy of signal extraction.

[0071] See also Figure 3 and Figure 7 As shown in the figure, the BRPDNet model is trained by using the constructed rPPG signal dataset and the asynchronous distillation technology in the distillation network. The training process of the BRPDNet model adopts a staged guidance method, and the goal is to enhance the ability of the BRPDNet model to extract multi-part skin video features. During training, multi-part skin videos are used as input, the true value of the PPG pulse waveform signal is used as the primary label, and the true value of the physiological parameter is used as the supplementary label. The BRPDNet model learns the mapping relationship between the video frames of multi-part skin videos and the corresponding PPG pulse waveform true value features, so that the BRPDNet model can accurately extract the rPPG pulse waveform features of each part according to the input multi-part rPPG signals, and output the comprehensive rPPG waveform signal after fusion weighting.

[0072] In the BRPDNet model, the skin characteristics (such as color, texture, reflectivity, etc.) and blood flow changes in different parts have different signal characteristics. Using physiological part cues to guide the feature weighting of different parts enables the BRPDNet model to assign attention weights to specific parts (such as "face" or "arm") based on physiological part cues. However, in ordinary distillation, the target and non-target class information of the teacher model are transmitted synchronously, resulting in the student model being unable to refine the focus on specific parts when optimizing the target.

[0073] The asynchronous distillation technology of the present invention dynamically adjusts the learning weights of the signals of each part by decoupling the optimization process of the information of different parts, so that the signal features of each part can be efficiently extracted; the importance of the signals of each part is regarded as uniform and equal, and the BRPDNet model adjusts the optimization process according to the part prompted by the physiological part prompt information to ensure that the signal features of all parts are fully learned, and at the same time provides support for the independent prediction and fusion of multi-part signals; facial signals provide more guidance in the training of the BRPDNet model due to their strong reflective characteristics and large amount of data, but since facial signals are blocked in scenes such as CT equipment, the asynchronous distillation technology of the present invention enhances the learning and optimization of signals of parts other than the face, so that in the absence of facial signals, the BRPDNet model can still independently and accurately predict the physiological signal features of parts other than the face; in the process of multi-part signal fusion, the model dynamically adjusts the weighting method of the signals of each part to ensure that the physiological signal features output during the fusion process are consistent or synchronized with the facial signals in terms of timing and amplitude; the features of the "non-target class" parts are asynchronously optimized with the weights of another cycle, so that the student model can maintain a certain understanding of the features of the non-target parts without interfering with the main target features, thereby achieving efficient separation and optimization of multi-part information.

[0074] See also Figure 8 As shown, Figure 8 The first row is an example of the UBFC dataset, the second row is a schematic diagram of the feature learning hotspots in the mask area of ​​the teacher model under the distillation network, the fourth row is a schematic diagram of the feature learning hotspots in the mask area of ​​the student model under the distillation network, and the fourth row is the pulse feature waveform prediction output corresponding to the example of the UBFC dataset.

[0075] Different from traditional distillation, in asynchronous distillation, the present invention decouples the loss weights α and 1-α into two independent weights α target and β non-target , and apply these weights to the target class and non-target class respectively. The specific formula is as follows: L=α target · L target +β non-target · L non-target (2); In formula (2), L target It is the loss for the target class, which mainly depends on the hard loss L of the true label. hard and guidance from teacher models.

[0076] L non-target It is the loss of non-target classes, which mainly depends on the soft label loss L output by the teacher model. soft .

[0077] In addition, in the BRPDNet model, ordinary distillation requires large computing resources for synchronous optimization of all parts. The asynchronous distillation of the present invention divides the learning process of different parts into different cycles, which not only reduces the learning complexity of the student model, but also ensures attention to the characteristics of each part. For the lightweight multi-part rPPG fusion physiological signal extraction model, this update method of asynchronous distillation divided by part and prompt word can effectively cover the feature learning needs of multiple parts while saving calculations, and improve the performance of lightweight models in multiple parts.

[0078] In summary, in the BRPDNet model that combines multiple parts such as face and limbs and combines physiological part prompts, asynchronous distillation has the following significant advantages over ordinary distillation: (1) Part-by-part refinement and optimization to improve the model’s adaptability to the characteristics of different parts.

[0079] (2) The weighted attention stability guided by the cue words effectively improves the generalization effect of the cue words in multiple parts.

[0080] (3) Computational resource optimization to reduce the learning burden of student models and achieve efficient multi-part lightweight design.

[0081] These advantages enable asynchronous distillation in the BRPDNet model to more accurately identify and separate blood flow signal features from different parts, thereby improving the accuracy of heart rate estimation and the generalization ability of the model, which is especially important in applications on real-time devices or mobile devices.

[0082] The present invention builds a model based on Python 3.9 environment and PyTorch framework. The optimizer uses the Adam optimizer with an initial learning rate of 0.0001. The regularization step size is set to 1 and the gamma is set to 0.75. The number of training rounds is set to 50 and the batch size is 1. The present invention performs model training and testing on an Intel Xeon W2245 CPU @ 3.90GHz, 4×NVIDIA RTX A1000 computer.

[0083] 3) Model optimization; The present invention uses loss functions such as mean absolute error (MAE), root mean square error (RMSE) and signal-to-noise ratio (SNR) to optimize the model parameters of the BRPDNet model.

[0084] Mean absolute error (MAE) is used to evaluate the average absolute difference between the predicted value and the true value of the rPPG signal. For rPPG signals, MAE helps measure the accuracy of heart rate and other physiological parameter detection. The smaller the MAE, the more accurate the model's prediction. The calculation formula of MAE is as follows: (3); In formula (3), n is the number of samples, xi and yi are the predicted value and true value of the model respectively.

[0085] Root mean square error (RMSE) is used to evaluate the root mean square of the squared error between the predicted value and the true value of the rPPG signal. RMSE provides a more sensitive error metric than MAE because it gives higher weight to larger errors. For rPPG signals, RMSE helps detect biases in heart rate estimation. The smaller the RMSE, the better the model performance. The calculation formula of RMSE is as follows: (4); In formula (4), n is the number of samples, xi and yi are the predicted value and true value of the model respectively.

[0086] Signal-to-noise ratio (SNR) is used to evaluate the ratio of signal strength to noise strength in rPPG signals. For image frame sequences, SNR can measure the clarity and noise level of the signal during imaging to ensure high-quality images. For rPPG signals, SNR helps evaluate the reliability and stability of signal detection. The larger the SNR, the higher the signal quality and the smaller the noise. The SNR calculation formula is as follows: (5); In formula (5), P signal Indicates the signal strength, P noise Indicates the noise intensity of the signal.

[0087] Through the analysis and verification of these rPPG signal data sets, the BRPDNet model performed outstandingly in quality and stability, and effectively supported the fusion application of rPPG signals from multiple parts.

[0088] In a CT equipment environment, the mean absolute error, the root mean square error and the signal-to-noise ratio are used to calculate the error for signals of specific parts other than the face, and finally the total error is weighted to improve the generalization ability of the overall model and the ability to adapt to complex environments.

[0089] Based on the above-constructed rPPG signal dataset and the trained BRPDNet model, the present invention proposes a method for fusing rPPG signals from multiple sites to comprehensively monitor hemodynamic changes and help doctors better assess the long-term and short-term physiological health status of patients.

[0090] See also Figure 7 As shown, multiple skin videos of the current patient are collected and input into the trained BRPDNet model after preprocessing including image enhancement and semantic segmentation. At the same time, the physiological part prompt words corresponding to the multiple skin videos are manually input and converted into physiological part codes or vectors.

[0091] The BRPDNet model first adjusts the attention weights of skin videos of different parts through the input physiological part codes or vectors, dynamically optimizes the feature extraction capabilities of specific areas, extracts the rPPG pulse waveform features of the skin video space of each part, and combines time series analysis to ensure the spatiotemporal integrity and stability of the rPPG pulse waveform features. The asynchronous distillation process of the BRPDNet model further optimizes the effect of feature learning, allowing the model to distinguish different target areas and reduce the impact of irrelevant areas on model performance. In this way, the model is able to extract accurate physiological signals in different parts and complex scenarios. The BRPDNet model then obtains a comprehensive rPPG waveform signal by fusing and weighting the rPPG pulse waveform features of each part. This process optimizes the signal contribution of different parts through a multi-layer feature weighting module to ensure that the final physiological parameters have high accuracy and robustness.

[0092] Finally, an adaptive algorithm based on light changes is used to post-process the integrated rPPG pulse waveform signal to correct the signal distortion caused by changes in light intensity. Specifically, according to the extracted brightness value, cross-attention is performed on the real and imaginary parts respectively, and finally the inverse fast Fourier transform is converted into a brightness-adapted rPPG waveform signal output. The frequency domain analysis method is then used to calculate the post-processed rPPG waveform signal, calculate the physiological signal values ​​including heart rate and respiratory rate, and output a comprehensive report containing these physiological data. These physiological data can be fed back to medical equipment operators in real time for clinical monitoring and decision support.

[0093] In addition, in order to measure the consistency between the BRPDNet model test results and the subjective evaluation, the present invention uses the PCC (Pearson correlation coefficient), that is, the Pearson correction coefficient. The PCC coefficient evaluates the prediction accuracy of the model by calculating the linear correlation between two variables. The PCC coefficient ranges from -1 to 1. When the PCC value is 1, it means that the two sets of data are completely positively correlated. The calculation formula of PCC is as follows: (6); In formula (6), xi and yi are the model prediction value and the true value respectively. and In rPPG signal detection, PCC can measure the correlation between the predicted value and the true value of physiological parameters such as heart rate to ensure the accuracy and consistency of signal detection.

[0094] The PTT experiment used contact PPG sensors in multiple parts of the body to measure the blood pulse transmission time in different parts of the body, and calculated the time delay between each part through sliding cross correlation. As shown in Table 1, the PTT experiment results showed that there were significant differences in the blood pulse transmission time in different parts, among which the maximum delay between the left triceps and the right ankle was 51.23 milliseconds.

[0095] Table 1 PTT test comparison results

[0096] As can be seen from Table 1, there are significant differences in the time it takes for blood to be transported from the heart to different sites. Specifically, sites close to the heart (such as the right forearm, right triceps, and back of the neck) have shorter delay times, while sites far from the heart (such as the left knee, left ankle, and right ankle) have longer delay times. This indicates that the transport of blood at more distant sites is more affected. In addition, negative values ​​indicate that the blood reaches the comparison site before the reference site, and positive values ​​indicate that the blood reaches the comparison site after the reference site.

[0097] At the same time, the rPTT experiment used high frame rate video to remotely measure the pulse transmission time of different parts. As shown in Table 2, the rPPG experiment found that the maximum delay was 46.75 milliseconds, which occurred between the face and the left leg. The rPPG experimental results show that although there is noise in the rPPG signal, the remote pulse transmission time can be effectively estimated through high frame rate video, providing reliable data support for non-contact physiological monitoring.

[0098] Table 2 Comparison results of rPPG experiments

[0099] The tests and the above experiments prove that the BRPDNet model provided by the present invention has the following advantages: 1. Non-contact multi-site signal fusion to make up for signal loss in coverage scenarios: When a patient enters a medical device such as a CT scan, the face and upper body are partially blocked. The present invention achieves reliable monitoring of physiological signals by extracting rPPG signals from the limbs (such as left and right arms, left and right legs, etc.).

[0100] The experimental data in Tables 2 and 3 show that the maximum rPTT delay between the left leg and the face is 46.75 milliseconds. These results prove that the use of signal fusion of the limbs can effectively reduce the impact of facial occlusion on physiological signal detection.

[0101] 2. Significantly improved accuracy and consistency: The results of ablation experiments show that compared with other technologies such as PhysNet and RhythmMamba, after adding dynamic convolution kernel generation, the MAE of rPPG signal detection of the present invention dropped from 1.64 to 1.55, and the PCC increased to 0.76, which significantly improved the accuracy of signal prediction. The principle is that hyperconvolution dynamically generates convolution kernels suitable for different parts by fusing prompt information from physiological parts. This adaptive convolution kernel mechanism makes the model more sensitive to input signals, especially under complex lighting and coverage conditions, which significantly improves the stability of the model.

[0102] 3. Asynchronous distillation and knowledge distillation to reduce computational complexity: Through Asynchronous Knowledge Distillation, more effective feature extraction and dynamic adjustment of convolution kernels are achieved, enabling the model to screen features more flexibly in complex scenarios.

[0103] Compared with TSCAN and DeepPhys, BRPDNet's inference time is only 0.29 seconds per time, but it still maintains close accuracy, indicating that this method is extremely efficient and practical in environments with limited computing resources.

[0104] 4. Ability to adapt to noise interference and complex environment: Under complex conditions such as lighting changes, facial occlusion, and even head movement, the present invention significantly reduces the impact of environmental changes on the detection results by combining limb signal detection. In particular, under low-light and strong-light conditions, the present invention exhibits stronger anti-interference capabilities.

[0105] For example, in the UBFC-rPPG test set, the MAE of the method of the present invention is 1.57 and the PCC is 0.73, which is much better than PhysNet and RhythmMamba, which shows the stability and accuracy of the model of the present invention in various complex environments.

[0106] In addition to the simple physiological signal monitoring results, the present invention can also combine other information such as CT images to provide a comprehensive health status assessment. These data can be used for clinical diagnosis, patient status monitoring during surgery, or to provide continuous monitoring support in long-term health management.

[0107] The present invention also provides a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned multi-site rPPG fusion physiological signal monitoring method.

[0108] The present invention also provides a computer-readable storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned multi-site rPPG fusion physiological signal monitoring method.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-site rPPG fusion physiological signal monitoring method, characterized in that: include: Step 1) Collect skin videos of multiple parts of the current patient; Step 2) preprocessing the collected skin videos of multiple parts to eliminate environmental noise and interference; Step 3) editing physiological part prompt information for indicating the source parts of the collected multi-part skin video signals; Step 4) The pre-processed multi-site skin videos and the edited physiological site prompt information are used as input data and simultaneously sent to a multi-site rPPG fusion physiological signal extraction model trained based on an improved hyper-convolutional neural network and using asynchronous distillation technology for processing; The multi-site rPPG fusion physiological signal extraction model adjusts the attention weights of skin videos of different sites through the input physiological site prompt information, dynamically optimizes the feature extraction capability of specific areas, extracts the rPPG pulse waveform features of the skin video space of each site, and combines time series analysis to ensure the spatiotemporal integrity and stability of the rPPG pulse waveform features, and finally performs weighted fusion on the rPPG pulse waveform features extracted from each site to generate a comprehensive rPPG waveform signal; Step 5) The extracted comprehensive rPPG pulse waveform signal is post-processed, and then the physiological signal values ​​including heart rate and respiratory rate are calculated through frequency domain analysis method, and a comprehensive report containing these physiological data is output.

2. The multi-site rPPG fusion physiological signal monitoring method according to claim 1, characterized in that: The multi-site skin video is a skin video obtained by collecting skin from multiple different sites of a patient using a camera device including a near-infrared camera or a high-frame-rate RGB camera. The multi-site skin video can reflect subtle changes in light reflected by the skin from multiple different sites of the patient so as to capture blood flow information.

3. The multi-site rPPG fusion physiological signal monitoring method according to claim 1, characterized in that: The physiological part prompt information is a physiological part code edited by using one-hot encoding to edit the physiological part prompt word input by the user, or is a vector corresponding to each of the multiple part visual features converted by the CLIP model into the physiological part prompt word input by the user; The physiological part prompt words are prompt words with part keywords manually input by the user according to the characteristics of different body parts in the collected multi-part skin videos; Each of the physiological part codes or the vectors corresponds to a separate or combined physiological part, and is used to indicate the source parts of the multi-part skin video signals and to guide the attention weighting of these parts.

4. The multi-site rPPG fusion physiological signal monitoring method according to claim 1, characterized in that: The architecture of the multi-site rPPG fusion physiological signal extraction model includes an input layer, a feature extraction module, a physiological site prompt module, a feature weighting module and a dense layer; The input layer is responsible for receiving the collected skin videos of multiple parts and performing preliminary standardization processing. The skin videos of multiple parts are skin videos obtained by shooting multiple different parts of the patient using a camera device; and is also responsible for receiving the physiological part prompt words input by the user; The feature extraction module is composed of multiple convolutional layers, superconvolution modules, time translation modules and dynamic attention layers, which are responsible for extracting the rPPG pulse waveform features of each part from the skin videos of multiple parts, and obtaining a comprehensive rPPG waveform signal after fusion and weighting for subsequent physiological signal analysis; wherein, The multi-layer convolution layer is the core of the model, responsible for extracting spatial features from multi-part skin videos, and the spatial features reflect the local changes of light reflected by the skin; each convolution layer scans the multi-part skin videos through multiple convolution kernels to extract low-level features including edges and textures; through the superposition of multiple layers of convolution, the model can gradually capture high-level features including characteristic changes in pulse waveforms in multi-part skin videos; The hyperconvolution module is responsible for dynamically adjusting the size and receptive field of the convolution kernel according to the body part sources of the multi-part skin video, so as to improve the model's ability to capture the details of the multi-part skin video during the feature extraction process, and ensure that the model can efficiently extract the unique signal features of each part of the skin video; The time translation module is responsible for injecting the time series of the skin videos of multiple parts into the multi-layer convolutional layer, so that the model can capture the time features in the skin videos of multiple parts, and is used to capture the dynamic fluctuations of the skin videos of multiple parts, and realize the time series analysis of the spatial features, so as to obtain stable and accurate spatiotemporal features, that is, rPPG pulse waveform features; the introduction of the time series ensures that the model has better processing capabilities for the time-dependent characteristics of the skin videos of multiple parts, so that the model can better capture the changing laws of the pulse waveform at different time points; The dynamic attention layer is responsible for automatically adjusting the model's attention to different areas in the multi-part skin video according to the characteristics of the multi-part skin video, thereby enhancing the model's attention to the high-quality signal area and reducing the weight of the noisy area; The physiological part prompt module is responsible for converting the physiological part prompt words input by the user into vectors matching the visual features of multiple parts through the CLIP model, or editing the physiological part prompt words input by the user into physiological part codes by using one-hot encoding, and passing the vectors or the physiological part codes as input to the superconvolution module, so as to provide a basis for the superconvolution module to identify the body part sources of the input multi-part skin video, and help the superconvolution module dynamically adjust the convolution kernel size and receptive field, so as to enhance the model's attention to specific body parts; The feature weighting module is responsible for fusing and weighting the rPPG pulse waveform features of various parts extracted by the feature extraction module according to the weighting mechanism through the weighted gate layer, so as to ensure that the model can effectively fuse the rPPG pulse waveform features of multiple parts and adjust the weighting ratio according to the quality of the rPPG pulse waveform features; The weighting mechanism weights the rPPG pulse waveform features of each part according to the quality and time sequence of the rPPG pulse waveform features of each part, that is, the rPPG pulse waveform features that are relatively stable and have higher signal quality will occupy a larger proportion in the model output, while the rPPG pulse waveform features with larger noise or instability will occupy a smaller proportion in the model output; The dense layer is responsible for weighting and nonlinearly mapping the feature vector output by the feature weighting module, integrating the multi-dimensional features into a compact representation, namely, a comprehensive rPPG waveform signal.

5. The multi-site rPPG fusion physiological signal monitoring method according to claim 1, characterized in that: The training method of the multi-site rPPG fusion physiological signal extraction model is: 1) Collect a certain number of skin videos of multiple parts of patients in clinical environments, and synchronously obtain the corresponding PPG pulse waveform signal true value and physiological parameter true value including heart rate, blood oxygen, and respiratory rate, to form an rPPG dataset that contains time-space aligned skin videos of multiple parts, PPG pulse waveform signal true value and physiological parameter true value; 2) Using the constructed rPPG signal dataset, the asynchronous distillation technology is adopted in the distillation network to train the multi-part rPPG fusion physiological signal extraction model. The training process of the model adopts a staged guidance method, and the goal is to enhance the model's ability to extract multi-part skin video features. During training, multi-part skin videos are used as input, the true value of the PPG pulse waveform signal is used as the primary label, and the true value of the physiological parameter is used as the supplementary label. The model learns the mapping relationship between the video frames of the multi-part skin video and the corresponding PPG pulse waveform true value features, so that the model can accurately extract the rPPG pulse waveform features of each part according to the input multi-part rPPG signal, and output the comprehensive rPPG waveform signal after fusion weighting; 3) Use a specific loss function to optimize the model parameters.

6. The multi-site rPPG fusion physiological signal monitoring method according to claim 5, characterized in that: The asynchronous distillation technology decouples the optimization process of different parts information and dynamically adjusts the learning weights of the signals of each part, so that the signal features of each part can be efficiently extracted; the importance of the signals of each part is considered to be uniform and equal, and the model adjusts the optimization process according to the parts prompted by the physiological part prompt information to ensure that the signal features of all parts are fully learned, and at the same time provide support for the independent prediction and fusion of multi-part signals; facial signals provide more guidance in model training due to their strong reflective characteristics and large amount of data, but since facial signals are blocked in the CT equipment scene, the asynchronous distillation technology enhances the learning and optimization of signals of parts other than the face, so that in the absence of facial signals, the model can still independently and accurately predict the physiological signal features of parts other than the face; in the process of multi-part signal fusion, the model dynamically adjusts the weighting method of the signals of each part to ensure that the physiological signal features output during the fusion process are consistent or synchronized with the facial signals in terms of timing and amplitude; the features of the "non-target" parts are asynchronously optimized with the weights of another cycle, so that the student model can maintain a certain understanding of the features of the non-target parts without interfering with the main target features, thereby achieving efficient separation and optimization of multi-part information.

7. The multi-site rPPG fusion physiological signal monitoring method according to claim 5, characterized in that: The loss function includes mean absolute error, root mean square error and signal-to-noise ratio; wherein, The mean absolute error is used to evaluate the mean absolute difference between the predicted value and the true value of the rPPG signal; The root mean square error is used to evaluate the root mean square of the square error between the predicted value and the true value of the rPPG signal; The signal-to-noise ratio is used to evaluate the ratio of signal intensity to noise intensity in the rPPG signal; In a CT equipment environment, the mean absolute error, the root mean square error and the signal-to-noise ratio are used to calculate the error for signals of specific parts other than the face, and finally the total error is weighted to improve the generalization ability of the overall model and the ability to adapt to complex environments.

8. The multi-site rPPG fusion physiological signal monitoring method according to claim 1, characterized in that: An adaptive algorithm based on illumination changes is used to post-process the integrated rPPG pulse waveform signal to correct the signal distortion caused by changes in light intensity. Specifically, according to the extracted brightness value, cross-attention is performed on the real and imaginary parts respectively, and finally the inverse fast Fourier transform is performed to convert it into a brightness-adapted rPPG waveform signal output.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the multi-site rPPG fusion physiological signal monitoring method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that: The computer-readable storage medium stores at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the multi-site rPPG fusion physiological signal monitoring method as described in any one of claims 1-8.

Images