A method and system for visual enhancement of physiological signals based on video decoupling and reconstruction

Abstract

This invention discloses a method and system for visual enhancement of physiological signals based on video decoupling and reconstruction, belonging to the field of physiological signal processing technology. The method includes: calculating absolute and relative optical flow from an input video sequence using a deep learning optical flow model; performing forward-backward consistency verification on the relative optical flow to generate a reliability mask; obtaining a face detection mask sequence using a standard face detection algorithm; performing a dot product between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence; decomposing the face video into a canonical appearance image and a pulse signal sequence using an appearance decomposer network based on a multi-stream U-Net architecture; performing motion compensation on the canonical appearance image based on the absolute optical flow using a distortion function to generate a dynamic background video; and amplifying the pulse signal sequence and adding it to the dynamic background video to obtain the output video. This invention achieves clear visualization of rPPG pulse signals without destroying skin texture details.

Description

Technical Field

[0001] This invention relates to the fields of intelligent medical care, computer vision, and physiological signal processing technology, specifically to a method and system for visual enhancement of physiological signals based on video decoupling and reconstruction. Background Technology

[0002] In recent years, extracting remote photoplethysmography (rPPG) signals from facial videos using ordinary cameras has become a research hotspot in the field of non-contact physiological monitoring. rPPG signals reflect changes in subcutaneous blood volume caused by the cardiac cycle and are fundamental for estimating key physiological indicators such as heart rate, heart rate variability, and blood pressure. However, the raw rPPG signal is extremely weak in video and cannot be directly observed by the human eye. Therefore, visually amplifying and "visualizing" the rPPG signal is of great significance for doctors to make intuitive judgments about patients' conditions or for users to obtain immediate biofeedback.

[0003] Current mainstream video upscaling technologies can be broadly categorized into two types. The first type is traditional signal processing methods, such as Euler Video Magnification (EVM), which works by filtering and amplifying video frames within a specific frequency band. The limitation of this method is that it indiscriminately amplifies all changes within the frequency band, including non-physiological noise caused by minute head movements and facial expressions, resulting in a large number of motion artifacts in the amplified result, severely interfering with the observation of the true pulse signal. The second type is deep learning-based synthesis methods, which directly generate the amplified video by training a neural network. However, most existing models use a simple paradigm of predicting the "color residual" and adding it to the original video. This approach essentially overlays a color block onto the original pixels, easily destroying the original high-level texture and details of the skin, causing the amplified area to appear blurry and distorted, like "pigments" floating on the skin's surface, lacking realism and greatly limiting its application in medical scenarios requiring high-fidelity visual effects.

[0004] In summary, the core challenge of existing technologies lies in their inability to effectively decouple the three intertwined pieces of information in videos: static identity and appearance information (skin texture, skin color), dynamic macroscopic motion information (head posture, facial expression changes), and weak physiological signal information (rPPG pulse). Therefore, there is an urgent need for an innovative method that can structurally separate these information and achieve high-fidelity, controllable visual enhancement of pure physiological signals. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for visual enhancement of physiological signals based on video decoupling and reconstruction, so as to achieve clear visualization of rPPG pulse signals.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention provides a method for visual enhancement of physiological signals based on video decoupling and reconstruction, comprising the following steps:

[0008] Absolute and relative optical flow are calculated from the input video sequence using a deep learning optical flow model. Forward-backward consistency verification is performed on the relative optical flow to generate a reliability mask. A standard face detection algorithm is used to obtain a face detection mask sequence from the input video sequence. The reliability mask sequence is multiplied by the face detection mask sequence to obtain a spatial attention mask sequence.

[0009] An appearance decomposer network based on a multi-stream U-Net architecture is employed to decompose the input face video into a normalized appearance image and a pulse signal sequence. The appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer comprises two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to concatenate the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, used to output the normalized appearance image and the pulse signal sequence, respectively.

[0010] Using a distortion function, motion compensation is performed on a single frame of the standard appearance image based on absolute optical flow to generate a dynamic background video; the separated pulse signal sequence is amplified and added to the dynamic background video to obtain the final output video.

[0011] Furthermore, the step of calculating absolute and relative optical flow from the input video sequence using a deep learning optical flow model, performing forward-backward consistency verification on the relative optical flow, and generating a reliability mask specifically includes:

[0012] For the input video sequence ,in Representing the total number of video frames, the absolute optical flow used for motion compensation and the relative optical flow used for reliability assessment are calculated using an optical flow model. The resulting absolute optical flow field sequence is expressed as follows: ,in Represents the frame number index; the relative optical flow field includes a forward optical flow sequence and a backward optical flow sequence, wherein the forward optical flow sequence is represented as... ,in Representative from the first Frame to the The optical flow of the frame, the backward optical flow sequence is represented as ,in Representative from the first Frame to the Optical flow of a frame;

[0013] Forward-backward consistency verification is performed on the relative optical flow, and the error map of each frame of the video is calculated. :

[0014]

[0015] in, Let be a differentiable image warping function. Represents the L2 norm;

[0016] Error plot Smoothing filtering is performed over time to obtain a more stable error map. ;

[0017] An adaptive thresholding method is used to filter the error map. Perform binarization to generate a reliability mask sequence. .

[0018] Further, the step of applying a standard face detection algorithm to the input video sequence to obtain a face detection mask sequence, and then performing a dot product between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence, specifically includes:

[0019] The face region mask of each frame in the input video is obtained using a standard face detection algorithm, resulting in a face detection mask sequence. ;

[0020] Reliability mask sequence With face detection mask sequence Element-wise multiplication yields the final spatial attention mask sequence. ,in This represents element-wise dot product.

[0021] Furthermore, the appearance decomposer network based on the multi-stream U-Net architecture decomposes the input face video into a normalized appearance image and a pulse signal sequence, specifically including:

[0022] Original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The appearance decomposer network function is represented as follows:

[0023]

[0024] in For the network The parameters that can be learned.

[0025] Furthermore, the original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The specific process includes:

[0026] Original video sequence The input is fed into the appearance decomposer network, which extracts the appearance, color, and texture features of the video through the video encoded stream, and outputs an appearance feature sequence. Motion pattern features of the video are extracted through the optical flow encoded stream to generate a motion feature sequence. ;

[0027] Spatial attention mask sequence Downsampling is the motion feature sequence The size, and the motion feature sequence Perform dot product and output the attention motion feature sequence. ;

[0028] After dual-stream encoding, the appearance feature sequence is processed through a feature fusion and decoding layer. and attentional motion feature sequences The features are then stitched together; in the decoder stage, the resolution of the feature maps is gradually restored through a series of upsampling and convolution operations, and the decoder features are output.

[0029] The features, after feature fusion and decoding layer concatenation and decoding, are input into the normalized appearance head and the pulse layer head. The normalized appearance head averages the decoded features over time, and then outputs a single, static normalized appearance image through a convolutional layer. The pulse layer head directly processes the decoder features containing the time dimension, outputting a dynamic pulse signal sequence of the same length as the original video. .

[0030] Furthermore, the process involves using a distortion function to perform motion compensation on the standard appearance image of a single frame based on absolute optical flow to generate a dynamic background video; amplifying the separated pulse signal sequence and adding it to the dynamic background video to obtain the final output video, specifically including:

[0031] Using differentiable twist functions According to the absolute optical flow field sequence A standard appearance image for a single frame. Frame-by-frame warping is performed to generate a dynamic background video sequence that is perfectly synchronized with the original video motion but does not contain its own pulse. ;

[0032] According to the preset magnification factor For pulse signal sequences The video is then amplified and added to the dynamic background video sequence to obtain the final enhanced video sequence. : .

[0033] Secondly, the present invention provides a physiological signal visual enhancement system based on video decoupling and reconstruction, comprising:

[0034] The preprocessing and motion analysis module is used to calculate the absolute and relative optical flow from the input video sequence using a deep learning optical flow model, perform forward-backward consistency verification on the relative optical flow, and generate a reliability mask; it also uses a standard face detection algorithm to obtain a face detection mask sequence from the input video sequence, and performs a dot product between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence.

[0035] Multi-stream video decomposition network module: This module uses an appearance decomposer network based on a multi-stream U-Net architecture to decompose an input face video into a normalized appearance image and a pulse signal sequence. The appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer includes two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to concatenate the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, used to output a normalized appearance image and a pulse signal sequence, respectively.

[0036] Motion compensation and signal reconstruction module: This module uses a distortion function to perform motion compensation on a single frame of the standard appearance image based on absolute optical flow, generating a dynamic background video; it amplifies the separated pulse signal sequence and adds it to the dynamic background video to obtain the final output video.

[0037] The beneficial effects of this invention are:

[0038] This invention innovatively constructs a three-element decomposition framework, which intelligently decomposes facial videos into: a normalized appearance layer, a motion field, and a pulse signal layer. After the pulse signal layer is independently and controllably amplified, it is then reconstructed with the normalized appearance layer through motion field compensation in a high-fidelity manner, thereby achieving clear visualization of the rPPG pulse signal without destroying any skin texture details. Attached Figure Description

[0039] Figure 1This is a structural diagram of a physiological signal visual enhancement system based on video decoupling and reconstruction.

[0040] Figure 2 This is a flowchart of the preprocessing and motion analysis module in this invention;

[0041] Figure 3 This is a flowchart of the multi-stream video decomposition network module in this invention;

[0042] Figure 4 This is a flowchart of the motion compensation and signal reconstruction module in this invention. Detailed Implementation

[0043] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] See Figures 1-4 The present invention provides a technical solution:

[0045] Example 1:

[0046] This invention provides a method for visual enhancement of physiological signals based on video decoupling and reconstruction, comprising the following steps:

[0047] S1. Calculate the absolute optical flow and relative optical flow from the input video sequence using a deep learning optical flow model, perform forward-backward consistency verification on the relative optical flow, and generate a reliability mask; use a standard face detection algorithm on the input video sequence to obtain a face detection mask sequence, and perform a dot product calculation between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence.

[0048] like Figure 2 As shown, step S1 specifically includes:

[0049] For the input video sequence ,in Representing the total number of video frames, the absolute optical flow used for motion compensation and the relative optical flow used for reliability assessment are calculated using an optical flow model. The resulting absolute optical flow field sequence is expressed as follows: ,in Represents the frame number index; the relative optical flow field includes a forward optical flow sequence and a backward optical flow sequence, wherein the forward optical flow sequence is represented as... ,in Representative from the first Frame to the The optical flow of the frame, the backward optical flow sequence is represented as ,in Representative from the first Frame to the Optical flow of a frame;

[0050] Forward-backward consistency verification is performed on the relative optical flow, and the error map of each frame of the video is calculated. :

[0051]

[0052] in, Let be a differentiable image warping function. Represents the L2 norm;

[0053] Error plot Smoothing filtering is performed over time to obtain a more stable error map. ;

[0054] An adaptive thresholding method is used to filter the error map. Perform binarization to generate a reliability mask sequence. Reliability mask This method identifies regions in the video where optical flow estimation is unreliable due to intense motion or occlusion. These regions fail to reflect changes in the rPPG pulse signal and are therefore removed during subsequent pulse signal amplification. The adaptive thresholding method determines an initial threshold based on the statistical quantiles of the error map. And according to the preset minimum reliable area ratio The design ensures that the shielding area is neither too large nor too small.

[0055] The face region mask of each frame in the input video is obtained using a standard face detection algorithm, resulting in a face detection mask sequence. ;

[0056] Reliability mask sequence With face detection mask sequence Element-wise multiplication yields the final spatial attention mask sequence. ,in This represents element-wise dot product.

[0057] The absolute and relative optical flow fields obtained in step S1 through the deep optical flow network can provide high-quality motion priors and spatial attention information for the subsequent decomposition network.

[0058] S2. An appearance decomposer network based on a multi-stream U-Net architecture is used to decompose the input face video into a normalized appearance image and a sequence of impulse signals; such as Figure 3As shown, the appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer includes two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, which are used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to stitch together the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, which are used to output a normalized appearance image and a pulse signal sequence, respectively.

[0059] The appearance decomposer network, based on a multi-stream U-Net architecture, decomposes the input face video into a normalized appearance image and a sequence of impulse signals, specifically including:

[0060] Original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The appearance decomposer network function is represented as follows:

[0061]

[0062] in For the network The parameters that can be learned.

[0063] The original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The specific process includes:

[0064] Original video sequence The input is fed into the appearance decomposer network, which extracts the appearance, color, and texture features of the video through the video encoded stream, and outputs an appearance feature sequence. Motion pattern features of the video are extracted through the optical flow encoded stream to generate a motion feature sequence. ;

[0065] Spatial attention mask sequence Downsampling is the motion feature sequence The size, and the motion feature sequence Perform dot product and output the attention motion feature sequence. Input reliability spatial attention mask sequence It is used as a spatial gating system to guide the video stream to focus only on features of reliable regions;

[0066] After dual-stream encoding, the appearance feature sequence is processed through a feature fusion and decoding layer. and attentional motion feature sequences The features are then stitched together; in the decoder stage, the resolution of the feature maps is gradually restored through a series of upsampling and convolution operations, and the decoder features are output.

[0067] The features, after feature fusion and decoding layer concatenation and decoding, are input into the normalized appearance head and the pulse layer head. The normalized appearance head averages the decoded features over time to eliminate all dynamic information, and then outputs a single, static normalized appearance image through convolutional layers. The pulse layer head directly processes the decoder features containing the time dimension, outputting a dynamic pulse signal sequence of the same length as the original video. .

[0068] Step S2 uses an appearance decomposer network based on a multi-stream U-Net architecture to accurately decompose a single, static, canonical appearance image representing identity and skin texture from the input face video. It accurately separates the rPPG pulse signals in face videos into independent, low-energy pulse signal sequences. In this study, motion compensation is performed on standardized facial appearance images using motion field information from facial videos to generate clear, shaky facial appearance videos that are completely consistent with the original facial video's dynamics.

[0069] S3. Using a distortion function, motion compensation is performed on the single-frame normal appearance image based on absolute optical flow to generate a dynamic background video; the separated pulse signal sequence is amplified and added to the dynamic background video to obtain the final output video, such as... Figure 4 As shown, it specifically includes:

[0070] Using differentiable twist functions According to the absolute optical flow field sequence A standard appearance image for a single frame. Frame-by-frame warping is performed to generate a dynamic background video sequence that is perfectly synchronized with the original video motion but does not contain its own pulse. ;

[0071] According to the preset magnification factor For pulse signal sequences The video is then amplified and added to the dynamic background video sequence to obtain the final enhanced video sequence. : .

[0072] Step S3 achieves the processing of the pulse signal sequence. The linear and controllable amplification is then combined with the motion-compensated facial appearance video to generate the final high-fidelity enhanced facial video with well-preserved texture details.

[0073] Example 2:

[0074] This invention provides a visual enhancement system for physiological signals based on video decoupling and reconstruction, such as... Figure 1 As shown, it includes:

[0075] The preprocessing and motion analysis module is used to calculate the absolute and relative optical flow from the input video sequence using a deep learning optical flow model, perform forward-backward consistency verification on the relative optical flow, and generate a reliability mask; it also uses a standard face detection algorithm to obtain a face detection mask sequence from the input video sequence, and performs a dot product between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence.

[0076] Multi-stream video decomposition network module: This module uses an appearance decomposer network based on a multi-stream U-Net architecture to decompose an input face video into a normalized appearance image and a pulse signal sequence. The appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer includes two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to concatenate the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, used to output a normalized appearance image and a pulse signal sequence, respectively.

[0077] Motion compensation and signal reconstruction module: This module uses a distortion function to perform motion compensation on a single frame of the standard appearance image based on absolute optical flow, generating a dynamic background video; it amplifies the separated pulse signal sequence and adds it to the dynamic background video to obtain the final output video.

[0078] This invention innovatively constructs a three-element decomposition framework, which intelligently decomposes facial videos into: a normalized appearance layer, a motion field, and a pulse signal layer. After the pulse signal layer is independently and controllably amplified, it is then reconstructed with the normalized appearance layer through motion field compensation in a high-fidelity manner, thereby achieving clear visualization of the rPPG pulse signal without destroying any skin texture details.

[0079] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for visual enhancement of physiological signals based on video decoupling and reconstruction, characterized in that, Includes the following steps: The absolute optical flow and relative optical flow are calculated from the input video sequence using a deep learning optical flow model. The relative optical flow is then checked for forward-backward consistency to generate a reliability mask. A face detection mask sequence is obtained by applying a standard face detection algorithm to the input video sequence. The reliability mask sequence is multiplied by the face detection mask sequence to obtain the spatial attention mask sequence. An appearance decomposer network based on a multi-stream U-Net architecture is employed to decompose the input face video into a normalized appearance image and a pulse signal sequence. The appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer comprises two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to concatenate the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, used to output the normalized appearance image and the pulse signal sequence, respectively. Using a warp function, motion compensation is performed on a single frame of the canonical appearance image based on absolute optical flow to generate a dynamic background video; The separated pulse signal sequence is amplified and added to the dynamic background video to obtain the final output video.

2. The method for visual enhancement of physiological signals based on video decoupling and reconstruction according to claim 1, characterized in that: The process of calculating absolute and relative optical flow from the input video sequence using a deep learning optical flow model, performing forward-backward consistency verification on the relative optical flow, and generating a reliability mask specifically includes: For the input video sequence ,in Representing the total number of video frames, the absolute optical flow used for motion compensation and the relative optical flow used for reliability assessment are calculated using an optical flow model. The resulting absolute optical flow field sequence is expressed as follows: ,in Represents the frame number index; the relative optical flow field includes a forward optical flow sequence and a backward optical flow sequence, wherein the forward optical flow sequence is represented as... ,in Representative from the first Frame to the The optical flow of the frame, the backward optical flow sequence is represented as ,in Representative from the first Frame to the Optical flow of a frame; Forward-backward consistency verification is performed on the relative optical flow, and the error map of each frame of the video is calculated. : in, Let be a differentiable image warping function. Represents the L2 norm; Error plot Smoothing filtering is performed over time to obtain a more stable error map. ; An adaptive thresholding method is used to filter the error map. Perform binarization to generate a reliability mask sequence. .

3. The method for visual enhancement of physiological signals based on video decoupling and reconstruction according to claim 2, characterized in that: The process involves applying a standard face detection algorithm to the input video sequence to obtain a face detection mask sequence, and then performing a dot product between the reliability mask sequence and the face detection mask sequence to obtain a spatial attention mask sequence. Specifically, this includes: The face region mask of each frame in the input video is obtained using a standard face detection algorithm, resulting in a face detection mask sequence. ; Reliability mask sequence With face detection mask sequence Element-wise multiplication yields the final spatial attention mask sequence. ,in This represents element-wise dot product.

4. The method for visual enhancement of physiological signals based on video decoupling and reconstruction according to claim 1, characterized in that: The appearance decomposer network, based on a multi-stream U-Net architecture, decomposes the input face video into a normalized appearance image and a sequence of impulse signals, specifically including: Original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The appearance decomposer network function is represented as follows: in For the network The parameters that can be learned.

5. The method for visual enhancement of physiological signals based on video decoupling and reconstruction according to claim 4, characterized in that: The original video sequence Absolute optical flow field sequence and spatial attention mask sequences Input the appearance decomposer network to obtain the normalized appearance image. and pulse signal sequence The specific process includes: Original video sequence The input is fed into the appearance decomposer network, which extracts the appearance, color, and texture features of the video through the video encoded stream, and outputs an appearance feature sequence. Motion pattern features of the video are extracted through the optical flow encoded stream to generate a motion feature sequence. ; Spatial attention mask sequence Downsampling is the motion feature sequence The size, and the motion feature sequence Perform dot product and output the attention motion feature sequence. ; After dual-stream encoding, the appearance feature sequence is processed through a feature fusion and decoding layer. and attentional motion feature sequences The features are then stitched together; in the decoder stage, the resolution of the feature maps is gradually restored through a series of upsampling and convolution operations, and the decoder features are output. The features, after feature fusion and decoding layer concatenation and decoding, are input into the normalized appearance head and the pulse layer head. The normalized appearance head averages the decoded features over time, and then outputs a single, static normalized appearance image through a convolutional layer. The pulse layer head directly processes the decoder features containing the time dimension, outputting a dynamic pulse signal sequence of the same length as the original video. .

6. The method for visual enhancement of physiological signals based on video decoupling and reconstruction according to claim 1, characterized in that: The process utilizes a distortion function to perform motion compensation on a single frame of a standard appearance image based on absolute optical flow, thereby generating a dynamic background video. The separated pulse signal sequence is amplified and added to the dynamic background video to obtain the final output video, which specifically includes: Using differentiable twist functions According to the absolute optical flow field sequence A standard appearance image for a single frame. Frame-by-frame warping is performed to generate a dynamic background video sequence that is perfectly synchronized with the original video motion but does not contain its own pulse. ; According to the preset magnification factor For pulse signal sequences The video is then amplified and added to the dynamic background video sequence to obtain the final enhanced video sequence. : .

7. A physiological signal visual enhancement system based on video decoupling and reconstruction, characterized in that, include: Preprocessing and motion analysis module: Used to calculate absolute and relative optical flow from input video sequences using a deep learning optical flow model, perform forward-backward consistency verification on the relative optical flow, and generate a reliability mask; A face detection mask sequence is obtained by applying a standard face detection algorithm to the input video sequence. The reliability mask sequence is multiplied by the face detection mask sequence to obtain the spatial attention mask sequence. Multi-stream video decomposition network module: This module uses an appearance decomposer network based on a multi-stream U-Net architecture to decompose an input face video into a normalized appearance image and a pulse signal sequence. The appearance decomposer network includes a dual-stream coding layer, a feature fusion and decoding layer, and a dual-stream output head layer. The dual-stream coding layer includes two independent and parallel encoder streams: a video coding stream and an optical flow coding stream, used to extract appearance and motion features, respectively. The feature fusion and decoding layer is used to concatenate the extracted appearance and motion features and restore the resolution of the feature map. The dual-stream output head layer includes two independent output heads: a normalized appearance head and a pulse layer head, used to output a normalized appearance image and a pulse signal sequence, respectively. Motion compensation and signal reconstruction module: Used to perform motion compensation on a single frame of a standard appearance image based on absolute optical flow using a distortion function to generate a dynamic background video; The separated pulse signal sequence is amplified and added to the dynamic background video to obtain the final output video.

Images