Multi-mode optical non-contact health monitoring system and method

Abstract

The invention belongs to the technical field of physiological parameter measurement and testing, and discloses a multi-mode optical non-contact health monitoring system and method. The system comprises a sensing measurement module, a data synchronous acquisition module, a micro-motion stabilization and correction module, a three-dimensional trajectory reconstruction module, a multi-modal data processing module and a physiological index calculation module. Optical imaging data are processed preferentially through a U-Net network and then fused with video micro-motion features, and the problem of multi-modal data coupling interference in a traditional method is solved. A mathematical mapping relation between blood flow parameters and physiological indexes is established through a diffusion optical spectrum mapping network, the segmented blood flow parameters are mapped to initial physiological indexes, and the time lag problem of single-modal data is solved through time sequence features extracted through a time sequence feature fusion network. And the weights of the time sequence characteristics and the preliminary physiological indexes are dynamically adjusted through a space-time attention mechanism, so that the robustness in a complex environment is enhanced, and the prediction precision and stability of a non-contact health monitoring system are improved.

Description

Technical Field

[0001] The present invention obtains multimodal information of human physiological indicators through optical imaging of the human face, relates to physiological parameter detection equipment, and particularly discloses a multimodal optical non-contact health monitoring system and method, belonging to the technical field of measurement and testing. Background Art

[0002] As an important development direction in the field of biomedical engineering, non-contact physiological detection technology has shown broad application prospects in health monitoring, clinical diagnosis and other fields in recent years. Existing technologies mainly use optics, video analysis, laser speckle contrast analysis (LASCA), optical coherence tomography (OCT) and other principles to achieve physiological parameter detection, but there are still significant technical bottlenecks in practical application:

[0003] 1. Lack of multimodal fusion technology

[0004] Current mainstream devices generally rely on a single detection modality. For example, optical detection technology based on photoplethysmography (PPG) can monitor heart rate, but it struggles to simultaneously acquire multidimensional physiological indicators such as blood oxygen and blood pressure. Video photoplethysmography (VPG) can acquire hemodynamic information contactlessly, but it is susceptible to interference from factors such as ambient lighting and skin pigmentation, resulting in unstable test results. This single-modality limitation makes it difficult for these devices to meet the needs of multi-parameter joint analysis in complex clinical scenarios.

[0005] 2. Conflict between motion interference and privacy protection

[0006] Video analysis technologies face serious motion artifact issues during dynamic monitoring. Even the slightest movement of the subject can distort blood flow signals. Existing solutions, such as the privacy protection method based on target recognition and dynamic frame processing proposed in Chinese patent CN112545339A, can protect personal information through blurring. However, this requires frame-by-frame feature extraction and target tracking of the video stream, which increases the computational complexity of the system and makes it difficult to ensure real-time performance. This contradiction has hindered the large-scale application of this technology in scenarios such as wearable devices.

[0007] 3. Limited accuracy of blood flow parameter detection

[0008] Traditional blood flow detection techniques have the problem of being difficult to balance resolution and dynamics: Although laser speckle technology can achieve high-resolution imaging of blood flow velocity, it cannot synchronously obtain dynamic parameters such as blood pressure; OCT technology can provide vascular structure information with micron-level resolution, but it is difficult to achieve real-time monitoring of hemodynamic parameters. This single-dimensionality of detection makes existing devices unable to meet clinical needs such as early diagnosis of microcirculation disorders.

[0009] In summary, there are significant technical gaps in existing non-contact physiological detection techniques in aspects such as multi-modal fusion algorithms, anti-motion interference mechanisms, and high-precision blood flow parameter detection. There is an urgent need to break through based on a non-contact health monitoring system based on multi-modal optics. Summary of the Invention

[0010] The object of the present invention is to address the deficiencies of the above background technology by providing a multi-modal optical non-contact health monitoring system and method. By performing optical processing on a human face image after anti-motion interference processing to obtain multi-modal optical data, and jointly detecting multiple physiological parameters through a data processing method that maps multi-modal optical data to human physiological indicators, the technical problems of single detection modality, susceptibility to motion interference, and limited accuracy of blood flow parameter monitoring in existing non-contact physiological detection techniques are solved, and the object of the invention of non-contact joint detection of human physiological parameters based on multi-modal optical data is achieved.

[0011] The present invention adopts the following technical solutions to achieve the above object of the invention:

[0012] The multi-modal optical non-contact health monitoring system includes: a perception measurement module, a data synchronous acquisition module, a micro-motion stabilization and correction module, a three-dimensional trajectory reconstruction module, a multi-modal data processing module, and a physiological index calculation module; the perception measurement module is used to obtain the facial video image, TOF depth map and IMU data of the human body; the data synchronous acquisition module is used to synchronously acquire the facial video image, TOF depth map and IMU data of the human body; the micro-motion stabilization and correction module is used to perform stabilization processing and motion artifact correction on the synchronously acquired facial video image, and output a stable facial video frame, the final local micro-motion field, the main frequency signal of the heart rate physiological frequency band, and the main frequency signal of the respiratory frequency physiological frequency band; the three-dimensional trajectory reconstruction module is used to reconstruct the local micro-motion three-dimensional trajectory according to the two-dimensional displacement time series data in the final local micro-motion field; the multi-modal data processing module is used to perform optical processing on the stable facial video frame to obtain multi-modal optical data, and align the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the local micro-motion three-dimensional trajectory; the physiological index calculation module is used to map the blood flow parameters to the preliminary blood pressure value, blood oxygen monitoring value and body temperature monitoring value through physical model driving based on the multi-modal optical data aligned in the three-dimensional space-time, extract the local micro-motion time series features related to the heart rate and respiratory frequency from the main frequency signals of the heart rate and respiratory frequency physiological frequency bands respectively, obtain the heart rate and respiratory frequency monitoring values, dynamically weight and fuse the preliminary blood pressure value and the blood pressure value calculated from the local micro-motion time series features related to the heart rate, obtain the blood pressure monitoring value, and output the heart rate, blood pressure, blood oxygen, body temperature, and respiratory frequency monitoring values.

[0013] As a further optimization solution of the multi-modal optical non-contact health monitoring system, the perception measurement module includes: an RGB camera for collecting the facial video image of the human body, a time-of-flight sensor for collecting the TOF depth map of the human body face, and an inertial measurement unit for collecting IMU data; the data synchronous acquisition module collects the facial video image, TOF depth map and IMU data of the human body through hardware synchronous technology.

[0014] As a further optimization solution of the multi-modal optical non-contact health monitoring system, the multi-modal optical module adopts a tunable multi-wavelength light source or a multi-wavelength laser, and is used to obtain multi-modal optical data including but not limited to laser speckle images, hyperspectral data, and infrared thermal images.

[0015] As a further optimization solution for the multimodal optical non-contact health monitoring system, the micro-motion stabilization and correction module includes: a stabilized image unit and an artifact correction unit; the stabilized image unit corrects the facial video frame image through the SURF algorithm correction model and outputs a stabilized facial video frame; the artifact correction unit predicts the pixel-level two-dimensional motion vector of the facial video frame image through a lightweight vision TRANSFORMER to generate a preliminary dense motion vector field, generates a preliminary local dense motion vector field through an improved pyramid Lucas-Kanade optical flow method, corrects the preliminary local dense motion vector field to obtain a corrected local micro-motion field, generates a complete motion field based on the corrected local micro-motion field and performs global motion component correction, and then obtains the final local micro-motion field from the corrected global motion component, and extracts the main frequency signals of the heart rate and respiratory rate physiological frequency bands from the final local micro-motion field respectively.

[0016] As a further optimization solution for the multimodal optical non-contact health monitoring system, the physiological index calculation module includes: a U-Net blood flow segmentation network, a diffuse optical spectroscopy mapping network, and a temporal feature fusion network; the U-Net blood flow segmentation network is used to extract multi-scale blood flow region features from multimodal optical data, reconstruct a blood flow region mask, generate a binary blood flow region segmentation result, and extract blood flow parameters of each blood flow region, and the blood flow parameters include but are not limited to blood flow velocity, blood oxygen saturation, and hemoglobin concentration; the diffuse optical spectroscopy mapping network is used to establish a mathematical mapping relationship that maps blood flow parameters to preliminary physiological indexes and outputs preliminary blood pressure values, blood oxygen monitoring values, and body temperature monitoring values; the temporal feature fusion network extracts local micro-motion temporal features related to heart rate and respiratory rate from the main frequency signals of the heart rate and respiratory rate physiological frequency bands respectively through the temporal feature fusion network, and dynamically weights and fuses the preliminary blood pressure value with the blood pressure value calculated from the local micro-motion temporal features related to heart rate through a spatio-temporal attention mechanism to obtain a blood pressure value monitoring value.

[0017] As a further optimization solution for the multimodal optical non-contact health monitoring system, the system further includes an environment adaptation module, and the environment adaptation module adjusts the motion of the sensing measurement module through a feedback control algorithm; or adjusts the calculation parameters of the micro-motion stabilization and correction module; or adjusts the calculation parameters of the three-dimensional trajectory reconstruction module; or adjusts the intensity of the multi-wavelength light source; the physiological index calculation module further includes a temperature calibration sub-module, and the temperature calibration sub-module compensates for the temperature drift error in the blood flow parameters based on the temperature data of infrared thermal imaging through Kalman filtering.

[0018] The multimodal optical non-contact health monitoring method is implemented based on the above system and includes the following steps:

[0019] S1, synchronously collect the human face video image, TOF depth map, and IMU data of the perception measurement module;

[0020] S2, perform stabilization processing and motion artifact correction on the collected face video image to obtain a stable face video frame, the final local micro motion field, the main frequency signal of the heart rate physiological frequency band, and the main frequency signal of the breathing rate;

[0021] S3, reconstruct the local micro motion three-dimensional trajectory according to the two-dimensional displacement time series data in the final local micro motion field;

[0022] S4, perform laser speckle blood flow imaging, hyperspectral imaging, and infrared thermal imaging on the stabilized face video image to obtain multi-modal optical data, and align the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the local micro motion three-dimensional trajectory;

[0023] S5, based on the multi-modal optical data aligned in three-dimensional space-time, map the blood flow parameters to the preliminary blood pressure value, blood oxygen monitoring value, and body temperature monitoring value through physical model driving. Extract the local micro motion time series features related to heart rate and breathing rate from the main frequency signal of the heart rate physiological frequency band and the main frequency signal of the breathing rate physiological frequency band through time series feature fusion technology to obtain heart rate and breathing rate monitoring values. Dynamically weight and fuse the preliminary blood pressure value with the blood pressure value calculated from the local micro motion time series features related to heart rate through the spatio-temporal attention mechanism to obtain the blood pressure monitoring value.

[0024] As a further optimization method of the multi-modal optical non-contact health monitoring method, S2 specifically includes the following steps:

[0025] S2A, register adjacent frame face video images through affine transformation to align the input face video frame images;

[0026] S2B, for the face video image after being aligned by S2A, detect global feature points using the SURF algorithm, obtain the global motion component by calculating the global affine transformation matrix, and perform geometric inverse transformation on the input face video frame image according to the global affine transformation matrix to output a stable face video frame;

[0027] S2C, perform pixel-level two-dimensional motion vector prediction on the face video image after being aligned by S2A through a lightweight vision TRANSFORMER to generate a preliminary dense motion vector field;

[0028] S2D, screen the global feature points on the face video image after global motion correction by S2B through the Hessian matrix, and divide the local area with the screened global feature points as the anchor points;

[0029] S2E processes the preliminary dense motion vector field using an improved pyramid Lucas-Kanade optical flow method to generate a preliminary local dense motion vector field;

[0030] S2F uses the RANSAC algorithm to detect outliers in the preliminary local dense motion vector field, eliminate abnormal motions, and obtain a corrected local micro motion field;

[0031] S2G superimposes the global motion component and the corrected local micro motion field to generate a complete motion vector field;

[0032] S2H uses the complete motion vector field to verify whether there are residual anomalies in the global motion component. If there are residual anomalies, they are eliminated through the RANSAC algorithm to obtain a corrected global motion component;

[0033] S2I subtracts the corrected global motion component from the complete motion field to obtain the final local micro motion field;

[0034] S2J preprocesses the final local micro motion field, extracts two-dimensional displacement time series data in the physiological frequency band of the preprocessed final local micro motion field, and performs multi-scale decomposition and hard threshold denoising using wavelet transform. The coefficients in the physiological frequency bands of heart rate and respiratory frequency are retained, and the main frequencies in the physiological frequency bands of heart rate and respiratory frequency are extracted after reconstructing the coefficients in the physiological frequency bands of heart rate and respiratory frequency.

[0035] As a further optimization scheme of the multi-modal optical non-contact health monitoring method, S2E specifically includes the following steps:

[0036] S2E1 performs multi-scale downsampling on each local area to construct a pyramid for optimizing optical flow estimation layer by layer from low resolution to high resolution;

[0037] S2E2 quickly estimates the motion range of the local area with the preliminary dense motion vector field generated by S2C as the initial value and performs low-resolution layer optimization;

[0038] S2E3 solves the optical flow equation based on the brightness constancy assumption to iteratively optimize the two-dimensional motion vectors of each pixel in the local area and performs high-resolution layer optimization.

[0039] As a further optimization scheme of the multi-modal optical non-contact health monitoring method, S3 reconstructs the three-dimensional trajectory of the local micro motion based on the two-dimensional displacement time series data in the final local micro motion field, specifically:

[0040] S3A obtains the simultaneously acquired TOF depth map, facial video image of the human body, and IMU data;

[0041] S3B uses statistical outlier removal and radius filtering to eliminate noise points in the TOF depth map data, generate a depth map of the human face area, combine it with the facial video image, and use Mask R-CNN to segment the depth map of the human face area to obtain a depth map focusing on physiological micro-movements;

[0042] S3C registers the point cloud data of the depth map focusing on physiological micro-movements through the iterative closest point algorithm, separates the overall head movement and local micro-movements, determines the target area from the separated local micro-movement point cloud data, and generates the three-dimensional trajectory of the local micro-movement according to the mapping relationship between the depth map of the target area and the three-dimensional coordinates of the facial video image established by the internal and external parameter calibrations of the TOF sensor.

[0043] The present invention adopts the above technical solutions and has the following beneficial effects:

[0044] (1) The present invention performs global motion compensation on continuous video frames through the SURF algorithm correction model, stabilizes and corrects motion artifacts, and is provided with an environment adaptive module, significantly improving the reliability and accuracy of non-contact health monitoring in complex environments such as motion and light changes.

[0045] (2) The present invention preferentially processes optical imaging data through the U-Net network and then fuses it with video micro-movement features to solve the problem of multi-modal data coupling interference in traditional methods. A mathematical mapping relationship between blood flow parameters and physiological indicators is established through the diffuse optical spectroscopy mapping network, and the segmented blood flow parameters are mapped to preliminary physiological indicators such as blood pressure, blood oxygen, and body temperature. The local micro-movement time series features related to heart rate and respiratory rate are extracted through the time series feature fusion network to solve the problem of time lag of single-modal data; then, the weights of the blood pressure value calculated by combining the local micro-movement time series features related to heart rate and the preliminary blood pressure mapping value are dynamically adjusted through the spatio-temporal attention mechanism to enhance the robustness in complex environments and improve the prediction accuracy and stability of the non-contact health monitoring system.

[0046] (3) The present invention combines multi-modal optical data such as laser speckle imaging, hyperspectral data, and infrared thermal imaging to simultaneously obtain multi-dimensional physiological indicators such as blood pressure, heart rate, and blood oxygen saturation, overcomes the limitations of traditional single-modalities such as only PPG or VPG, meets the requirements of multi-parameter joint analysis in complex clinical scenarios, and improves the comprehensiveness of the detection dimension.

[0047] (4) The present invention maps two-dimensional optical flow to three-dimensional displacement through a kinematic model, combines Kalman filtering to suppress noise, provides spatial dynamic information, significantly improves the analysis accuracy and robustness of physiological signals such as skin vibration and chest and abdomen undulation, and provides a unified spatio-temporal reference framework for multi-modal data fusion.

[0048] (5) In the present invention, physical constraints such as Beer-Lambert's law are introduced into the diffuse optical spectroscopy mapping network to constrain the relationship between the predicted value of blood oxygen saturation and the optical reflection intensity, enhance physical interpretability, ensure that the output results conform to medical laws, and improve doctors' trust in the system output.

[0049] (6) The present invention is provided with an environment adaptive module, which dynamically adjusts the intensity of the multi-wavelength light source through feedback control to automatically adapt to different lighting conditions, reduce the influence of external interference on the quality of optical signals, and ensure detection stability.

[0050] (7) Through multi-level corrections such as the SURF algorithm, RANSAC outlier rejection, and Kalman filtering, the present invention separates global motion from local micro-motion, and can still accurately extract physiological signals such as heart rate and respiratory rate in scenarios where the patient moves slightly or the camera shakes. Description of the Drawings

[0051] Figure 1 It is a schematic diagram of the system module in Embodiment 1.

[0052] Figure 2 It is a flowchart of the monitoring method in Embodiment 2.

[0053] Figure 3 It is a flowchart of using the lightweight vision TRANSFORMER and the SURF algorithm in S2 of Embodiment 2 to correct motion artifacts to obtain a stable facial video frame.

[0054] Figure 4 It is a flowchart of generating a refined local dense motion vector field by using the improved pyramid Lucas-Kanade optical flow method in S5 of Embodiment 2. Detailed Embodiments

[0055] The technical solutions of the present invention will be further described in detail below in conjunction with specific embodiments. However, these embodiments do not limit the present invention. Any similar structures and their similar changes using the present invention should be included in the protection scope of the present invention. The commas in the present invention all represent the relationship of "and", and the English letters in the present invention are case-sensitive.

[0056] Embodiment 1

[0057] As Figure 1 shown, in an embodiment of the present invention, a multi-modal optical non-contact health monitoring system is provided, including: a perception measurement module, a data synchronous acquisition module, a micro-motion stabilization and correction module, a three-dimensional trajectory reconstruction module, a multi-modal data processing module, and a physiological index calculation module.

[0058] Perception measurement module, including: Red, Green, Blue (RGB) camera, Time of Flight (TOF) sensor and Inertial Measurement Unit (IMU). The RGB camera is used to obtain the facial video image of the human body. The facial video image contains the texture and color information of the human face. The RGB camera is one of a high-frame-rate camera, a high-speed CMOS camera array, and a PTZ-type camera. Among them, the high-speed CMOS camera array uses a beam splitter prism to separate reflected light of different wavelengths to a dedicated sensor to achieve multi-spectral synchronous acquisition, with a frame rate ≥ 200 fps to avoid crosstalk. The Time of Flight sensor is used to obtain the TOF depth map of the human face. The TOF depth map reflects the depth information of the human face. The Inertial Measurement Unit is used to collect the IMU data of the perception measurement module and record the motion state of the perception measurement module, such as acceleration and angular velocity.

[0059] Data synchronous acquisition module, used to synchronously acquire the facial video image, TOF depth map and IMU data of the human body, specifically ensuring the spatio-temporal consistency of the acquired data through hardware synchronization such as GPIO triggering.

[0060] Micro-motion stabilization and correction module, used to perform stabilization processing and motion artifact correction on the acquired facial video image, and output a stable facial video frame, the final local micro-motion field, the main frequency signal of the heart rate physiological frequency band, and the main frequency signal of the respiratory frequency physiological frequency band. The micro-motion stabilization and correction module includes a lightweight vision TRANSFORMER and a SURF algorithm correction model; the lightweight vision TRANSFORMER adopts an encoder-decoder structure, and predicts the two-dimensional motion vector of each pixel in the facial video frame through the encoder-decoder structure to generate a preliminary dense motion vector field , capture subtle motions such as skin vibrations and chest and abdomen undulations, then improve the motion estimation accuracy of the local area through an improved pyramid Lucas-Kanade optical flow method to generate a preliminary local dense motion vector field. The preliminary local dense motion vector field provides a high-resolution input for subsequent motion correction and signal analysis. Correct the preliminary local dense motion vector field to obtain the corrected local micro-motion field, generate a complete motion field based on the corrected local micro-motion field and perform global motion component correction, and then obtain the final local micro-motion field from the corrected global motion component. The final local micro-motion field includes a skin vibration micro-motion field and a chest and abdomen undulation micro-motion field. Extract the main frequency signal of the heart rate physiological frequency band and the main frequency signal of the respiratory frequency physiological frequency band from the final local micro-motion field respectively; the SURF algorithm correction model relies on the global motion component to achieve precise correction. This design combines the stability of traditional feature point detection with the data-driven advantages of deep learning, and is one of the core technical breakthroughs of the non-contact health monitoring system.

[0061] A three-dimensional trajectory reconstruction module for reconstructing the three-dimensional trajectory of local micro-movement based on the two-dimensional displacement time-series data in the final local micro-movement field.

[0062] A multi-modal data processing module for optically processing stable facial video frames to obtain multi-modal optical data; wherein, the multi-modal optical data includes laser speckle images, hyperspectral data, and infrared thermal imaging. Align the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the three-dimensional trajectory of local micro-movement to ensure the consistency of the multi-modal optical data in space and time; preferably, the multi-modal optical module includes a tunable multi-wavelength light source or multi-wavelength lasers. The tunable multi-wavelength light source integrates RGB, near-infrared, and far-infrared light sources. The RGB light source can select three-color light sources with wavelengths of 660nm, 530nm, and 450nm. The near-infrared light source can select near-infrared light with wavelengths of 850nm and 940nm. The far-infrared light source can select far-infrared light of 10-14μm. The tunable multi-wavelength light source can be provided by a tunable laser; the multi-wavelength lasers include laser light sources with wavelengths of 660nm, 850nm, and 940nm, and the multi-wavelength lasers can be provided by a multi-laser array; a multi-modal tunable laser light source composed of multi-wavelength lasers or tunable multi-wavelength light sources can be used for laser speckle blood flow imaging, hyperspectral imaging, and infrared thermal imaging of the stable facial video output by the micro-movement stabilization and correction module to obtain multi-modal optical data including laser speckle images, hyperspectral data, and infrared thermal imaging. The laser speckle image is used to capture the dynamic changes in the micro-arterial blood flow velocity, the hyperspectral data is used to capture the dynamic changes in the micro-arterial oxygen saturation and hemoglobin concentration, and the infrared thermal imaging is used to capture the changes in facial skin temperature to assist in hemodynamic analysis.

[0063] A physiological index calculation module maps blood flow parameters to preliminary blood pressure values, blood oxygen monitoring values, and body temperature monitoring values through physical model driving based on the multi-modal optical data aligned in three-dimensional space-time. Extract the local micro-movement time-series features related to heart rate and respiratory rate from the main frequency signals in the physiological frequency bands of heart rate and respiratory rate through a time-series feature fusion network, and obtain heart rate and respiratory rate monitoring values. Dynamically weighted fusion of the preliminary blood pressure value and the blood pressure value calculated from the local micro-movement time-series features related to heart rate is performed through a spatio-temporal attention mechanism to obtain the dynamically weighted fusion blood pressure monitoring value, and output heart rate, blood pressure, blood oxygen, body temperature, and respiratory rate monitoring values; the physical models include a U-Net blood flow segmentation network, a Diffuse Optical Spectroscopy (DOS) mapping network, and a time-series feature fusion network (Long Short-Term Memory, LSTM).

[0064] The U-Net blood flow segmentation network includes an encoder and a decoder. The multi-modal optical data extracts multi-scale blood flow region features through the encoder and reconstructs the blood flow region mask through the decoder. The encoder is a 5-layer convolutional block, each layer containing Conv2D + BN + ReLU, and the output size is halved layer by layer, which are 1024→512→256→128→64 in sequence. The decoder is a 5-layer transposed convolutional block. The convolutional block contains transposed convolution + skip connection, fuses low-level texture and high-level semantic information, and outputs a blood flow region mask with the same resolution as the input.

[0065] The U-Net blood flow segmentation network also includes an input layer and an output layer. The input layer is used to receive laser speckle images and hyperspectral data with a resolution of ≥1024×768 pixels and the number of channels is the multi-wavelength spectral dimension. The multi-wavelength spectrum includes light with wavelengths of 660nm, 850nm, and 940nm. The output layer generates a binary blood flow region segmentation result through the Sigmoid activation function. The binary blood flow region segmentation result is used to distinguish arterioles, veins, and capillaries, and after extracting the blood flow parameters of each blood flow region, it is output to the diffusion mapping network and the optical spectral mapping network. Among them, the blood flow parameters include: blood flow velocity, blood oxygen saturation SpO2, and hemoglobin concentration.

[0066] The diffuse optical spectral mapping network includes 3 fully connected layers and a ReLU activation layer. The 3 fully connected layers have a neuron structure of 512→256→128. Based on the physical models of multi-modal optical data such as laser speckle images, hyperspectral data, and infrared thermal imaging, it calculates and outputs preliminary physiological indicators such as blood pressure, blood oxygen, and body temperature. Combining a loss function with physical constraints such as the Beer-Lambert law correction term, it simultaneously optimizes the blood flow region segmentation accuracy and the physical rationality of physiological indicator prediction, establishes a mathematical mapping relationship between blood flow parameters and blood pressure, blood oxygen, and body temperature, and maps the blood flow parameters of each segmented blood flow region to preliminary blood pressure monitoring values, blood oxygen monitoring values, and body temperature monitoring values. While improving the blood flow region segmentation accuracy through the loss function with physical constraints, it outputs a blood oxygen saturation value that conforms to physical laws, providing double guarantees for the reliability and interpretability of the medical AI system.

[0067] The temporal feature fusion network includes a bidirectional LSTM layer. The LSTM layer has a structure of 128 units. Based on the periodic analysis of the main frequency signals of physiological frequency bands such as heart rate and respiratory rate extracted from local micro-motion fields such as skin vibration motion fields, it extracts local micro-motion temporal features related to heart rate and respiratory rate respectively, and obtains heart rate monitoring values and respiratory rate monitoring values. It dynamically adjusts the weights of the blood pressure value calculated from the local micro-motion temporal features related to heart rate and the preliminary blood pressure monitoring value through a spatio-temporal attention mechanism to enhance the robustness in complex environments.

[0068] The physiological parameter calculation module further includes a temperature calibration sub-module, which compensates for the temperature drift error in the blood flow parameters through Kalman filtering based on the temperature data collected by the infrared thermal imaging module. The infrared thermal imaging module collects skin temperature data in real time through an infrared camera. The infrared thermal imaging sub-module collects the infrared rays emitted by an object through an infrared lens, focuses them on an infrared detector, and the infrared detector converts the infrared radiation energy into an electrical signal. Then, the signal processing circuit amplifies and filters the electrical signal, and finally the image processing module converts the electrical signal into a visual infrared thermal image, thereby reflecting the temperature distribution on the surface of the object. Embed the TinyML lightweight model in the edge device to directly generate preliminary health warnings such as abnormal heart rate and blood pressure fluctuations. Complete the desensitization processing of the original video on the local device, such as facial blurring, by embedding a hardware encryption chip such as TPM 2.0 in the video acquisition module. Introduce differential privacy noise in the desensitization processing stage: for the encrypted feature vector Add Laplace noise , so that the encrypted feature vector after adding Laplace noise satisfies , where is the noise scale parameter; ensure data irreversibility; only upload the noise-added feature vector to avoid the cloud interaction dependence of federated learning. Desensitize the encrypted feature vector through differential privacy noise, and only upload the noise-added data to avoid the leakage of sensitive information, comply with medical data privacy protection regulations such as HIPAA, and at the same time reduce the dependence on cloud federated learning.

[0069] The system further includes an environment adaptation module, which adjusts the movement of the sensing measurement module through a feedback control algorithm; or adjusts the calculation parameters of the micro-motion stabilization and correction module; or adjusts the calculation parameters of the three-dimensional trajectory reconstruction module; or adjusts the multi-wavelength light source intensity to automatically adapt to different lighting conditions, reduce the influence of external interference on the optical signal quality, and ensure the detection stability. The calculation formula for the multi-wavelength light source intensity is:

[0070] ;

[0071] where is the adjustment coefficient, which is optimized in real time through Kalman filtering; is the adjusted light source intensity; is the light source intensity before adjustment.

[0072] Embodiment 2

[0073] As Figure 2 shown, in an embodiment of the present invention, a multi-modal optical non-contact health monitoring method is provided. The monitoring method includes:

[0074] S1. Synchronously collect the human face video image, TOF depth map, and IMU data of the perception measurement module;

[0075] S2. Perform stabilization processing and motion artifact correction on the collected face video image to obtain a stable face video frame, the final local micro-motion field, the main frequency signal of the heart rate physiological frequency band, and the main frequency signal of the respiratory rate;

[0076] S3. Reconstruct the local micro-motion three-dimensional trajectory based on the two-dimensional displacement time series data in the final local micro-motion field;

[0077] S4. Perform laser speckle blood flow imaging, hyperspectral imaging, and infrared thermal imaging on the stabilized face video image to obtain multi-modal optical data, and align the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the local micro-motion three-dimensional trajectory;

[0078] S5. Based on the multi-modal optical data aligned in three-dimensional space-time, map the blood flow parameters to the preliminary blood pressure value, blood oxygen monitoring value, and body temperature monitoring value through physical model driving. Extract the local micro-motion time series features related to heart rate and respiratory rate from the main frequency signals of the heart rate physiological frequency band and respiratory rate physiological frequency band respectively through the time series feature fusion network to obtain the heart rate and respiratory rate monitoring values. Dynamically weight and fuse the preliminary blood pressure value with the blood pressure value calculated from the local micro-motion time series features related to heart rate through the spatio-temporal attention mechanism to obtain the blood pressure monitoring value;

[0079] S2 performs stabilization and correction of motion artifacts on the collected face video image through the micro-motion stabilization and correction module, as Figure 3 shown. The specific execution of S2 includes S2A~S2J.

[0080] S2A. Input the aligned image pair: Register adjacent frame face video images through affine transformation , ensure the spatial consistency of the two-frame face video image, and eliminate the influence of global motions such as head translation and rotation.

[0081] S2B. Global motion correction: For the face video image after being aligned by S2A, use the SURF algorithm to detect global feature points such as the corners of the eyes and the tip of the nose, and obtain the global motion component by calculating the global affine transformation matrix to eliminate camera jitter or the overall displacement of the patient. Optionally, perform histogram equalization or adaptive gamma correction on each frame of the face video image to reduce the interference of light changes on feature extraction; among them, the parameter of gamma correction is dynamically adjusted, and the pixel value after gamma correction is , where I is the original pixel value. raw is the original pixel value.

[0082] The specific steps of S2B include S2B1 to S2B5:

[0083] S2B1. Detect the global feature points of adjacent-frame facial video images through the SURF algorithm, obtain the matching global feature point pairs. The global feature points include the key points of the head contour and facial features, such as the key points of the eye corners, nose tips, etc. Estimate the overall motion between adjacent-frame facial video images, such as head translation, rotation, or camera jitter. Among them, the saliency of the key points is measured by the determinant of the Hessian matrix. The larger the absolute value of the determinant, the more significant the image structure at this key point, such as high-contrast spots or corners;

[0084] S2B2. Use the RANSAC algorithm to remove the abnormal matching global feature point pairs;

[0085] S2B3. Perform confidence scoring on the remaining matching global feature point pairs after being processed by S2B2 through MobileNetV2, retain the high-confidence matching global feature point pairs, such as retaining the matching global feature point pairs with a confidence score ≥ 0.8, and output a stable set of sparse matching global feature point pairs;

[0086] S2B4. Use the least squares method or SVD decomposition. For the set of sparse matching global feature point pairs output by S2B3, calculate the global affine transformation matrix based on multiple matching global feature point pairs in adjacent-frame facial video images :

[0087] , where are rotation, scaling, and shear parameters; is the translation component, that is, the displacement along the x-axis and y-axis; the third row is fixed as , for homogeneous coordinate transformation;

[0088] S2B5. Use the global affine transformation matrix to predict the global motion components of all global feature points : ;

[0089] Among them, for each global feature point / pixel point in the current facial video image the predicted motion vector of each global feature point / pixel point is obtained through transformation , where ) is the global feature point / pixel point after being transformed by the global affine transformation matrix coordinates.

[0090] For example, input: the matching global feature point pairs in adjacent-frame facial video images ;

[0091] Calculation: Solve by the least squares method such that ;

[0092] Output: Predict the global motion component and the predicted motion vector for each global feature point is .

[0093] S2C, Generate the initial dense motion vector field: Perform pixel-level motion prediction on the face video image aligned by S2A through a lightweight vision TRANSFORMER, and output the two-dimensional motion vector of each pixel , forming a preliminary dense motion vector field , covering all image pixels; specifically, the encoder uses MobileViT as the backbone network, divides the aligned face video image into image patches, extracts global motion features through the self-attention mechanism, adds a lightweight upsampling module in the decoder, such as transposed convolution or sub-pixel convolution, gradually restores the spatial resolution, and outputs a dense motion vector field with the same resolution as the input; finally, the global motion features extracted by the decoder are mapped through a 1x1 convolution to obtain a two-dimensional motion vector , generating a preliminary dense motion vector field ; Embed window attention and shifted window in MobileViT to capture long-range motion dependencies while reducing computational complexity, and model the spatio-temporal correlation between adjacent frames through cross-frame attention to improve the motion vector prediction accuracy.

[0094] S2D, Local region division: Screen the global feature points on the face video image corrected by S2B global motion through the Hessian matrix, and divide the local regions with the screened global feature points as anchor points, such as 16×16 pixel blocks.

[0095] Screen the global feature points on the face video image corrected by S2B global motion through the Hessian matrix, specifically including S2D1~S2D5:

[0096] S2D1, Construct the scale space: Convolve the face video image corrected by S2B global motion with Gaussian kernels of different standard deviations to generate multi-scale images;

[0097] S2D2, Calculate the Hessian matrix: ; where represents the face video image corrected by S2B global motion at point along The convolution result of the second-order Gaussian derivative in the direction; it represents the facial video image after S2B global motion correction at the point along the and direction of the convolution result of the mixed second-order Gaussian derivative; it represents the convolution result of the second-order Gaussian derivative in the direction of the facial video image after S2B global motion correction at the point ; represents the standard deviation of the Gaussian kernel;

[0098] S2D3, calculate the response value: evaluate the saliency of key points through the determinant det(H) of the Hessian matrix calculated by S2D2;

[0099] S2D4, non-maximum suppression: screen local extreme points in the scale space and neighborhood constructed by S2D1, that is, screen out global feature points with response values higher than the threshold τ;

[0100] S2D5, global feature point localization: accurately locate to the sub-pixel level, and eliminate low-contrast or edge response points from the global feature points screened by S2D4.

[0101] S2E, optical flow method refinement: in each local area, use the improved pyramid Lucas-Kanade optical flow method to process the preliminary dense motion vector field to generate a refined local dense motion vector field.

[0102] As Figure 4 shown, S2E specifically includes S2E1~S2E3:

[0103] S2E1, pyramid construction: perform multi-scale downsampling on each local area to construct a pyramid for optimizing optical flow estimation layer by layer from coarse to fine; for example, construct a 3-layer pyramid, and optimize the two-dimensional motion vectors of the local area layer by layer from the low-resolution layer to the high-resolution layer to reduce the computational complexity through coarse-fine layer optimization;

[0104] S2E2, low-resolution layer optimization: use the two-dimensional motion vectors predicted by the lightweight vision TRANSFORMER as the initial value to quickly estimate the large-scale motion of the local area;

[0105] S2E3, high-resolution layer optimization: based on the brightness constancy assumption, solve the optical flow equation and iteratively optimize the two-dimensional motion vectors of each pixel in the local area to refine the micro-motions of each local area, such as skin vibration and chest and abdomen undulation; where is the spatial gradient, is the time gradient, and are the horizontal and vertical displacements of the pixel.

[0106] The improved pyramid Lucas-Kanade optical flow method fuses the output of the lightweight vision TRANSFORMER, predicts the global motion trend through an encoder-decoder structure similar to FlowNet, reduces the number of iterations of the optical flow method in complex motion scenarios, and then uses the preliminary dense motion vector field generated by the lightweight vision TRANSFORMER as the initial value of the optical flow method, replacing the zero initialization of the traditional method, improving the motion estimation accuracy of the local area, and generating a refined local dense motion vector field, that is, the preliminary local dense motion vector field .

[0107] S2F, Abnormal motion rejection: Use the RANSAC algorithm to detect outliers in the preliminary local dense motion vector field, reject abnormal motions such as random noise or non-physiological motions, and retain micro-motions related to physiological signals such as skin vibrations and chest and abdomen fluctuations to obtain a corrected local micro-motion field ; Among them, the number of RANSAC iterations and the inlier error threshold need to be set based on experiments. For example, iterate 500 times and the inlier error threshold is 1 pixel;

[0108] S2G, Superimpose to generate a complete motion field: Superimpose the global motion component with the corrected local micro-motion field to generate a complete motion vector field , and the formula is:

[0109] ;

[0110] S2H, Correct the global motion component: Use the complete motion vector field to verify whether there are residual anomalies in the global motion component . If there are residual anomalies, they will be removed through the RANSAC algorithm to obtain a corrected global motion component ; Further eliminate residual artifacts; the residual anomalies can also reduce the impact of outliers on the model parameters by combining the squared loss for normal data and the absolute loss for outliers; or by calculating the Z-score of the motion vectors and removing outliers that deviate from the mean by more than a threshold, such as removing outliers with Z > 3; or by comparing the optical flow vectors of adjacent frames and removing outliers with significantly inconsistent directions or amplitudes; or by identifying isolated points that do not belong to any dense clusters as outliers based on density clustering; or by detecting outliers through fitting residuals based on motion model verification; or by training a Convolutional Neural Network (CNN) or an Autoencoder to distinguish normal and abnormal motion patterns; the RANSAC algorithm can also be combined with one or more of the above outlier detection methods to further improve the accuracy and adaptability of global motion component correction.

[0111] S2I, Separate the local motion field: Subtract the corrected global motion component from the complete motion field to obtain the final local micro-motion field , Only retain physiological-related motions, such as skin vibration and chest and abdomen undulation;

[0112] ;

[0113] For example, the global motion is a slight right shift of the patient's head, and at the same time, the local micro-motion is the slight vibration of the facial skin due to the heartbeat. The SURFSURF algorithm correction model detects the global feature points: the whole head moves right by 2 pixels and rotates by 1°;

[0114] Obtain the global motion component by performing an affine transformation matrix on the global feature points , and for simplicity of the model, the global affine transformation matrix only including horizontal translation is: ;

[0115] ;

[0116] If the actual motion of a certain pixel, i.e., the complete motion vector field, is , assuming no correction is required for the global motion component, the final local micro-motion field reflects skin vibration.

[0117] S2J, Motion field post-processing: For the final local micro-motion field Perform preprocessing including Kalman filtering and Gaussian filtering to achieve temporal smoothing and spatial smoothing, and suppress high-frequency noise; among them, the state equation of Kalman filtering is a uniform motion model, and the convolution kernel size of Gaussian filtering can be 3×3; then extract the two-dimensional displacement time series data of the physiological frequency band of 0.1-4 Hz of the finally preprocessed local micro-motion field through FFT. The extracted physiological frequency band is required to cover the physiological frequency bands of heart rate and respiratory rate, where the heart rate physiological frequency band is 0.8-3 Hz, and the respiratory rate physiological frequency band is 0.1-0.5 Hz; and use wavelet transform for multi-scale decomposition and hard threshold denoising, and retain the coefficients of the heart rate physiological frequency band and the coefficients of the respiratory rate physiological frequency band; after reconstructing the coefficients of the heart rate physiological frequency band and the coefficients of the respiratory rate physiological frequency band, extract the main frequency signals corresponding to the heart rate and respiratory rate physiological frequency bands through FFT or power spectral density analysis respectively.

[0118] The specific steps of using wavelet transform for multi-scale decomposition and hard threshold denoising are as follows:

[0119] S2J1, Signal preprocessing: Eliminate baseline drift from the two-dimensional displacement data through first-order difference or polynomial fitting;

[0120] S2J2, Discrete wavelet decomposition: Perform multi-scale decomposition on the two-dimensional displacement time series data of the finally preprocessed local micro-motion field, and separate high-frequency noise, physiological frequency band signals, and low-frequency artifacts. Each scale corresponds to a different frequency range; for example: high-frequency scale decomposition, obtain : corresponding to transient noise, that is, high-frequency noise >4 Hz, is the proxy of high-frequency noise, and estimate the global noise level by statistically calculating the median of its absolute value; medium-frequency scale decomposition, obtain : covering the physiological frequency band of 0.5-4 Hz; low-frequency scale decomposition, obtain : corresponding to baseline drift, that is, low-frequency artifacts <0.5 Hz.

[0121] S2J3, Physiological frequency band extraction: Retain the wavelet coefficients of the physiological frequency band, that is, the wavelet coefficients of 0.5-4 Hz, corresponding to the heart rate and respiratory rate , for non-physiological frequency bands, such as high-frequency noise D1, D2 and low-frequency motion artifacts A J Perform hard threshold processing; among them, the hard threshold is adaptively set according to noise estimation, and the general hard threshold , in the present invention, the hard threshold ; set the wavelet coefficients of non-physiological frequency bands to zero, and only retain the wavelet coefficients of the medium-frequency scale, that is, the wavelet coefficients of the heart rate physiological frequency band and the wavelet coefficients of the respiratory rate physiological frequency band;

[0122] S2J4, Signal reconstruction: Reconstruct the retained wavelet coefficients of the heart rate physiological frequency band and the wavelet coefficients of the respiratory rate physiological frequency band into a pure signal;

[0123] S2J5, Physiological signal analysis: The main frequency of the pure signal is extracted through FFT or Power Spectral Density (PSD), such as the main frequency of the heart rate physiological frequency band and the main frequency of the respiratory rate physiological frequency band. The main frequency of the heart rate physiological frequency band is 1.2 Hz. The peak value of the pulse wave is identified by the adaptive threshold method, and the heart rate variability is calculated.

[0124] In addition, the original facial video frame image can be reversely deformed according to the inverse global affine transformation matrix to eliminate the jitter or overall displacement of the camera and output the stabilized facial video frame. , The reverse deformation realizes the spatial alignment of the image sequence through the inverse geometric transformation while retaining the local micro-motion information; the reverse deformation and the wavelet transformation have clear division of labor and jointly improve the robustness of the system.

[0125] Among them, the reverse deformation is the key step to eliminate the global motion artifacts through the inverse geometric transformation. The reverse deformation not only ensures the spatial alignment of the image sequence but also retains the local micro-motion information, providing a reliable data basis for high-precision physiological monitoring.

[0126] In S3, the kinematic model, such as the perspective projection model or the rigid body motion model, is used to map the two-dimensional displacement time series data in the final local micro-motion field to three-dimensional motion, reflecting the three-dimensional dynamic changes of the human body surface or organs; then the Kalman filter is used to smooth the three-dimensional trajectory in time series to suppress high-frequency noise.

[0127] The formula for three-dimensional trajectory reconstruction is:

[0128]

[0129] Among them, is the three-dimensional displacement vector of the local micro-motion, is the horizontal and vertical displacements of the target point cloud in the camera coordinate system; is the depth change of the target point cloud along the optical axis direction, which requires additional constraints or assumptions, such as weak perspective projection; is the camera internal parameter matrix; is the final local micro-motion field the two-dimensional displacement vector in;

[0130] Through the normalized coordinate system mapping:

[0131] ;

[0132] Assume the depth i.e., planar motion or known depth , then the three-dimensional displacement of the local micro-motion is: ,

[0133] If the depth is unknown, by integrating a TOF camera with millimeter-level accuracy, directly obtain the depth changes in the target area, such as the face. ; The specific steps are as follows: Through the internal parameter calibration and external parameter calibration of the TOF sensor, establish the mapping relationship between the depth map and the three-dimensional coordinates of the camera. The internal parameter calibration of the TOF sensor includes focal length and distortion parameters, and the external parameter calibration of the TOF sensor includes the positional relationship with the RGB camera.

[0134] S3 specifically includes S3A to S3C.

[0135] S3A: Multi-modal data synchronous acquisition: Obtain the synchronously acquired TOF depth map, facial video image, and IMU data.

[0136] S3B: Point cloud preprocessing: Use statistical outlier removal and radius filtering to eliminate the noise points in the TOF depth map data and generate the depth map of the human face area; Combine the facial video image, and use Mask R-CNN to segment the depth map of the human face area to obtain the depth map focusing on physiological micro-movements.

[0137] S3C: Motion compensation and trajectory generation: Register the point cloud data of the depth map focusing on physiological micro-movements through the Iterative Closest Point (ICP) algorithm, separate the overall head movement such as rigid body transformation and local micro-movement such as skin vibration, and determine the target area from the separated local micro-movement point cloud data; Generate the three-dimensional trajectory of the local micro-movement according to the mapping relationship between the depth map of the target area and the three-dimensional coordinates of the camera established by the internal parameter calibration and external parameter calibration of the TOF sensor.

[0138] Depth changes in the target area are much smaller than its distance to the camera , approximated as planar motion,

[0139] At this time, the three-dimensional displacement is simplified to: ;

[0140] Depth changes in the target area are much smaller than its distance to the camera , in typical scenarios , can be approximated as planar motion. Based on the weak perspective projection model, the three-dimensional displacement is simplified to: ;

[0141] Among them, is the average depth measured by the TOF sensor, is the camera internal parameter matrix, is the two-dimensional displacement vector in the corrected local micro-motion field; The relative error of this simplified model is less than 0.1%, meeting the medical-grade accuracy requirements.

[0142] In the present invention, two-dimensional optical flow is mapped into three-dimensional displacement through a kinematic model, and the three-dimensional trajectory of local micro-movement after separating the overall movement of the head is smoothed in time series by combining Kalman filtering, high-frequency noise is suppressed, spatial dynamic information is provided, the analysis accuracy and robustness of physiological signals such as skin vibration and chest and abdomen undulation are significantly improved, and a unified spatio-temporal reference framework is provided for multi-modal data fusion.

[0143] The process of mapping blood flow parameters into preliminary blood pressure values, blood oxygen monitoring values, and body temperature monitoring values through physical model driving based on multi-modal optical data aligned in three-dimensional space-time in S5 is as follows:

[0144] Data input: The blood flow parameters of each blood flow region extracted from the U-Net blood flow segmentation network are standardized to obtain blood flow parameter samples, and the blood flow parameter samples are input into a three-layer fully connected network;

[0145] Model output: The three-layer fully connected network outputs preliminary physiological indicators such as systolic blood pressure , diastolic blood pressure , blood oxygen , body temperature ;

[0146] Loss calculation:

[0147] Data loss: Calculate the mean square error between the predicted value of the physiological indicator obtained from the i-th blood flow parameter sample and the true label as; ;

[0148] ;

[0149] Physical constraint loss: Based on the Beer-Lambert law, constrain the predicted value of blood oxygen saturation to be consistent with the ratio of red light / near-infrared light reflection intensity , and calculate the physical constraint loss function based on the Beer-Lambert law as:

[0150]

[0151] Among them, and are calibration parameters to force the network output to conform to the optical absorption law; N represents the total number of data samples;

[0152] Parameter optimization: For the mathematical mapping relationship that maps blood flow parameters into blood oxygen saturation, minimize the total loss through a gradient descent algorithm such as Adam Perform parameter fine-tuning. For the mathematical mapping relationship that maps blood flow parameters to blood pressure and body temperature, perform parameter fine-tuning according to the data loss to ensure that the output of the diffuse optical spectroscopy mapping network conforms to the physical model.

[0153] Based on the physical model of laser speckle images, the blood flow velocity , pulse wave conduction time are mapped to the preliminary blood pressure value BP:

[0154] ;

[0155] where are all regression coefficients calibrated through clinical data.

[0156] Based on the physical model of hyperspectral data, the red light / near-infrared light reflectance ratio is mapped to the blood oxygen saturation monitoring value :

[0157] ;

[0158] where are all parameters determined through the calibration curve.

[0159] Based on the physical model of infrared thermal imaging, the blood flow velocity , facial skin temperature are mapped to the body temperature monitoring value :

[0160] ;

[0161] where is the facial skin temperature, is the correction coefficient, is the reference blood flow velocity.

[0162] In S5, through the time series feature fusion network, local micro-motion time series features related to heart rate and respiratory rate are respectively extracted from the main frequency signals of the heart rate physiological frequency band and the respiratory rate physiological frequency band to obtain heart rate and respiratory rate monitoring values. Specifically:

[0163] The main frequency of the heart rate physiological frequency band obtained from the final local micro-motion field and the main frequency of the respiratory rate physiological frequency band are input into the LSTM, predicted by the LSTM, and the heart rate and the respiratory rate are extracted:

[0164] ;

[0165] ;

[0166] Among them, the main frequency of the respiratory rate physiological frequency band is the two-dimensional motion displacement in the final local micro-motion field Perform wavelet transform and extract the dominant frequency in the 0.1-0.5 Hz frequency band.

[0167] In S5, the preliminary blood pressure value and the blood pressure value calculated by the local micro-motion temporal features related to heart rate are dynamically weighted and fused through the spatiotemporal attention mechanism to obtain the blood pressure monitoring value. Specifically:

[0168] Dynamically adjust the weights of blood pressure values calculated from local micro-motion temporal features related to heart rate and preliminary blood pressure values through the spatiotemporal attention mechanism 、 , enhancing robustness in complex environments; the spatiotemporal attention mechanism analyzes the temporal changes of physiological indicators in the temporal dimension. By dynamically weighting the confidence of data from different modalities through the attention mechanism, multimodal data fusion is performed to achieve accurate analysis of the temporal changes of physiological indicators. Combined with the adaptive weighting of preliminary physiological indicators, the prediction accuracy and stability of the non-contact health monitoring system are ultimately improved; the temporal changes of physiological indicators are analyzed in the temporal dimension; specifically: periodic capture: modeling rhythmic changes in heartbeat and breathing; trend tracking: identifying long-term fluctuations in blood pressure and blood oxygen; noise suppression: reducing the weight of interfering time steps. This significantly improves the reliability and accuracy of non-contact health monitoring in complex environments such as motion and lighting changes.

[0169] According to the dynamically adjusted weight , weighted systolic blood pressure value calculated from the local micro-motion temporal characteristics related to heart rate and initial systolic blood pressure ,, optimize the blood pressure estimation value through Kalman filtering, and obtain the real-time prediction value of systolic blood pressure, that is, the systolic blood pressure monitoring value The dynamic weighted fusion method of diastolic pressure is the same as that of systolic pressure, and will not be described in detail in the present invention.

[0170] ;

[0171] Among them, pulse wave transmission time Obtained based on the main frequency signal of the heart rate physiological frequency band.

[0172] During initial use, the user measures baseline values using a medical device. , the system predicts the blood pressure in real time Dynamically adjust weights based on differences from baseline values , output more accurate final blood pressure :

[0173] ;

[0174] Among them, represents the final blood pressure value after error compensation; represents the user's personalized weight coefficient; represents the blood pressure value directly predicted by an algorithm such as a diffuse optical spectroscopy mapping network; represents the user's baseline blood pressure value, i.e., the initial calibration value; The present invention introduces physical constraints such as the Beer-Lambert law into the diffuse optical spectroscopy mapping network, constrains the relationship between the predicted value of blood oxygen saturation and the optical reflection intensity, enhances the physical interpretability of the model, ensures that the output result conforms to medical laws, and improves the doctor's trust in the system output. Clinical verification: Synchronously collect data with a gold standard device, optimize the regression coefficient through cross-validation, and ensure that the heart rate error ≤ ±2 BPM, blood pressure error ≤ ±5 mmHg, and SpO2 error ≤ ±2%.

[0175] To verify the effect of the present invention, the blood pressure measurement error performance of the traditional method and the present invention in two dynamic scenarios was compared, and the specific data are shown in Table 1.

[0176] Table 1

[0177] Scene Error of traditional method (mmHg) Error of the present invention (mmHg) Horizontal movement of the head ±8.2 ±3.5 Light mutation (2000 lux) ±12.1 ±4.8

[0178] As can be seen from Table 1, in the scenario of horizontal head movement, the error of the traditional method is ±8.2 mmHg, while the error of the present invention is only ±3.5 mmHg; in the scenario of sudden light change (2000 lux), the error of the traditional method reaches ±12.1 mmHg, and the error of the present invention is ±4.8 mmHg. It can be seen that in these two dynamic scenarios, the present invention has obvious advantages in error control compared with the traditional method, and the measurement result is more accurate.

[0179] The present invention combines multi-modal optical data such as laser speckle imaging, hyperspectral data, and infrared imaging data to synchronously obtain multi-dimensional physiological indicators such as blood pressure, heart rate, and blood oxygen saturation, overcomes the limitations of traditional single-modalities such as only PPG or VPG, meets the needs of multi-parameter joint analysis in complex clinical scenarios, and improves the comprehensiveness of the detection dimension. Through multi-level corrections such as the SURF algorithm, RANSAC outlier rejection, and Kalman filtering, the global motion and local micro-motion are separated, and physiological signals such as heart rate and respiratory rate can still be accurately extracted in the scenario of slight patient movement or camera jitter.

[0180] The present invention preferentially processes optical imaging data through a U-Net network and then fuses it with video micro-motion features to solve the problem of multi-modal data coupling interference in traditional methods. A mathematical mapping relationship between blood flow parameters and physiological indicators is established through a diffuse optical spectroscopy mapping network, and the segmented blood flow parameters such as blood flow velocity, blood oxygen saturation SpO2, and hemoglobin concentration are mapped to preliminary physiological indicators such as blood pressure, blood oxygen, and body temperature. The time series features of heart rate and respiratory rate extracted by the time series feature fusion network are used to solve the problem of time lag in single-modal data; then, the spatio-temporal attention mechanism dynamically adjusts the weights of the blood pressure value calculated based on the local micro-motion time series features related to heart rate and the preliminary blood pressure mapping value, enhancing the robustness in complex environments and improving the prediction accuracy and stability of the non-contact health monitoring system.

[0181] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

Claims

1. A multimodal optical non-contact health monitoring system, characterized in that, Including: A perception measurement module for acquiring facial video images, TOF depth maps, and IMU data of a human body; A data synchronous acquisition module for synchronously acquiring facial video images, TOF depth maps, and IMU data of a human body; A micro-motion stabilization and correction module for stabilizing and correcting motion artifacts of the synchronously acquired facial video images, and outputting stable facial video frames, final local micro-motion fields, main frequency signals in the physiological frequency band of heart rate, and main frequency signals in the physiological frequency band of respiratory rate; A three-dimensional trajectory reconstruction module for reconstructing a local micro-motion three-dimensional trajectory according to the two-dimensional displacement time series data in the final local micro-motion field; A multi-modal data processing module for optically processing the stable facial video frames to obtain multi-modal optical data, and aligning the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the local micro-motion three-dimensional trajectory; and A physiological index calculation module for mapping blood flow parameters to preliminary blood pressure values, blood oxygen monitoring values, and body temperature monitoring values through physical model driving based on the multi-modal optical data aligned in three-dimensional space-time, respectively extracting local micro-motion time series features related to heart rate and respiratory rate from the main frequency signals in the physiological frequency bands of heart rate and respiratory rate, obtaining heart rate and respiratory rate monitoring values, dynamically weighted fusing the preliminary blood pressure value with the blood pressure value calculated from the local micro-motion time series features related to heart rate, obtaining blood pressure monitoring values, and outputting heart rate, blood pressure, blood oxygen, body temperature, and respiratory rate monitoring values.

2. The multimodal optical non-contact health monitoring system according to claim 1, wherein The perception measurement module includes: an RGB camera for acquiring facial video images of a human body, a time-of-flight sensor for acquiring TOF depth maps of the human face, and an inertial measurement unit for acquiring IMU data; the data synchronous acquisition module acquires facial video images, TOF depth maps, and IMU data of a human body through hardware synchronization technology.

3. The multimodal optical non-contact health monitoring system according to claim 2, characterized in that, The multi-modal optical module uses a tunable multi-wavelength light source or a multi-wavelength laser for obtaining multi-modal optical data including but not limited to laser speckle images, hyperspectral data, and infrared thermal imaging.

4. The multimodal optical non-contact health monitoring system according to claim 3, wherein The micro-motion stabilization and correction module includes: A stable image unit for correcting the facial video frame image through a SURF algorithm correction model and outputting a stable facial video frame; and An artifact correction unit for performing pixel-level two-dimensional motion vector prediction on the facial video frame image through a lightweight vision TRANSFORMER to generate a preliminary dense motion vector field, generating a preliminary local dense motion vector field through an improved pyramid Lucas-Kanade optical flow method, correcting the preliminary local dense motion vector field to obtain a corrected local micro-motion field, generating a complete motion field according to the corrected local micro-motion field and performing global motion component correction, and then obtaining the final local micro-motion field from the corrected global motion component, and respectively extracting the main frequency signals in the physiological frequency bands of heart rate and respiratory rate from the final local micro-motion field.

5. The multimodal optical non-contact health monitoring system according to claim 4, characterized in that, The physiological index calculation module includes: The U-Net blood flow segmentation network is used to extract multi-scale blood flow region features from the multi-modal optical data, reconstruct the blood flow region mask, generate a binary blood flow region segmentation result, and extract blood flow parameters of each blood flow region. The blood flow parameters include, but are not limited to, blood flow velocity, blood oxygen saturation, and hemoglobin concentration; The diffuse optical spectroscopy mapping network is used to establish a mathematical mapping relationship that maps blood flow parameters to preliminary physiological indicators, and outputs a preliminary blood pressure value, a blood oxygen monitoring value, and a body temperature monitoring value; and, The temporal feature fusion network extracts local micro-motion temporal features related to heart rate and respiratory rate from the main frequency signals in the physiological frequency bands of heart rate and respiratory rate through the temporal feature fusion network, and dynamically weights and fuses the preliminary blood pressure value with the blood pressure value calculated from the local micro-motion temporal features related to heart rate through the spatio-temporal attention mechanism to obtain the blood pressure monitoring value.

6. The multi-modal non-contact health monitoring system according to claim 5, wherein, The system further includes an environment adaptation module, and the environment adaptation module adjusts the movement of the sensing measurement module through a feedback control algorithm; or adjusts the calculation parameters of the micro-motion stabilization and correction module; or adjusts the calculation parameters of the three-dimensional trajectory reconstruction module; or adjusts the intensity of the multi-wavelength light source; The physiological index calculation module further includes a temperature calibration sub-module, and the temperature calibration sub-module compensates for the temperature drift error in the blood flow parameters through Kalman filtering based on the temperature data of the infrared thermal imaging.

7. Multimodal optical non-contact health monitoring method, characterized in that Implemented based on the system according to any one of claims 1 to 6, including the following steps: S1, synchronously collect the human face video image, the TOF depth map, and the IMU data of the sensing measurement module; S2, perform stabilization processing and motion artifact correction on the collected face video image to obtain a stable face video frame, the final local micro-motion field, the main frequency signal in the heart rate physiological frequency band, and the main frequency signal in the respiratory rate physiological frequency band; S3, reconstruct the local micro-motion three-dimensional trajectory according to the two-dimensional displacement time series data in the final local micro-motion field; S4, perform laser speckle blood flow imaging, hyperspectral imaging, and infrared thermal imaging on the stabilized face video image to obtain multi-modal optical data, and align the multi-modal optical data according to the three-dimensional spatio-temporal reference frame of the local micro-motion three-dimensional trajectory; S5, based on the multi-modal optical data aligned in three-dimensional space-time, map the blood flow parameters to a preliminary blood pressure value, a blood oxygen monitoring value, and a body temperature monitoring value through physical model driving, extract local micro-motion temporal features related to heart rate and respiratory rate from the main frequency signals in the heart rate physiological frequency band and the respiratory rate physiological frequency band respectively through temporal feature fusion technology to obtain heart rate and respiratory rate monitoring values, and dynamically weight and fuse the preliminary blood pressure value with the blood pressure value calculated from the local micro-motion temporal features related to heart rate through the spatio-temporal attention mechanism to obtain the blood pressure monitoring value.

8. The multimodal optical non-contact health monitoring method according to claim 7, characterized in that The specific steps of S2 include the following steps: S2A, register adjacent frame face video images through affine transformation, and perform alignment processing on the input face video frame image; S2B. For the facial video image after S2A alignment processing, use the SURF algorithm to detect global feature points, obtain the global motion component by calculating the global affine transformation matrix, perform a geometric inverse transformation on the input facial video frame image according to the global affine transformation matrix, and output a stable facial video frame; S2C. Use a lightweight vision TRANSFORMER to perform pixel-level two-dimensional motion vector prediction on the facial video image after S2A alignment processing to generate a preliminary dense motion vector field; S2D. Screen the global feature points on the facial video image after S2B global motion correction through the Hessian matrix, and divide the local area with the screened global feature points as the anchor points; S2E. Use an improved pyramid Lucas-Kanade optical flow method to process the preliminary dense motion vector field to generate a preliminary local dense motion vector field; S2F. Use the RANSAC algorithm to detect outliers in the preliminary local dense motion vector field, remove abnormal motions, and obtain a corrected local micro motion field; S2G. Superimpose the global motion component and the corrected local micro motion field to generate a complete motion vector field; S2H. Use the complete motion vector field to verify whether there are residual anomalies in the global motion component. If there are residual anomalies, remove them through the RANSAC algorithm to obtain a corrected global motion component; S2I. Subtract the corrected global motion component from the complete motion field to obtain the final local micro motion field; S2J. Preprocess the final local micro motion field, extract the two-dimensional displacement time series data in the physiological frequency band of the preprocessed final local micro motion field, and perform multi-scale decomposition and hard threshold denoising using wavelet transform. Retain the heart rate physiological frequency band coefficient and the respiratory frequency physiological frequency band coefficient, and extract the main frequencies of the heart rate and respiratory frequency physiological frequency bands after reconstructing the heart rate physiological frequency band coefficient and the respiratory frequency physiological frequency band coefficient.

9. The multimodal optical non-contact health monitoring method according to claim 8, wherein The specific steps of S2E are as follows: S2E1. Perform multi-scale downsampling on each local area to construct a pyramid for optimizing optical flow estimation layer by layer from low resolution to high resolution; S2E2. Use the preliminary dense motion vector field generated by S2C as the initial value to quickly estimate the motion range of the local area and perform low-resolution layer optimization; S2E3. Based on the assumption of brightness constancy, solve the optical flow equation to iteratively optimize the two-dimensional motion vector of each pixel in the local area and perform high-resolution layer optimization.

10. The multimodal optical non-contact health monitoring method according to claim 9, characterized in that, S3 reconstructs the local micro motion three-dimensional trajectory according to the two-dimensional displacement time series data in the final local micro motion field. Specifically: S3A. Obtain the simultaneously collected TOF depth map, the facial video image of the human body, and the IMU data; S3B. Use statistical outlier removal and radius filtering to eliminate the noise points in the TOF depth map data, generate the depth map of the human facial area, combine it with the facial video image, and use Mask R-CNN to segment the depth map of the human facial area to obtain the depth map focusing on physiological micro motions; S3C registers the point cloud data of the depth map that focuses on physiological micro-movements through the Iterative Closest Point (ICP) algorithm, separates the overall head movement from the local micro-movements, determines the target area from the separated local micro-movement point cloud data, and generates the three-dimensional trajectory of the local micro-movement according to the mapping relationship between the depth map of the target area and the three-dimensional coordinates of the facial video image established by the calibration of the internal parameters and external parameters of the TOF sensor.

Images