Human health condition multi-mode real-time monitoring system and device

Abstract

The invention relates to the field of health condition monitoring, in particular to a human body health condition multi-mode real-time monitoring system and device. The system comprises an acquisition module configured to acquire multi-modal data in real time; the pre-processing module is configured to perform pre-processing based on the acquired multi-modal data, and the pre-processing comprises performing fusion processing on the data of each flexible pressure sensor; and the health state discrimination module is configured to obtain fusion features based on the modal data by using the trained deep learning-based multi-modal neural network based on the preprocessed multi-modal data, and then perform health state discrimination based on the fusion features. According to the technical scheme, non-contact, full-time and multi-azimuth accurate and reliable monitoring of human health medical key information in various living environments can be achieved, the requirements of various scenes such as homes, hospitals and outdoors are met, and powerful technical support is provided for early warning of serious diseases.

Description

Technical Field

[0001] The present disclosure relates to the field of health status monitoring, and in particular to a multimodal real-time monitoring system and method for human health status. Background Art

[0002] In recent years, with the increasing aging of society, the incidence of chronic diseases or major illnesses such as hypertension, stroke, and coronary heart disease has continued to rise. Real-time monitoring and anomaly warning of key health conditions, such as heart rate, respiratory rate, gait, and body posture, have gradually become an important component of healthcare management. Traditional monitoring methods rely primarily on wearable devices (such as Holter monitors and smart bracelets) and video camera systems. While wearable devices can directly collect information such as heart rate, respiratory rate, and gait, they require long-term, close-fitting wear. Furthermore, the sensors are made of relatively rigid materials, which can cause discomfort and poor user compliance. Furthermore, data collection stability and accuracy are limited by factors such as wearing position, fit, and electrode interference. While video camera technology can capture and analyze human posture through computer vision, enabling non-contact monitoring, it is susceptible to interference from ambient light and occlusion in practical applications, resulting in discontinuous or distorted monitoring data. Therefore, ensuring accurate collection of human health data while improving user compliance and the accuracy and reliability of monitoring data has become a pressing challenge in current health monitoring technology.

[0003] Existing technical solutions fall into two main categories: single-sensor monitoring solutions and multi-sensor data fusion solutions. Single-sensor solutions (such as wearable ECG monitors and video surveillance systems) are effective in different application scenarios. In recent years, some research and applications have begun exploring multi-sensor data fusion solutions, such as combining millimeter-wave radar, computer vision, and flexible electronic sensors. Certain patents (such as "CN118512157A: A Multi-Source Sensorless Monitoring System and Device Thereof" and "CN115089157A: A Contactless Home Elderly Care Monitoring System and Method") propose using millimeter-wave radar in conjunction with video cameras and thermal infrared cameras to monitor heart rate, respiratory rate, and body temperature.

[0004] Although existing technologies have made great progress in the field of human health monitoring, they still have some obvious shortcomings in meeting the needs of continuous, non-contact, and high-precision monitoring: First, wearable device-based monitoring solutions require users to wear Holter monitors, smart bracelets, and other sensors for extended periods of time. This lack of comfort leads to high user abandonment rates, and it's difficult to maintain a consistent and stable fit during daily activities or sleep. This makes it difficult to maintain consistent data quality over time, reducing monitoring continuity and accuracy.

[0005] Secondly, while non-contact monitoring methods based on video cameras avoid the discomfort of wearing a device directly, their effectiveness depends heavily on environmental conditions. Existing monitoring technologies struggle to meet the demand for real-time, uninterrupted, comprehensive health monitoring. Millimeter-wave radars can suffer from large measurement errors when obscured or when a large number of people are present. Video cameras are easily affected by lighting and pose privacy concerns. Smart mattresses can only monitor a person's health while in bed, limiting their scope. Furthermore, existing multimodal monitoring technologies can only monitor a few indicators, such as heart rate and respiratory rate, making it difficult to reliably monitor key physiological indicators such as body movement and sleep movements, as well as emergency health incidents like sleep apnea and nighttime falls. Summary of the Invention

[0006] To address the problem that existing human health monitoring methods are unable to comprehensively and accurately monitor vital signs and behavioral activity information such as heart rate, respiration, sleep, and gait, this disclosure proposes a multimodal real-time monitoring system and method for human health based on multiple sensing technologies such as millimeter-wave radar, flexible electronics, and computer vision. Through the organic integration and innovation of technical solutions, it achieves high-precision, non-contact, full-time monitoring of key indicators such as human heart rate, respiratory rate, walking gait, and body posture, and can issue timely warnings when abnormalities occur. The specific technical solution is as follows.

[0007] In the first aspect, the present disclosure proposes a multimodal real-time monitoring system for health status, which includes an acquisition module, a preprocessing module, and a health status judgment module: the acquisition module is configured to acquire multimodal data in real time, and the multimodal data includes millimeter-wave radar echo data, computer vision image data, and various flexible pressure sensor data; the preprocessing module is configured to perform preprocessing based on the acquired multimodal data, and the preprocessing includes fusing the data of various flexible pressure sensors; the health status judgment module is configured to obtain fusion features based on various modal data based on the preprocessed multimodal data using a trained multimodal neural network based on deep learning, and then judge the health status based on the fusion features; the health status includes vital sign information and behavioral activity posture monitoring, the vital sign information includes heart rate and respiratory rate, and the behavioral activity posture includes body balance, gait, sitting posture, sleep movement, and falls.

[0008] In a second aspect, the present disclosure provides a computer-readable storage medium, characterized in that it stores a computer program that can be loaded by a processor and execute any system of the present disclosure.

[0009] In a third aspect, the present disclosure proposes a multimodal real-time monitoring device for human health status, the device comprising a millimeter-wave radar module, a computer vision module, a flexible pressure sensing module and a central processing unit; the millimeter-wave radar module is configured to acquire millimeter-wave radar echo data in real time; the computer vision module is configured to acquire computer vision image data in real time; the flexible pressure sensing module is configured to acquire flexible pressure sensing data in real time; the central processing unit is configured to use the millimeter-wave radar echo data, the computer vision data and each flexible pressure sensing data as multimodal data, and perform preprocessing based on the multimodal data, the preprocessing comprising fusing the data of each flexible pressure sensor; the preprocessed multimodal data is used to obtain fusion features based on each modal data using a trained multimodal neural network based on deep learning, and then health status is judged based on the fusion features; wherein the health status comprises vital sign information and behavioral activity posture monitoring, the vital sign information comprises heart rate and respiratory rate, and the behavioral activity posture comprises body balance, gait, sitting posture, sleep movement and falls.

[0010] The beneficial technical effects of the present disclosure are: comprehensive use of millimeter-wave radar, computer vision and flexible pressure sensing technology, and full combination of the advantages of these technologies, including: millimeter-wave radar can detect human walking gait and body posture in the absence of natural light without contact, and can also be used for real-time monitoring of heart rate and respiratory rate; when the radar or video acquisition equipment is blocked by an object, mattresses, seat cushions, insoles and other smart terminals equipped with flexible pressure sensors can be used to collect and identify the user's body posture or walking gait information. The built-in flexible materials avoid direct contact with the user, and the collection process is non-sensing and non-intrusive. By integrating the comprehensive and complementary applications of the above-mentioned sensing technologies and further improving the monitoring accuracy through multi-modal data fusion analysis and processing, non-contact, full-time, and multi-directional accurate and reliable monitoring of key human health and medical information can be achieved in various living environments, meeting the needs of various scenarios such as home, hospital and outdoor, and providing strong technical support for early warning of major diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 , Schematic diagram of a multimodal real-time monitoring device for human health status.

[0013] Figure 2 , flow chart of the multimodal real-time monitoring system for human health status. DETAILED DESCRIPTION

[0014] This solution applies multimodal non-contact real-time monitoring and data fusion technology to the real-time and uninterrupted monitoring of human health status, organically integrating millimeter-wave radar, computer vision and flexible electronic sensing technologies. Through the complementary advantages of their respective technologies, it can achieve the precise collection and analysis of key information such as human heart rate, respiratory rate, gait and posture. It can comprehensively monitor vital signs such as human heart rate and respiratory rate, as well as normal or abnormal behavioral activities such as body balance, gait, sitting posture, sleep movements, and falls in real time. It can promptly and accurately detect and issue alarms when emergency abnormal conditions such as sleep apnea and falls occur.

[0015] The following is a clear and complete description of how the technical solution of this case comprehensively utilizes millimeter-wave radar, high-definition video acquisition equipment, and smart terminals embedded with flexible pressure sensors (such as mattresses, seat cushions, insoles, etc.) to form a unified multimodal real-time monitoring system for human health status, achieving real-time, non-contact, full-time and space-time monitoring of multiple indicators such as human heart rate, respiratory rate, walking gait, and body posture, and promptly triggering early warnings when abnormalities occur. Obviously, the described implementation methods are only part of the implementation methods of this case, not all of them. Based on the implementation methods in this case, all other implementation methods obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] The multimodal real-time monitoring system for human health status adopts a layered architecture design, consisting of a front-end data acquisition layer, a data processing layer, and a back-end cloud platform management layer.

[0017] The front-end data acquisition layer consists of three modules: millimeter-wave radar module, computer vision module and flexible pressure sensing module. These modules work together to collect human physiological and motion data.

[0018] The data processing layer sends the raw data collected by each module to the central processing unit through a high-speed data bus and a low-latency communication interface for data synchronization, preprocessing and multimodal data fusion.

[0019] The back-end cloud platform management layer is used for data storage, historical data comparison, real-time warning information push, remote monitoring and expert intervention.

[0020] The above architectural design not only ensures the continuity of data collection, but also provides stable support for subsequent data processing and cloud-based intelligent analysis.

[0021] See also Figure 1The multimodal health monitoring system includes a millimeter-wave radar module, a computer vision module, a flexible pressure sensing module, a central processing unit (CPU), a communication module, an anomaly detection and alarm module, and a power supply structure. The millimeter-wave radar module, computer vision module, and flexible pressure sensing module are front-end sensing modules that collect and transmit data with the CPU. The CPU utilizes a trained deep learning-based multimodal neural network to acquire vital sign information and monitor behavioral activity and posture based on the millimeter-wave radar echo data collected by the millimeter-wave radar module, the computer vision image data collected by the computer vision module, and the flexible pressure sensing data collected by the flexible pressure sensing module. This deep learning-based multimodal neural network incorporates historical data comparison and online learning mechanisms. When a modality is abnormal or missing, the weight of other modal data is automatically increased, achieving error correction and ensuring the continuity and stability of overall monitoring. Furthermore, a power supply interface provides power to each module. The anomaly detection and alarm module issues an alarm when the CPU detects an abnormality, and transmits it to the user via the communication module.

[0022] The millimeter-wave radar module is a terminal that uses millimeter-wave radar to detect vital signs and behavioral activity. It enables contactless detection of the human body without natural lighting, capturing real-time motion status, gait information, and heart and respiratory rate data. Millimeter-wave signals have high penetration and resolution, effectively penetrating obstructions such as clothing and curtains, ensuring continuous and accurate data collection.

[0023] Specifically, see Figure 2 The main components of the millimeter-wave radar module include an FM transmitter, a receiver, and a signal microprocessor. The FM transmitter system consists of four parallel transmission chains, each with independent phase and amplitude control. Using a frequency-modulated continuous wave (FMCW) millimeter-wave radar, the transmitted wave frequency is 60 GHz. After reaching the human body, the echo signal is received by the receiving antenna and then mixed with the transmitted signal to generate an analog intermediate frequency (IF). After low-pass filtering and ADC sampling, it is sent to the signal processing microcontroller, which encapsulates the radar echo digital signal and sends it to the central processing unit.

[0024] Millimeter-wave radar can continuously scan the human body in a non-contact state, collecting human health parameters such as heart rate, respiratory rate, gait and body posture. It has high resolution and penetrating ability, can penetrate clothing and some obstructions, and can work stably even in insufficient natural light or at night.

[0025] Therefore, millimeter-wave radar can be used to extract vital signs and behavioral activity features. The vital sign information includes heart rate and respiratory rate. The behavioral activity posture monitoring information includes information features such as gait rhythm, body posture deviation, and falls.

[0026] The computer vision module collects video images from high-definition cameras installed in various locations indoors and uses deep learning and image processing algorithms to extract behavioral features, enabling recognition and analysis of gestures. Under favorable environmental conditions, this module provides auxiliary data to correct and supplement millimeter-wave radar data.

[0027] Specifically, see Figure 2 , multiple high-definition cameras are arranged at different angles and positions in the indoor environment, and the characteristics of human behavior activities are extracted through video acquisition. The behavior activity characteristics contain information such as motion trajectory, activity posture and skeleton key points.

[0028] Equipped with a deep learning-based image processing algorithm, the module can accurately identify human targets in complex backgrounds and perform multi-angle data fusion. By comparing and analyzing video images and millimeter-wave radar data, monitoring accuracy is further improved.

[0029] To address the limitations of millimeter-wave radar and vision equipment in obstructed or specialized environments, this system incorporates intelligent devices embedded with flexible pressure sensors, such as smart mattresses, seat cushions, and insoles. Made of flexible materials, these sensors not only ensure long-term user comfort but also efficiently capture human posture and gait information by collecting real-time pressure distribution information across contact areas. These sensors can be used to monitor respiratory rate, heart rate, sleep movements, gait, and other data in different states (lying down, sitting, and walking).

[0030] Therefore, the flexible pressure sensing module can also extract vital signs and behavioral activity features. This data can be used as auxiliary information to effectively complement the primary data collected. In other words, when the millimeter-wave radar or video equipment is obstructed, the flexible pressure sensing module can serve as a backup monitoring device to ensure the continuity of monitoring data.

[0031] The central processing unit (CPU) serves as the data processing core of the system, providing a reliable platform for the operation of algorithms such as data synchronization acquisition and preprocessing, feature extraction, multimodal fusion, and health anomaly monitoring. The CPU can be composed of a high-performance computer or server.

[0032] Each data acquisition module obtains human body information from different dimensions: the millimeter-wave radar module obtains tiny changes in human body movement through continuous scanning of millimeter waves, converts them into digital signals and transmits them to the central processing unit; the computer vision module captures video frames, and after real-time pre-processing of the collected image data, extracts the key points of the human skeleton, activity posture and movement trajectory; the flexible pressure sensing module uses the built-in flexible pressure sensor to convert the pressure distribution data of the human body contact area into electrical signals, and uploads them to the central processing unit after preliminary filtering processing.

[0033] To ensure consistency and timing synchronization of data collected by each module, the central processing unit uses a unified timestamp mechanism to synchronously preprocess millimeter-wave radar data, video data, and pressure data. Millimeter-wave data is filtered and echo signal extracted using digital signal processing (DSP). Video data is processed using background subtraction and image enhancement algorithms to suppress background noise and enhance the signal-to-noise ratio. Flexible electronic data undergoes low-pass filtering and signal smoothing to eliminate minor jitter and interference during acquisition. The purpose of data preprocessing is to provide high-quality, low-noise raw input for subsequent multimodal data fusion, ensuring the stability and accuracy of the overall monitoring system. This preprocessing includes fusing the data from each flexible pressure sensor. See the Data Fusion Analysis section below for details.

[0034] This component uses a multimodal data fusion algorithm to integrate and process data collected by millimeter-wave radar, computer vision, and flexible pressure sensing modules. Leveraging artificial intelligence, big data analysis, and deep learning technologies, it preprocesses data, extracts features, and detects anomalies, enabling accurate assessment of human health status.

[0035] This part uses a modular data fusion algorithm to integrate and process three types of data collected by millimeter-wave radar, computer vision, and flexible sensors. It utilizes artificial intelligence, big data analysis, and deep learning technologies to preprocess, extract features, and detect anomalies, enabling accurate assessment of human health status. By deeply fusing spatiotemporal data using weighted averaging, Kalman filtering, and Bayesian reasoning, and then using a neural network model for adaptive weight adjustment, accurate monitoring results are maintained despite environmental changes or sensor failures. This approach leverages the strengths of each method to compensate for potential limitations of a single sensing method.

[0036] Each module's data is first extracted using independent algorithms. The millimeter-wave radar module is used to extract heart rate, respiratory rate, and motion characteristics. The main ideas are: When the human body performs physiological activities such as heartbeat and breathing, the chest cavity will fluctuate slightly at the millimeter or sub-millimeter level. The micro-movement causes the frequency and phase of the frequency modulated continuous (FMCW) millimeter wave radar to change. By extracting the target phase to obtain the chest cavity displacement change, the purpose of detecting respiratory rate and heart rate is achieved. This embodiment uses FMCW millimeter wave radar, and its transmission signal is :

[0037] in, is the signal amplitude, is the initial frequency, is the frequency slope of the FM continuous millimeter wave.

[0038] The radar echo delay is , the echo signal is :

[0039] The echo signal is mixed with the transmitted signal and the intermediate frequency signal is obtained after passing through a low-pass filter. :

[0040] Where A is the echo amplitude and τ is the time delay. The intermediate frequency signal frequency is , the phase is , where λ is the wavelength, c is the speed of light, R is the distance from the radar, B is the bandwidth of the FMCW, and T is the period of a chirp.

[0041] Physiological signs such as respiration and heart rate manifest as microvibrations in specific areas of the human body surface. These signs can be captured by selecting a micro-Doppler signal at an appropriate distance. The selected FMCW radar operates in the millimeter wave band. Assuming a wavelength of 1 mm, even micro-vibrations with an amplitude of only 100 µm will produce a micro-Doppler signal with an amplitude of approximately 1.26 rad. This gives the FMCW millimeter wave radar signal a high resolution for these micro-vibration signs. Based on the micro-Doppler signal, physiological signs such as respiration and heart rate can be accurately measured.

[0042] When people walk and engage in daily activities, the echo signal detected by millimeter-wave radar has a micro-Doppler effect. By analyzing the mapping relationship between human joint movement and micro-Doppler characteristics, an overall human echo model can be constructed. The radar echo of each part of the human body is modulated by its own motion characteristics, showing the non-stationary characteristics of the signal. The swing of the upper limbs is mainly to balance the whole body torque, which makes the human gait more natural and coordinated, and can reduce the body deflection caused by the swing of the legs. The movement pattern of the shoulder joint in any gait of the human body is consistent with the human body's movement pattern. It can be modeled using sinusoidal motion, namely:

[0043] in, is the maximum swing amplitude of the upper arm, is the arm swing frequency, is the initial swing phase, and the Doppler frequency of the radar echo reflected by the boom is obtained f UA :

[0044] in, is the Doppler frequency coefficient of the upper arm swing, when hour, Reaching the maximum value F UA :

[0045] It can be seen that the Doppler frequency of the radar echo reflected by the boom obeys the sinusoidal variation law, and the period of the sinusoidal modulation is determined by the swing arm period.

[0046] The Doppler shift of the forearm is:

[0047] in, is the Doppler frequency coefficient of the upper arm swing, is the angle between the upper and lower arms. Since the swing frequencies of the upper and lower arms are equal and reach the maximum frequency shift almost at the same time, when When F LA :

[0048] It can be seen that the Doppler frequency of the radar echo reflected by the forearm also obeys the sinusoidal variation law.

[0049] By analogy, the Doppler change patterns and movement postures of the thigh and calf can be obtained.

[0050] The range-Doppler relationship graph obtained by the above method is input into the convolutional neural network and the long short-term memory neural network (CNN+LTSM) to extract the spatial-temporal characteristics of the millimeter-wave radar. :

[0051] in, is the radar sub-network function, which maps the pre-processed echo R to the feature vector space. Learn parameters for it; is the feature vector space output by the radar sub-network, with a dimension of .

[0052] For computer vision data, deep learning and convolutional neural network (CNN) technology can be used to extract human skeleton key points, motion trajectories and posture change features through real-time processing of continuous video frames. First, a pre-trained posture estimation model (such as OpenPose, HRNet, etc.) is used to detect and segment the input video stream, and automatically locate the main joints of the human body, including multiple key positions such as the head, shoulders, wrists, knees and ankles; then, based on these static detection results, the key points in the continuous frames are tracked through time series analysis and optical flow methods to construct the human motion trajectory. At the same time, each key point is connected according to the human skeleton structure, and the human key point detection is used to align the whole body or joint area to form a dynamic skeleton model, and the convolutional neural network is used to extract the features of the computer vision image. :

[0053] in, is the computer vision image sub-network function, which maps the image I to semantic features. is the convolutional network weight; It is the feature vector space output by the computer vision image sub-network, with a dimension of .

[0054] For flexible pressure sensing data, combined with time series data analysis technology, the dynamic changes of pressure images over time are tracked, and the spatiotemporal variation characteristics of pressure distribution during human movement are accurately captured. The pressure time series data p(t) after bandpass filtering to remove low-frequency baseline and high-frequency noise is sent to the long short-term memory neural network in frames, and time domain waveform features such as respiratory peak are extracted. :

[0055] in, is the pressure sub-network function, which is a long short-term memory neural network in the present invention. For its parameters; is the feature vector space output by the pressure sub-network, with a dimension of .

[0056] Based on the above feature extraction, data fusion analysis is performed below.

[0057] Since there are multiple flexible pressure sensors, the time domain and frequency domain data inside each flexible pressure sensor are first locally fused, and then the Kalman filter algorithm is used to smooth and predict the data to obtain a pressure estimation value of a pressure sensor.

[0058] Next, the locally fused modal data are input into a multimodal neural network based on deep learning, and the trained model is used to adaptively adjust the data weights, ultimately outputting a unified health parameter indicator.

[0059] Specifically, the feature vectors of the three sub-networks of millimeter-wave radar, computer vision image and pressure extracted in the previous section are combined with deep learning attention fusion and bilinear pooling methods to learn and calculate the weights of each modality. , weighted sum of each modal feature to dynamically focus on the modality of proprietary information:

[0060] in, is the attention weight of the three modal data: millimeter-wave radar, computer vision image, and pressure; softmax is the normalization function in the deep learning attention mechanism, which smoothly converts the unnormalized modal data into a probability distribution; 、 are the weights and biases of attention fusion, which are used to calculate the attention scores of each modality. Then, according to the weights of each modality, the weighted fusion feature vector F is calculated. fuse :

[0061] The fused features are input into a Bayesian neural network (BNN) to determine health status. Depending on the health indicator, this is divided into classification and regression. Classification is used to determine abnormal health events such as falls, while regression is used to determine whether continuous health indicators such as heart rate and respiratory rate are within the normal range. For classification, a linear mapping is used to calculate the probability of each health status category:

[0062]

[0063] in, and are the classification layer weights and biases, z is the normalized health category score, is the predicted probability that the healthy category belongs to the kth category, .

[0064] For each continuous health indicator, BNN regression analysis is used to predict the distribution and identify the health status. A confidence threshold is set during classification. If the value is below the threshold, it is marked as "normal" and triggers a secondary confirmation. The confidence interval of the predicted distribution is used during regression. To determine whether the heart rate and other indicators are abnormal, where k is the confidence percentage coefficient, 、 Calculated by the following formula:

[0065]

[0066] in, is the predicted value of heart rate and other indicators obtained by the t-th Monte Carlo simulation sampling, and T is the number of Monte Carlo simulation samplings.

[0067] Data fusion eliminates blind spots between individual sensors and leverages the strengths of different sensors in different environments. For example, when millimeter-wave radar is unable to collect continuous data due to obstruction, the local pressure information collected by the flexible electronic terminal can serve as an important supplement. Furthermore, in good video conditions with ample lighting, the dynamic skeletal model data provided by the vision module can accurately correct the millimeter-wave data. The fused data is more robust and accurate, providing a more reliable reflection of human health.

[0068] After completing multimodal data fusion and analyzing the fused data, the system monitors various health indicators in real time, using time series data analysis and pattern recognition to determine whether any abnormalities exist. The system can provide real-time warnings for abnormal heart rate, breathing, gait disturbances, or abnormal posture, and uploads this monitoring information to a cloud platform via wireless communication for remote viewing and timely action by medical staff or family members.

[0069] The methods for judging abnormal situations mainly include: 1. Threshold judgment: Set the normal fluctuation range of various parameters such as heart rate, respiratory rate, gait rhythm, and body posture deviation. When the monitored value exceeds the preset threshold, it is preliminarily judged as abnormal.

[0070] 2. Historical Data Comparison: Compare current data with historical health data to determine whether anomalies are short-term fluctuations or long-term anomalies. By synchronously aligning, denoising, and smoothing multidimensional health data such as heart rate, respiration, and gait, a rolling mean and standard deviation baseline for short-term (less than 7 days) and long-term (more than 30 days) data is established. Static thresholds (such as upper and lower limits of heart rate, posture stability, etc.), dynamic thresholds (baseline ± 2σ), weighted averages, and DS (Dempster–Shafer) evidence fusion methods are combined to monitor health data in real time and construct an anomaly fusion discrimination model. When data only briefly exceeds the short-term baseline (e.g., within 1 minute), does not exceed the upper limit of the static threshold, and quickly returns to normal, it can be considered an occasional short-term fluctuation. If it continuously exceeds the long-term threshold, recurs frequently, or there are multi-signal coordinated anomalies, it is determined to be a long-term anomaly.

[0071] 3. Multimodal Validation: Cross-validation is performed using data fusion results. When multiple data sources all show anomalies, the anomaly is judged to be highly reliable. The calculation strategy for anomaly reliability based on multimodal information is as follows: On offline historical datasets, k-fold cross-validation is used to evaluate the true positive rate and false positive rate of the anomaly fusion discrimination model constructed based on the DS method. The receiver operating characteristic (ROC) curve is plotted, and the area under the curve (AUC) and the optimal threshold are calculated. Based on these parameters, the 95% confidence interval of the anomaly judgment result obtained in real time based on real-time data is calculated, and a high / low confidence rating is distinguished. This creates a quantitative, calibrated, and multi-dimensional collaborative anomaly reliability assessment system.

[0072] When the system determines that the abnormal situation reaches the set trigger standard (such as being unable to stand up on one's own after falling), the early warning module will be automatically activated, and early warning information can be sent through various methods such as local alarm, remote data upload, and linkage emergency mechanism.

[0073] This comprehensive embodiment is applicable to a variety of scenarios, including homes, hospital wards, nursing homes, and sports and wellness centers. Through the flexible layout of front-end sensors and remote management from a cloud platform, the system not only enables precise monitoring of individual users, but also enables regional monitoring of the health status of individuals in multi-person environments. In multi-scenario applications, the system automatically adjusts sensor operating modes based on environmental conditions, such as enhancing millimeter-wave radar data acquisition in low-light environments and optimizing video image recognition algorithms in brightly lit environments, thereby maintaining high-precision monitoring results.

[0074] Through the description of the above embodiments, those skilled in the art can clearly understand that corresponding devices can be implemented according to the system disclosed herein. Exemplarily, a multimodal real-time monitoring device for human health status includes a millimeter-wave radar module, a computer vision module, a flexible pressure sensing module and a central processing unit; the millimeter-wave radar module is configured to obtain millimeter-wave radar echo data in real time; the computer vision module is configured to obtain computer vision image data in real time; the flexible pressure sensing module is configured to obtain flexible pressure sensing data in real time; the central processing unit is configured to use the millimeter-wave radar echo data, computer vision data and each flexible pressure sensing data as multimodal data, and perform preprocessing based on the multimodal data, the preprocessing including fusing the data of each flexible pressure sensor; the preprocessed multimodal data is used to obtain fusion features based on each modal data using a trained multimodal neural network based on deep learning, and then health status is judged based on the fusion features; wherein the health status includes vital sign information and behavioral activity posture monitoring, the vital sign information includes heart rate and respiratory rate, and the behavioral activity posture includes body balance, gait, sitting posture, sleep movement, and falls.

[0075] Through the above description of the embodiments, those skilled in the art will clearly understand that the disclosed system and apparatus can be implemented using software plus necessary general-purpose hardware. Of course, they can also be implemented using dedicated hardware, including application-specific integrated circuits, dedicated CPUs, dedicated memories, and dedicated components. Generally speaking, any function performed by a computer program can be easily implemented using corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present disclosure, software implementation is often the preferred embodiment.

[0076] Although the embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to the specific embodiments and application areas described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. A person of ordinary skill in the art, guided by this specification and without departing from the scope of protection of the claims of the present disclosure, may devise various other forms, all of which fall within the scope of protection of the present disclosure.

Claims

1. A multimodal real-time monitoring system for health status, characterized in that: The system includes an acquisition module, a preprocessing module, and a health status determination module: The acquisition module is configured to acquire multimodal data in real time, wherein the multimodal data includes millimeter wave radar echo data, computer vision image data, and various flexible pressure sensor data; The preprocessing module is configured to perform preprocessing based on the acquired multimodal data, wherein the preprocessing includes fusing the data of each flexible pressure sensor; The health status discrimination module is configured to obtain fusion features based on the preprocessed multimodal data using a trained multimodal neural network based on deep learning, and then perform health status discrimination based on the fusion features; The health status includes vital sign information and behavior and activity posture monitoring. The vital sign information includes heart rate and respiratory rate. The behavior and activity posture includes body balance, gait, sitting posture, sleeping movement, and falls.

2. The system according to claim 1, wherein: The fusion features obtained based on each modality data include obtaining space-time features with vital signs and behavioral activity characteristics based on the preprocessed millimeter wave radar echo data. , extracting behavioral activity features based on pre-processed computer vision image data , based on the pre-processed flexible pressure sensing data, the time domain waveform features with vital signs and behavioral activity characteristics are obtained ,Will 、 、 The fusion is performed to obtain the fusion features.

3. The system according to claim 2, characterized in that Will 、 、 The fusion feature is obtained by first calculating the weight of each modal data by using the attention mechanism, and then using bilinear pooling to obtain the weighted fusion feature.

4. The system according to claim 1, wherein: The health status discrimination based on fusion features is implemented using a Bayesian neural network.

5. The system according to claim 4, characterized in that The regression discrimination in the Bayesian neural network is used to judge whether the preset vital sign information is within the preset normal range based on the fusion features, and the classification discrimination in the Bayesian neural network is used to judge whether the preset behavioral activity posture is normal based on the fusion features.

6. The system according to claim 1, wherein: The fusion processing of the data of each flexible pressure sensor includes first fusing the time domain and frequency domain of the data of each flexible pressure sensor, and then using the Kalman filter algorithm to smooth and predict the data to obtain an estimated pressure value.

7. The system according to claim 1, wherein: The preprocessing includes: filtering the millimeter wave radar echo data and extracting the echo signal, performing background subtraction and image enhancement on the computer vision image, and performing low-pass filtering and signal smoothing on the flexible pressure sensor data.

8. The system according to claim 1, wherein: When the health status is judged as "abnormal", a secondary confirmation is triggered, and cross-validation is performed based on the fusion features to obtain the abnormal credibility.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the system according to any one of claims 1 to 8.

10. A multimodal real-time monitoring device for human health status, characterized in that: The device includes a millimeter wave radar module, a computer vision module, a flexible pressure sensing module and a central processing unit; The millimeter wave radar module is configured to acquire millimeter wave radar echo data in real time; The computer vision module is configured to acquire computer vision image data in real time; The flexible pressure sensing module is configured to obtain flexible pressure sensing data in real time; The central processing unit is configured to treat the millimeter wave radar echo data, the computer vision data, and the flexible pressure sensor data as multimodal data and perform preprocessing based on the multimodal data, wherein the preprocessing includes fusing the data of the flexible pressure sensors; The pre-processed multimodal data is used to obtain fusion features based on each modality data using a trained multimodal neural network based on deep learning. The health status is then determined based on the fusion features. The health status includes vital signs information and behavioral activity posture monitoring. The vital signs information includes heart rate and respiratory rate. The behavioral activity posture includes body balance, gait, sitting posture, sleeping movements, and falls.

Images