A vehicle-mounted health monitoring method, program product, electronic device and storage medium

Abstract

The application provides a vehicle-mounted health monitoring method, a program product, an electronic device and a storage medium. The method comprises: collecting continuous original data through a multi-modal acquisition module arranged on a vehicle; the multi-modal acquisition module comprises a radar module, an infrared camera and a MEMS sensor; performing feature extraction on an original radio frequency signal sequence, an original infrared thermal image sequence and an original vibration signal sequence respectively to obtain physiological feature data, infrared thermodynamic feature data and behavior feature data; performing feature fusion on the physiological feature data, the infrared thermodynamic feature data and the behavior feature data based on space-time correlation to obtain global features; and inputting the global feature data into a health detection model to obtain a health monitoring result. Through the multi-modal acquisition module, synchronous original data covering three dimensions of physiology, thermodynamics and behavior is obtained, thereby laying a solid data foundation for subsequent comprehensive and accurate analysis of the health status of a driver or a passenger.

Description

Technical Field

[0001] This application relates to the field of health monitoring technology, and more specifically, to an in-vehicle health monitoring method, program product, electronic device, and storage medium. Background Technology

[0002] In-vehicle health monitoring is crucial for improving driving safety and occupant health management. Current technologies mainly rely on contact-based biosensors, such as sensor vests, patches, or optical heart rate wristbands. These devices need to be worn directly on the body, which presents problems such as inconvenience and discomfort for the user. Another method is camera-based monitoring, but this suffers from inaccurate monitoring. Summary of the Invention

[0003] The purpose of this application is to provide an in-vehicle health monitoring method, program product, electronic device, and storage medium to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide an in-vehicle health monitoring method, comprising: acquiring continuous raw data through a multimodal acquisition module installed in the vehicle; the multimodal acquisition module includes a radar module, an infrared camera, and a MEMS sensor; the raw data includes a raw radio frequency signal sequence acquired by the radar module, a raw infrared thermal image sequence acquired by the infrared camera, and a raw vibration signal sequence acquired by the MEMS sensor; extracting features from the raw radio frequency signal sequence, the raw infrared thermal image sequence, and the raw vibration signal sequence respectively to obtain physiological feature data, infrared thermodynamic feature data, and behavioral feature data; fusing the physiological feature data, infrared thermodynamic feature data, and behavioral feature data based on spatiotemporal correlation to obtain global features; and inputting the global feature data into a health detection model to obtain health monitoring results.

[0005] In the aforementioned implementation process, multimodal acquisition modules were deployed to obtain synchronous raw data covering three dimensions: physiological, thermodynamic, and behavioral. This laid a solid data foundation for subsequent comprehensive and accurate analysis of the driver's or passenger's health status. Feature extraction from the three raw signals transformed the raw, high-dimensional, and noisy signal data into low-dimensional feature vectors with clear physiological and behavioral significance, improving the accuracy of subsequent health status assessment. Deep feature fusion based on spatiotemporal correlation was used to discover and utilize the inherent relationships between multimodal data, generating more comprehensive global features. The health detection model uses the fused global features for final decision-making, more reliably identifying the driver's true health status.

[0006] Optionally, in this embodiment of the application, after acquiring continuous raw data through a multimodal acquisition module located in the vehicle, the method further includes: performing a fast Fourier transform on the raw radio frequency signal sequence to generate a micro-Doppler time-series signal containing physiological information; using a raw vibration signal sequence synchronized with the raw radio frequency signal sequence as reference noise to separate noise from the micro-Doppler time-series signal to obtain a purified micro-Doppler time-series signal; and performing feature extraction on the raw radio frequency signal sequence to obtain physiological feature data, including: performing feature extraction on the purified micro-Doppler time-series signal to obtain physiological feature data.

[0007] In the aforementioned implementation process, by performing Fast Fourier Transform and related signal processing, the system successfully extracted low-dimensional time-series signals that intuitively reflect the micro-movements of the human chest cavity (i.e., breathing and heartbeat) from the complex high-dimensional raw radio frequency signals. By utilizing time-synchronized MEMS vibration signals as reference noise and employing adaptive filtering techniques, the system can extremely effectively identify and separate strong interference noise introduced by vehicle vibration from aliased radar signals, improving the accuracy and reliability of monitoring physiological parameters such as heart rate and respiratory rate.

[0008] Optionally, in this embodiment, the original vibration signal sequence synchronized with the original radio frequency signal sequence is used as reference noise to separate noise from the micro-Doppler timing signal to obtain a purified micro-Doppler timing signal. This includes: obtaining the original vibration signal sequence synchronized with the original radio frequency signal sequence based on the timestamp of the original radio frequency signal sequence; using the micro-Doppler timing signal sequence as the main channel input of an adaptive filtering algorithm, and using the time-synchronized original vibration signal sequence as the reference input channel of the adaptive filtering algorithm; learning the relationship between the main channel input and the reference input channel through the adaptive filtering algorithm to obtain a predicted noise signal; using the predicted noise signal to simulate interference signals present in the micro-Doppler timing signal; and performing noise separation on the micro-Doppler timing signal based on the predicted noise signal to obtain the purified micro-Doppler timing signal.

[0009] In the above implementation process, a highly realistic predictive noise signal is generated by feeding the micro-Doppler signal and vibration signal as the main input and reference input, respectively, into an adaptive filter. The predictive noise signal is then directly subtracted from the aliased micro-Doppler signal, achieving physical separation of vibration interference and physiological micro-motion signals. This results in a pure physiological signal with a significantly improved signal-to-noise ratio and fundamentally improved waveform quality. This makes the weak periodic waveforms caused by heartbeat and respiration, which were previously submerged by strong vehicle vibration noise, clearly discernible, thus improving the accuracy and reliability of the health monitoring system's output results.

[0010] Optionally, in this embodiment of the application, feature extraction is performed on the original radio frequency signal sequence to obtain physiological feature data, including: performing a short-time Fourier transform on the original radio frequency signal sequence to convert the original radio frequency signal sequence from the time domain to the time-frequency domain, generating a spectrum; the spectrum is used to reflect the distribution of the original radio frequency signal sequence over time and frequency; and based on the spectrum, physiological feature data is obtained using a peak detection algorithm and / or a spectrum tracking algorithm.

[0011] In the above implementation process, by performing a short-time Fourier transform on the original radio frequency signal sequence and generating a spectrum, the one-dimensional time-series signal is converted into a two-dimensional time-frequency image, thereby enabling clear observation and quantification of the frequency of physiological signals (respiration and heartbeat) and their changes over time. Based on this, using peak detection and spectrum tracking algorithms, key physiological characteristic data such as instantaneous respiratory rate, instantaneous heart rate, and heart rate variability can be robustly and accurately extracted from the spectrum. This method effectively utilizes the frequency domain characteristics of the signal, has strong anti-interference capabilities, and can significantly improve the accuracy of monitoring vital signs such as heart rate and respiratory rate in complex environments.

[0012] Optionally, in this embodiment of the application, feature extraction is performed on the original infrared thermal image sequence to obtain infrared thermodynamic feature data, including: identifying a target region for each frame in the original infrared sequence to obtain at least one target region; acquiring temperature data of the target region in each frame; forming a temperature waveform sequence based on the temperature data of the same target region in each frame of the original infrared sequence; the temperature waveform sequence is used to characterize the waveform signal of the temperature change of the same target region over time; and performing thermodynamic feature extraction based on the temperature waveform sequence to obtain infrared thermodynamic feature data.

[0013] In the aforementioned implementation process, by performing frame-by-frame target area identification and temperature data extraction on the original infrared sequence, the two-dimensional image sequence information is transformed into a one-dimensional temperature waveform sequence that can intuitively reflect physiological state. Based on this waveform sequence, thermodynamic feature extraction can obtain feature data closely related to breathing patterns, blood circulation status, and autonomic nervous system activity, such as respiratory rate, temperature stability, and trends. These thermodynamic features provide an independent information dimension for health status assessment, distinct from radar physiological signals. By fusing with other modal features, they can mutually corroborate and complement each other, thereby comprehensively improving the accuracy and reliability of monitoring driver health status (such as stress and fatigue).

[0014] Optionally, in this embodiment of the application, feature extraction is performed on the original vibration signal sequence to obtain behavioral feature data, including: filtering out vehicle vibration signals from the original vibration signal sequence to obtain behavioral signals; and performing feature extraction on the behavioral signals to obtain behavioral feature data.

[0015] In the aforementioned implementation process, by filtering out vehicle-related vibration noise from the original vibration signal, behavioral signals purely generated by the driver's physical activity were extracted. Based on this, behavioral characteristic data such as body activity level, muscle tension, and abnormal movement index were calculated, providing direct behavioral evidence for assessing the driver's fatigue level, stress state, and sudden physical abnormalities. These characteristics provide a quantitative description of the driver's state from a completely new dimension (behavioral science), effectively complementing physiological and thermodynamic characteristics, enriching the information sources for the system's comprehensive decision-making, thereby significantly enhancing the ability to perceive health status in complex real-world driving scenarios and comprehensively improving the accuracy and robustness of monitoring.

[0016] Optionally, in this embodiment of the application, physiological feature data, infrared thermodynamic feature data, and behavioral feature data are fused based on spatiotemporal correlation to obtain global features, including: acquiring time-synchronized physiological feature data, infrared thermodynamic feature data, and behavioral feature data; constructing a spatiotemporal correlation graph structure using the time-synchronized physiological feature data, infrared thermodynamic feature data, and behavioral feature data as nodes; the spatiotemporal correlation graph structure is used to characterize the temporal correlation and / or spatial correlation between different types of feature data; and performing feature fusion based on the spatiotemporal correlation graph structure to obtain global features.

[0017] In the aforementioned implementation process, by constructing a spatiotemporal correlation graph structure containing spatial and temporal edges, abstract multivariate time series data is transformed into an intuitive, relation-rich graph representation. This step explicitly defines the temporal interactions between different modal features and between the features themselves, embedding prior physiological knowledge (such as the correlation between physiological and thermal signals) and data-driven temporal dependencies into the model structure, thereby improving the comprehensive and highly fused feature vectors for health detection.

[0018] Optionally, in the embodiments of this application, the edges in the spatiotemporal correlation graph structure include spatial edges and temporal edges; spatial edges are the edges that connect the corresponding time-synchronized physiological characteristic data, infrared thermodynamic characteristic data and behavioral characteristic data within the same node; temporal edges refer to the edges that connect different nodes in chronological order.

[0019] In the above implementation process, a spatiotemporal relational graph structure that deeply expresses the intrinsic relationships of data is constructed by defining spatial and temporal edges. Spatial edges force the model to learn the interactions and constraints between physiological, thermodynamic, and behavioral characteristics at the same time, and utilize the complementarity of multimodal information to achieve consistent modeling and cross-validation of instantaneous states. Temporal edges establish the causal relationships and evolution paths of features over time, enabling the model to perceive the dynamic changing trends of health status.

[0020] Optionally, in this embodiment of the application, feature fusion is performed based on the spatiotemporal correlation graph structure to obtain global features, including: aggregating time-synchronized physiological feature data, infrared thermodynamic feature data and behavioral feature data corresponding to the same node to generate node features; and splicing multiple node features in chronological order between nodes to obtain global features.

[0021] In the above implementation process, the resulting global features can locally reflect different types of feature data related at the same time, and globally reflect the changing patterns of different types of feature data over time. Global features can contain more comprehensive and complete information.

[0022] Secondly, embodiments of this application also provide an in-vehicle health monitoring device, comprising: a data acquisition module for acquiring continuous raw data through a multimodal acquisition module installed in the vehicle; the multimodal acquisition module includes a radar module, an infrared camera, and a MEMS sensor; the raw data includes raw radio frequency signal sequences acquired by the radar module, raw infrared thermal image sequences acquired by the infrared camera, and raw vibration signal sequences acquired by the MEMS sensor; a feature extraction module for extracting features from the raw radio frequency signal sequences, raw infrared thermal image sequences, and raw vibration signal sequences respectively to obtain physiological feature data, infrared thermodynamic feature data, and behavioral feature data; a feature fusion module for fusing the physiological feature data, infrared thermodynamic feature data, and behavioral feature data based on spatiotemporal correlation to obtain global features; and a health monitoring module for inputting the global feature data into a health detection model to obtain health monitoring results.

[0023] Thirdly, embodiments of this application also provide a computer program product, including computer program instructions, which are executed by a processor to perform the method provided in the first aspect or any implementation thereof.

[0024] Fourthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing computer program instructions, which are executed by the processor to perform the method provided in the first aspect or any implementation thereof.

[0025] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any implementation thereof.

[0026] This application discloses an in-vehicle health monitoring method, program product, electronic device, and storage medium. By deploying a multimodal acquisition module, synchronous raw data covering three dimensions—physiological, thermodynamic, and behavioral—is acquired, laying a solid data foundation for subsequent comprehensive and accurate analysis of the driver's or passenger's health status. Feature extraction from the three raw signals transforms the raw, high-dimensional, noisy signal data into low-dimensional feature vectors with clear physiological and behavioral significance, improving the accuracy of subsequent health status determination. Deep feature fusion based on spatiotemporal correlation discovers and utilizes the inherent relationships between multimodal data to generate more comprehensive global features. The health detection model uses the fused global features for final decision-making, more reliably identifying the driver's true health status. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic flowchart illustrating an in-vehicle health monitoring method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the vehicle-mounted health monitoring device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0031] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0032] In-vehicle health monitoring is crucial for improving driving safety and passenger health management. It can monitor the driver's heart rate, stress level, and fatigue status in real time, promptly detect sudden health events (such as myocardial infarction and fainting), thereby preventing traffic accidents and providing data support for long-term health tracking.

[0033] Current technologies primarily rely on contact-based biosensors, such as electrocardiogram (ECG) patches or optical heart rate wristbands. These devices need to be worn directly on the skin, which has significant drawbacks: first, they are inconvenient to wear and cause a foreign body sensation, affecting driving comfort; second, their signals are easily interfered with by driving bumps and body movements, producing motion artifacts and reducing data reliability; and third, they depend on the user's active wearing, making it difficult to achieve continuous, unobtrusive, routine monitoring, resulting in a poor user experience.

[0034] This application provides an in-vehicle health monitoring method. By deploying a multimodal acquisition module, it acquires synchronous raw data covering three dimensions: physiological, thermodynamic, and behavioral, laying a solid data foundation for subsequent comprehensive and accurate analysis of the driver's or passenger's health status. Feature extraction from the three raw signals transforms the raw, high-dimensional, noisy signal data into low-dimensional feature vectors with clear physiological and behavioral significance, improving the accuracy of subsequent health status determination. Through deep feature fusion based on spatiotemporal correlation, the inherent relationships between multimodal data are discovered and utilized to generate more comprehensive global features. The health detection model uses the fused global features for final decision-making, more reliably identifying the driver's true health status.

[0035] Please see Figure 1 The illustration shows a flowchart of an in-vehicle health monitoring method provided in an embodiment of this application. The in-vehicle health monitoring method provided in this application can be applied to electronic devices, which may include physical devices such as servers, PCs, tablets, or smartphones, or virtual devices such as virtual machines or containers. The electronic device can be a single device, a combination of multiple devices, or a cluster of a large number of devices. The in-vehicle health monitoring method may include: Step S110: Collect continuous raw data through the multimodal acquisition module set in the vehicle; the multimodal acquisition module includes a radar module, an infrared camera, and a MEMS sensor; the raw data includes the raw radio frequency signal sequence collected by the radar module, the raw infrared thermal image sequence collected by the infrared camera, and the raw vibration signal sequence collected by the MEMS sensor.

[0036] Step S120: Extract features from the original radio frequency signal sequence, the original infrared thermographic sequence, and the original vibration signal sequence to obtain physiological feature data, infrared thermodynamic feature data, and behavioral feature data.

[0037] Step S130: The physiological feature data, infrared thermodynamic feature data, and behavioral feature data are fused based on spatiotemporal correlation to obtain global features.

[0038] Step S140: Input the global feature data into the health detection model to obtain the health monitoring results.

[0039] In step S110, the radar module can penetrate clothing to precisely capture millimeter-level micro-movements in the chest cavity caused by heartbeat and respiration in a non-contact manner, acquiring the raw radio frequency signal sequence. The raw radio frequency signal sequence refers to the raw data stream generated after the ultra-wideband radar module emits electromagnetic pulses and receives the echoes reflected back from the human body. The raw radio frequency signal sequence contains rich raw information such as the distance, phase, and signal strength of the target (e.g., the driver's chest cavity), which is the material basis for subsequent extraction of physiological signals such as heartbeat and respiration.

[0040] An infrared camera captures a series of thermal images at a certain frame rate, forming the raw infrared thermal image sequence. Each pixel in each frame represents a temperature value, and the entire sequence records the dynamic changes in temperature distribution over time for a target, such as a driver's face and other localized areas of their body. Infrared cameras do not rely on ambient visible light and can obtain thermodynamic information related to blood circulation and respiratory heat flow by detecting facial thermal radiation distribution, providing a crucial supplementary dimension for health status assessment, all without requiring contact.

[0041] MEMS sensors are inertial sensors based on microelectromechanical systems, such as accelerometers and gyroscopes integrated into vehicles. MEMS sensors are used to acquire linear acceleration and rotational angular velocity of a vehicle in three-dimensional space at high frequencies, and the raw vibration signal sequence reflects the vehicle's vibration and motion state.

[0042] In one implementation, the radar module, infrared camera, and MEMS sensor in the multimodal acquisition module can be placed in the same or different locations, and the installation location and number can be set according to the actual situation. For example, the radar module can be installed on the center line of the vehicle roof, with a beamwidth covering the driver's seat, passenger seat, and rear seats; the infrared camera can be installed above the steering wheel with a field of view of 60°×45°; and the MEMS sensor can be fixed to the seat rail. The MEMS sensor can be installed under the seat frame or in the chassis near the seat. It can measure the real vibrations generated during vehicle operation with high fidelity. This signal can be used as a noise reference source to purify the radar signal, and it also contains unique vibration components caused by the body movements of the driver or passengers.

[0043] As one implementation method, to facilitate subsequent health monitoring, a unified hardware or software clock can be used to precisely synchronize the three raw data streams with timestamps, ensuring complete temporal alignment during subsequent processing. The collected raw data is cached in sequence for subsequent feature extraction.

[0044] It should be noted that all data collected in this application's embodiments shall comply with regulations such as the Personal Information Protection Act and GDPR. For example, this data will be collected with the user's consent and in accordance with privacy protection design, and the collected data will be used for health monitoring with the user's consent, without disclosing any data.

[0045] In step S120, the process of extracting features from the original radio frequency signal sequence to obtain physiological feature data includes, for example: First, performing a distance-dimensional Fast Fourier Transform (FFT) on the signal to lock onto the distance unit corresponding to the driver's chest cavity and focus on the target signal. Next, performing a Doppler-dimensional FFT converts the signal to the frequency domain, generating a spectrum. On the spectrum, using a peak detection algorithm or a segmented detection method, the frequency points with the highest energy are found in the respiratory and heart rate bands, respectively, to calculate the instantaneous respiratory rate and instantaneous heart rate. Furthermore, time-domain or frequency-domain analysis can be performed on the heart rate sequence (e.g., calculating the standard deviation of the difference between adjacent heartbeats) to obtain higher-level features such as heart rate variability as physiological feature data.

[0046] The process of extracting features from raw infrared thermographic sequences to obtain infrared thermodynamic feature data can be as follows: First, the target region can be determined from the frame images of the raw infrared thermographic sequence. Here, the target region refers to areas that can reflect health conditions, such as the nasal alar region and forehead region, which can indicate whether a person is experiencing fever, and therefore can be used as the target region. Then, a continuous temperature waveform sequence of the target region is obtained from multiple frames corresponding to the raw infrared thermographic sequence. Thermodynamic features are extracted from this temperature waveform sequence to obtain infrared thermodynamic feature data. For example, fluctuation analysis can be performed to calculate its fluctuation amplitude, frequency (cross-validated with respiratory rate), and stability (variance) as infrared thermodynamic feature data.

[0047] The process of extracting features from the original vibration signal sequence to obtain behavioral feature data can be as follows: First, vehicle vibration signals can be filtered out from the original vibration signal sequence to obtain behavioral signals. Vehicle vibration signals refer to low-frequency, high-energy components related to vehicle vibration, which may interfere with health monitoring, so they are filtered out. High-frequency, low-energy components generated by the physical activities of the driver or passengers are retained; this portion is used as the behavioral signal. Feature extraction is then performed on the behavioral signal to obtain behavioral feature data. For example, the variance of the filtered signal can be calculated as an indicator of physical activity; the signal power in a specific frequency band (such as the frequency band corresponding to muscle tremors) can be analyzed as an indicator of muscle tension.

[0048] In step S130, spatiotemporal correlation refers to the inherent connections and mutual constraints between feature data of different modalities in the temporal and spatial dimensions. Temporal correlation is reflected in the synchronicity or causal relationship of different features changing over time; spatial correlation is reflected in the inherent connection between different physiological phenomena in the physical space of the human body (such as the heartbeat inevitably causing blood flow and changes in body surface temperature). Utilizing this correlation for fusion enables the model to learn a consistent representation that conforms to physiological laws.

[0049] Before fusion, the three types of feature data—physiological feature data, infrared thermodynamic feature data, and behavioral feature data—are aligned on the time axis. For the three types of feature data at the same time, an attention mechanism can be used to dynamically calculate the correlation strength (spatial correlation) between feature nodes of different modalities. For example, the system can learn whether there is a strong causal relationship between sudden changes in heart rate and sudden bodily movements. For the same type of feature data at different times, the concept of recurrent neural networks or temporal convolution can be used to capture the change pattern of the same feature over time (temporal correlation). For example, the change pattern of heart rate over time. Then, the features learned from temporal and spatial correlations are aggregated to obtain global features. Global features refer to global feature vectors containing spatiotemporal context information of multimodal data.

[0050] By using deep feature fusion based on spatiotemporal correlation, the system no longer views heart rate, body temperature, or movement in isolation, but analyzes them as an organic whole. This approach can discover and utilize the inherent, physiologically logical consistency constraints between multimodal data, thereby effectively suppressing single-modal noise interference, correcting erroneous estimates, and generating a more comprehensive and robust state representation. This is the most crucial step in significantly improving monitoring accuracy.

[0051] In step S140, global feature data is input into the health monitoring model to obtain health monitoring results. The health monitoring model may include a classifier (such as a multilayer perceptron). Internally, the health monitoring model maps the input high-dimensional global feature vector to the final decision space through a series of nonlinear transformations (implemented through activation functions) and weighted calculations. The output layer of the health monitoring model can be a softmax function, and its output is the health monitoring result, such as a discrete label representing stress level (e.g., low, medium, high), or a probability value representing the occurrence of a specific health risk event (e.g., fatigued driving, sudden arrhythmia).

[0052] In the implementation of the above embodiments: by deploying multimodal acquisition modules, synchronous raw data covering three dimensions—physiological, thermodynamic, and behavioral—was acquired, laying a solid data foundation for subsequent comprehensive and accurate analysis of the driver's or passenger's health status. Feature extraction from the three raw signals transformed the raw, high-dimensional, noisy signal data into low-dimensional feature vectors with clear physiological and behavioral significance, improving the accuracy of subsequent health status determination. Through deep feature fusion based on spatiotemporal correlation, the inherent relationships between multimodal data were discovered and utilized to generate more comprehensive global features. The health detection model uses the fused global features for final decision-making, more reliably identifying the driver's true health status.

[0053] Optionally, in this embodiment of the application, after acquiring continuous raw data through the multimodal acquisition module located in the vehicle, the method further includes: A Fast Fourier Transform (FFT) is performed on the original radio frequency signal sequence to generate a micro-Doppler time-series signal containing physiological information. The FFT converts the signal from the time domain to the frequency domain, enabling analysis of the frequency components and their intensity. The micro-Doppler time-series signal refers to a time-varying Doppler frequency shift caused by subtle movements of the chest cavity (heartbeat and respiration). Its waveform fluctuations correspond to changes in chest cavity displacement.

[0054] The calculation process is as follows: For each cycle of the original radio frequency signal sequence, the data acquired undergoes a Fast Fourier Transform (FFT) along the range dimension (i.e., the sampling point dimension). The FFT transforms the signal from the time-sampling point domain to the range-frequency domain. The transformation result forms a "range profile," in which a distinct peak appears at a specific range cell. The distance corresponding to this peak is the distance between the radar and the target (such as the driver's chest cavity). The specific range gate containing information about the subtle movements of the chest cavity can be determined through the FFT.

[0055] Extract the signal phase information within the locked distance gate. Since the micro-movements of the chest cavity (breathing and heartbeat) modulate the phase of the reflected electromagnetic waves, this phase change is proportional to the displacement. Arrange the extracted phase information from multiple consecutive cycles in chronological order to form a phase-time sequence. Demodulate and expand this phase sequence to obtain the micro-Doppler time-series signal that directly reflects the chest cavity displacement.

[0056] Using the original vibration signal sequence synchronized with the original radio frequency signal sequence as reference noise, noise separation is performed on the micro-Doppler timing signal to obtain the purified micro-Doppler timing signal.

[0057] Reference noise refers to a signal that is highly correlated with the noise to be eliminated but uncorrelated with the useful signal, and is used as a template to guide noise elimination during the filtering process.

[0058] Vibrations generated by a vehicle during operation (transmitted to the body through the tires and suspension system) constitute a mechanical, overall motion. This motion causes the entire vehicle body, including the driver's seat and body, to sway collectively, at low frequencies, and with high amplitude. Sensitive UWB radar cannot distinguish between this overall body sway and the subtle movements of the chest cavity caused by cardiopulmonary activity. Both modulate the phase of the radar echo, resulting in their superposition in the generated micro-Doppler timing signal. Therefore, vehicle vibration is a powerful, correlated additive noise in the radar's physiological signal. Its noise waveform is highly similar in time domain morphology to the vehicle vibration waveform acquired by MEMS sensors.

[0059] MEMS sensors are fixedly mounted on the vehicle body, and the raw vibration signal sequences they measure reflect the vehicle's own vibrations almost purely and with high fidelity. Therefore, this signal can be regarded as a reference template for the vibration noise component in radar signals.

[0060] One separation method is to employ an adaptive filter algorithm, which has two inputs: a main channel input and a reference input channel. The micro-Doppler time series signal sequence is used as the main channel input of the adaptive filtering algorithm, while the time-synchronized original vibration signal sequence is used as the reference input channel. The adaptive filtering algorithm learns the relationship between the main channel input and the reference input channel to obtain a predicted noise signal. This predicted noise signal is used to simulate interference signals present in the micro-Doppler time series signal. Based on the predicted noise signal, noise separation is performed on the micro-Doppler time series signal to obtain a purified micro-Doppler time series signal.

[0061] Feature extraction is performed on the original radio frequency signal sequence to obtain physiological feature data, including: feature extraction of the purified micro-Doppler time sequence signal to obtain physiological feature data.

[0062] Physiological characteristic data refers to indicators and parameters calculated from purified physiological signals that can quantitatively characterize the state of the human cardiovascular and respiratory systems. Examples of physiological characteristic data include instantaneous respiratory rate, mean respiratory rate, respiratory depth, mean heart rate, and / or heart rate variability.

[0063] For example, respiratory rate can be extracted by peak detection of the signal to find the peak point of each respiratory cycle (one inhalation and one exhalation), and the time interval between consecutive peaks can be calculated. The reciprocal of the time interval is the instantaneous respiratory rate. Alternatively, a fast Fourier transform can be performed on the signal to find the energy peak in the 0.1-0.3 Hz frequency band. The corresponding frequency is the respiratory rate.

[0064] The respiratory signal is superimposed with higher-frequency, smaller-amplitude fluctuations, usually caused by the heartbeat. For heart rate and its variability extraction: First, a bandpass filter is used to filter out low-frequency and high-frequency noise such as respiration, retaining the frequency band corresponding to the heartbeat (e.g., 0.8-2.0Hz), obtaining the heartbeat signal. Then, peak detection is performed to find the position corresponding to each heartbeat, and the interval between adjacent heartbeats is calculated; its reciprocal is the instantaneous heart rate. Further, statistical analysis is performed on this series of heartbeat intervals, such as calculating its standard deviation or root mean square difference, to obtain the heart rate variability index, which measures autonomic nervous system function.

[0065] In the implementation of the above embodiments: by performing Fast Fourier Transform and related signal processing, the system successfully extracted low-dimensional time-series signals that intuitively reflect the micro-movements of the human chest cavity (i.e., breathing and heartbeat) from the complex high-dimensional raw radio frequency signals. By using time-synchronized MEMS vibration signals as reference noise and processing them with adaptive filtering technology, the system can dynamically learn and track the characteristics of vibration noise, thereby effectively identifying and separating strong interference noise introduced by vehicle vibration from aliased radar signals. Compared with traditional fixed filters, this method can better adapt to vibration changes under different road conditions and vehicle speeds, significantly improve the signal-to-noise ratio, and enhance the accuracy and reliability of monitoring physiological parameters such as heart rate and respiratory rate.

[0066] Optionally, in this embodiment, the original vibration signal sequence synchronized with the original radio frequency signal sequence is used as reference noise to separate noise from the micro-Doppler timing signal, obtaining a purified micro-Doppler timing signal, including: Based on the timestamps of the original radio frequency (RF) signal sequence, a raw vibration signal sequence synchronized with the original RF signal sequence is obtained. Time synchronization means that each frame of data acquired by the radar and the data points acquired by the MEMS sensor have a strict alignment relationship on the time axis, ensuring that they describe the physical phenomenon at the same moment. For example, two first-in-first-out (FIFO) data buffers can be maintained, and each data block stored in the buffer carries its precise acquisition timestamp. By querying and searching, segments of the original vibration signal sequence with timestamps that are exactly the same as the original RF signal sequence or within the allowable small error range are found.

[0067] The original radio frequency signal sequence is used as the main channel input of the adaptive filtering algorithm, and the original vibration signal sequence synchronized with time is used as the reference input channel of the adaptive filtering algorithm. The relationship between the main channel input and the reference input channel is learned through the adaptive filtering algorithm to obtain the predicted noise signal. The predicted noise signal is used to simulate the interference signal present in the micro-Doppler timing signal.

[0068] The algorithm for an adaptive filter has two inputs: the main input channel is a noisy micro-Doppler timing signal (containing the useful signal and vibration noise); the reference input channel is the original vibration signal sequence (only as a reference for vibration noise).

[0069] The most recent set of sampled values ​​of the signal from the reference input channel (i.e., the MEMS vibration signal) is fed into the filter. The current sampled value of the main channel input (i.e., the noisy micro-Doppler signal) is used as the desired response. The filter calculates its output predicted noise signal, denoted as y(n), which can be calculated as the inner product of the weight vector and the reference input vector.

[0070] The filter output y(n) is compared with the micro-Doppler timing signal sequence d(n) input to the main channel, and the error signal e(n) = d(n) - y(n) is calculated. This error signal represents the difference between the current predicted noise and the actual noise, and is also a preliminary estimate of the "useful signal" expected by the system. Based on the error signal e(n), the adaptive filter updates the filter's weight coefficients using the formula of the normalized least mean square algorithm. The filter weights are continuously adjusted in this process, making its output y(n) increasingly approximate the vibration noise contained in the main channel d(n). Finally, the filter converges, determining the predicted noise signal y(n).

[0071] Based on the predicted noise signal, noise separation is performed on the micro-Doppler time series signal to obtain the purified micro-Doppler time series signal. Noise separation refers to the process of subtracting noise components from a noisy signal to extract the useful signal. The separation method can be to perform a scalar subtraction operation at each discrete time point. The current sampled value d(n) of the noisy micro-Doppler time series signal input from the main channel is subtracted from the current value y(n) of the predicted noise signal generated by the adaptive filter. The calculation formula is: s(n) = d(n) - y(n). The result of the subtraction, s(n), is the sampled value of the purified micro-Doppler time series signal at the current time point. By continuously performing this subtraction operation over time, the complete purified signal sequence can be obtained.

[0072] In the implementation of the above embodiments: by feeding the micro-Doppler signal and vibration signal as the main input and reference input respectively into an adaptive filter, a highly realistic predictive noise signal is generated. The predictive noise signal is directly subtracted from the aliased micro-Doppler signal, achieving physical separation of vibration interference and physiological micro-motion signals. This results in a pure physiological signal with significantly improved signal-to-noise ratio and fundamentally improved waveform quality. This makes the weak periodic waveforms caused by heartbeat and respiration, which were originally submerged by strong vehicle vibration noise, clearly discernible, improving the accuracy and reliability of the health monitoring system's output results.

[0073] Optionally, in this embodiment of the application, feature extraction is performed on the original radio frequency signal sequence to obtain physiological feature data, including: A short-time Fourier transform is performed on the original radio frequency signal sequence to convert it from the time domain to the time-frequency domain, generating a spectrum. The spectrum is used to reflect the distribution of the original radio frequency signal sequence over time and frequency.

[0074] The Short-Time Fourier Transform (SFT) is a mathematical tool used to analyze non-stationary signals (i.e., signals whose statistical properties change over time). It obtains information about the signal's distribution in time and frequency by dividing a long-duration signal into multiple shorter, overlapping segments and performing a Fourier Transform on each segment separately. A spectrogram is a visual representation of the SFT result; it can be a two-dimensional image where the X-axis represents time, the Y-axis represents frequency, and the color intensity (or brightness) of each pixel represents the signal energy (or amplitude) at that specific moment and frequency.

[0075] The calculation method is as follows: First, the continuous micro-Doppler time sequence signal (usually purified) is divided into multiple short time intervals, called "frames". Each frame is typically several seconds long, and there is partial overlap between adjacent frames to ensure temporal continuity.

[0076] Then, a Fast Fourier Transform is performed on each windowed frame of the signal to transform it from the time domain to the frequency domain, obtaining the signal spectrum within that time segment. The calculated spectra for each frame are then arranged in chronological order to generate a spectrum diagram. This spectrum clearly shows the distribution of signal energy in the time-frequency plane.

[0077] Based on the spectrogram, physiological characteristic data are obtained using peak detection algorithms and / or spectral tracking algorithms. For example, for respiratory rate extraction, the respiratory signal typically appears as a continuously varying, high-energy spectral line in the frequency range of 0.1 Hz to 0.3 Hz on the spectrogram. The spectral tracking algorithm automatically finds and tracks the strongest frequency trajectory within this band; the instantaneous value of this trajectory is the instantaneous respiratory rate.

[0078] For heart rate extraction, the heartbeat signal is represented by another spectral line in the frequency range of 0.8 Hz to 2.0 Hz (i.e., 48 beats / minute to 120 beats / minute). The spectral tracking algorithm is also used to track within this frequency band to obtain the instantaneous heart rate value.

[0079] Statistical analysis (such as calculating its standard deviation) can be performed on the tracked heart rate interval sequence to obtain the heart rate variability index.

[0080] In the implementation of the above embodiments: by performing a short-time Fourier transform on the original radio frequency signal sequence and generating a spectrum, the one-dimensional time-series signal is converted into a two-dimensional time-frequency image, thereby enabling clear observation and quantification of the frequency of physiological signals (respiration and heartbeat) and their changes over time. Based on this, using peak detection and spectrum tracking algorithms, key physiological characteristic data such as instantaneous respiratory rate, instantaneous heart rate, and heart rate variability can be robustly and accurately extracted from the spectrum. This method effectively utilizes the frequency domain characteristics of the signal, has strong anti-interference capabilities, and can significantly improve the accuracy of monitoring vital signs such as heart rate and respiratory rate in complex environments.

[0081] Optionally, in this embodiment of the application, feature extraction is performed on the original infrared thermal image sequence to obtain infrared thermodynamic feature data, including: Target region identification is performed on each frame of the original infrared sequence to obtain at least one target region. Target region identification refers to the process of automatically locating and segmenting specific regions related to physiological information in each frame of infrared thermal image through image processing or computer vision algorithms.

[0082] For example, each frame of the original infrared sequence can be processed using a pre-trained facial landmark detection model or image segmentation algorithm. This model or algorithm will accurately identify key facial features, such as the nasal alar region (the main outlet for respiratory airflow) and the forehead region (reflecting core body temperature changes). These identified regions are the target regions.

[0083] Obtain the temperature data of the target region in each frame. For each frame, calculate the average temperature value of all pixels within each target region. For example, calculate the average temperature of all pixels within the "nasal wing region" to obtain the temperature data of the target region, i.e., the temperature value of the "nasal temperature region".

[0084] A temperature waveform sequence is formed based on the temperature data of the same target area in each frame of the original infrared sequence. The temperature waveform sequence is used to characterize the waveform signal of the temperature change of the same target area over time. The temperature waveform sequence refers to the waveform signal formed by extracting and arranging the average temperature values ​​of the same target area in a series of temporally continuous infrared thermal image frames in chronological order.

[0085] The average temperature values ​​of the same target region (such as the nasal alar region) calculated for each frame in the sequence are arranged in order according to their corresponding timestamps to form a one-dimensional temperature waveform sequence. This waveform sequence characterizes the temperature change of that specific body region over time.

[0086] Thermodynamic features are extracted based on temperature waveform sequences to obtain infrared thermodynamic feature data. For example, respiratory heat flux fluctuation features are extracted: the temperature waveform in the nasal alar region exhibits obvious periodic fluctuations, the frequency of which is highly correlated with the respiratory rate. The respiratory rate can be extracted by performing spectral analysis or peak detection on this waveform.

[0087] Extracting temperature stability characteristics: Calculate the variance or standard deviation of the forehead temperature waveform over a period of time. This value may increase under stress or tension, reflecting skin temperature fluctuations caused by sympathetic nerve activity.

[0088] Extracting temperature trend features: Calculate the slope of the temperature waveform within a specific time window or perform linear fitting to determine whether the body temperature is rising, falling, or stable.

[0089] In the implementation of the above embodiments: by performing frame-by-frame target area identification and temperature data extraction on the original infrared sequence, the two-dimensional image sequence information is transformed into a one-dimensional temperature waveform sequence that can intuitively reflect the physiological state. Based on this waveform sequence, thermodynamic feature extraction can obtain feature data closely related to breathing patterns, blood circulation status, and autonomic nervous system activity, such as respiratory rate, temperature stability, and trends. These thermodynamic features provide an independent information dimension for health status assessment, distinct from radar physiological signals. For example, by analyzing the frequency of temperature fluctuations, cross-validation with the respiratory frequency extracted by radar can be performed, enhancing the robustness of respiratory monitoring; while the stability of forehead temperature changes can reflect sympathetic nervous system activity, thereby inferring stress or fatigue status. By fusing with other modal features, mutual verification and complementarity can be achieved, thereby comprehensively improving the accuracy and reliability of monitoring the driver's health status (such as stress and fatigue).

[0090] Understandably, the raw vibration signal sequence acquired by MEMS sensors contains both vehicle vibration signals and behavioral signals. In the "noise separation" stage, the goal is to extract heartbeat and breathing signals from the radar signal. Both vehicle vibration and behavioral signals are interference to the raw radio frequency signal sequence, contaminating the pure heartbeat signal. Therefore, we use the entire raw vibration signal sequence as reference noise to guide the filter in eliminating all vibration interference associated with it from the radar signal.

[0091] In the feature extraction stage, our goal is to retain behavioral data that reflects body activity. Therefore, it is necessary to filter out vehicle vibration signals from the original vibration signal sequence to obtain behavioral signals.

[0092] This application embodiment deeply mines the data value of the original vibration signal sequence. Sensor data is utilized twice, addressing two key issues: "denoising" and "sensing features." Optionally, in this embodiment of the application, feature extraction is performed on the original vibration signal sequence to obtain behavioral feature data, including: The vehicle vibration signal is filtered out from the original vibration signal sequence to obtain the behavioral signal. The behavioral signal refers to the vibration component that remains after filtering out the vehicle vibration signal, mainly generated by the physical activities of the driver or passenger (such as muscle tremors, center of gravity adjustment, posture changes, startle reactions, etc.).

[0093] The filtering process, for example, involves using a digital high-pass or band-pass filter. The filter's cutoff frequency is carefully set to preserve the higher-frequency vibration components (typically above a few hertz) generated by human activity while filtering out the low-frequency components associated with vehicle vibrations. The original vibration signal sequence is then convolved through this digital filter, directly outputting a behavioral signal from which the low-frequency vehicle vibration components have been filtered out.

[0094] Feature extraction is performed on the behavioral signals to obtain behavioral feature data. The purified behavioral signals are then analyzed to extract features characterizing the driver's behavioral state: Body activity level is extracted; the variance of the behavioral signals over a period of time is calculated. A larger variance value indicates more frequent body activity by the driver, potentially signifying restlessness, discomfort, or frequent posture adjustments.

[0095] Muscle tension can also be extracted: perform a Fast Fourier Transform on the behavioral signal to calculate its signal power in a specific frequency band (such as the band associated with muscle tremors, like 8-12 Hz). An increased power value may indicate that the muscle is under tension.

[0096] And extracting abnormal movement index: monitoring whether the absolute value of the amplitude of the behavioral signal exceeds a preset threshold. If it exceeds the threshold, an "abnormal movement event" (such as a sudden startle or a large body slide) may be recorded. The number of such events per unit time can be used as the abnormality index.

[0097] In the implementation of the above embodiments: by filtering out vehicle-related vibration noise from the original vibration signal, behavioral signals purely generated by the driver's physical activity were extracted. Based on this, behavioral characteristic data such as body activity level, muscle tension, and abnormal movement index were calculated, providing direct behavioral evidence for judging the driver's fatigue level, tension state, and sudden physical abnormalities. These characteristics quantify the driver's state from a behavioral perspective, effectively complementing physiological and thermodynamic characteristics. For example, sudden abnormal physical movements (such as startle reflexes) can be combined with a sudden increase in heart rate to more accurately judge sudden health events. This fusion of multi-dimensional information enriches the information sources for the system's comprehensive decision-making, thereby significantly enhancing the ability to perceive health states in complex real-world driving scenarios and comprehensively improving the accuracy and robustness of monitoring.

[0098] Optionally, in this embodiment of the application, physiological feature data, infrared thermodynamic feature data, and behavioral feature data are fused based on spatiotemporal correlation to obtain global features, including: Acquire time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data. For example, physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data with consistent timestamps can be obtained based on the timestamps of these three types of characteristic data.

[0099] Using time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data as nodes, a spatiotemporal correlation graph structure is constructed; the spatiotemporal correlation graph structure is used to characterize the temporal and / or spatial correlations between different types of characteristic data.

[0100] In a graph structure, a node represents the basic unit of an entity or data point. In this scheme, each feature triple (i.e., the set of physiological feature data, infrared thermodynamic feature data, and behavioral feature data at a given time point) is defined as a node in the graph.

[0101] Next, the edges are constructed. The edges in the spatiotemporal relationship graph structure include spatial edges and temporal edges. Spatial edges are the edges that connect the corresponding time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data within the same node. Temporal edges are the edges that connect different nodes in chronological order.

[0102] For spatial edges: Within each node, pairwise fully connected or attention-weighted connections are established between the three features, serving as spatial edges. For example, an edge connecting physiological feature data and infrared thermodynamic feature data within a node is a spatial edge; an edge connecting infrared thermodynamic feature data and behavioral feature data within a node is also a spatial edge; and an edge connecting physiological feature data and behavioral feature data within a node is also a spatial edge.

[0103] The weights of spatial edges can be pre-defined or dynamically calculated using a learnable attention mechanism. This mechanism evaluates the pairwise correlation strength between different modal features at the current moment. For example, if it learns a strong positive correlation between "increased heart rate" and "increased perinasal heat flow," then the weight of the spatial edge between them will be larger.

[0104] For time edges: Connecting the node at the current time t with nodes within a past time period (e.g., t-1) in chronological order forms a time edge. Since the timestamps of the three features within a node are already aligned, the timestamp of any feature within the node is the node's timestamp.

[0105] As one implementation method, the time edge can be further subdivided into edges connecting similar features. For example, for multiple nodes with continuous time (t, t-1, t-2, etc.), each node includes three features. The physiological feature data of multiple nodes are connected in chronological order using a first time edge; the infrared thermodynamic feature data and behavioral feature data of a single node are connected in chronological order using a second time edge and a third time edge, respectively. In this way, adjacent nodes are connected by three edges, with each edge corresponding to one of the three features.

[0106] The weights of the time edges can also be learned and used to characterize the strength of time dependence.

[0107] Feature fusion is performed based on a spatiotemporal correlation graph structure to obtain global features. For example, time-synchronized physiological feature data, infrared thermodynamic feature data, and behavioral feature data within the same node are aggregated to generate node features. Aggregation methods can include concatenation or addition. Multiple node features are then concatenated according to the temporal order between nodes to obtain global features.

[0108] The resulting global features can reflect different types of feature data that are related at the same time locally, and can reflect the changing patterns of different types of feature data over time globally.

[0109] As another implementation method, feature fusion can also be carried out using graph neural networks on a graph structure. Each node receives information from its neighboring nodes through its connected spatial and temporal edges to form new node information.

[0110] For example, for spatial edges, neighbors are features of other modalities within the same node. For instance, the physiological feature node V_physio might receive messages from the behavioral feature node V_behavior ("The body just swayed") and from the thermodynamic feature node V_thermal ("Face temperature is rising").

[0111] For time edges, neighbors are nodes with the same characteristics from previous time steps. For example, the current physiological characteristic node will receive a message from the physiological characteristic node from the previous time step ("heart rate has been rising in the past few seconds").

[0112] Aggregation functions, such as weighted summation, are used, where the weights are determined by attention weights on both spatial and temporal edges. A nonlinear transformation function (such as ReLU) is then used to generate a richer, updated feature representation of the node based on the new node information. This process can be performed in one or more rounds, allowing information to propagate further throughout the graph.

[0113] After multiple rounds of information propagation and updates, a global pooling operation is performed on the updated features of all nodes (or the nodes at the most recent time points). This can be done using attention pooling, assigning an importance weight to each node's features and then summing the results, or simply taking the average. The pooling result is the final output global feature vector.

[0114] In the implementation of the above embodiments: by constructing a spatiotemporal correlation graph structure containing spatial and temporal edges, multivariate time series data is transformed into a graph representation, explicitly modeling the spatiotemporal dependencies between multimodal features. Spatial edges capture the interactions between different modal features at the same time (such as the correlation between heart rate and body temperature), while temporal edges capture the changing patterns of the same feature over time (such as the trend changes in heart rate). This structure is superior to traditional sequence models (such as RNNs), enabling more flexible fusion of heterogeneous features and explicitly defining the temporal interactions between different modal features and the features themselves. It embeds prior physiological knowledge (such as the correlation between physiological signals and thermal signals) and data-driven temporal dependencies into the model structure, providing a comprehensive and highly fused feature vector for health monitoring.

[0115] As one example, the health detection model is a deep learning-based multi-task classifier, whose core input is the aforementioned fused global features. The health detection model can employ a deep neural network structure primarily composed of multi-layer fully connected networks with an attention mechanism. The global feature vector is first input into the model, undergoing nonlinear transformations and higher-order feature abstraction through multiple fully connected layers. Each layer is followed by an activation function (such as ReLU) to introduce nonlinearity and enhance the model's expressive power.

[0116] In this process, an attention mechanism is used to dynamically weight the importance of different feature dimensions, enabling the model to focus on the key information most relevant to health status. The model's output layer uses a softmax activation function, ultimately transforming it into probability distributions corresponding to different health states (such as normal, fatigued, and different stress levels). Through deep nonlinear mapping, the model learns a deep correspondence between health status and complex features from the fused global features.

[0117] Training a health monitoring model can be a supervised learning process. First, a large set of labeled training samples is needed, each containing a global feature vector and its corresponding true health status label (e.g., stress level determined by experts based on multimodal data). During training, the global features of the samples are input into the model to obtain prediction results; the cross-entropy loss between the prediction results and the true labels is calculated to quantify the current performance gap of the model. Backpropagation is used to propagate the loss gradient back through the entire network and the front-end feature fusion module, and optimizers such as Adam are used to dynamically adjust all parameters in the model (including weights and biases) to minimize the loss function. To prevent overfitting, regularization techniques such as Dropout and weight decay are often integrated during training. This iterative optimization process continues until the model loss converges, with the ultimate goal of enabling the model to accurately infer health or stress status from the fused features.

[0118] Please see Figure 2The diagram shown is a structural schematic of an in-vehicle health monitoring device provided in an embodiment of this application; this application provides an in-vehicle health monitoring device 200, including: The data acquisition module 210 is used to acquire continuous raw data through a multimodal acquisition module installed in the vehicle; the multimodal acquisition module includes a radar module, an infrared camera, and a MEMS sensor; the raw data includes raw radio frequency signal sequences acquired by the radar module, raw infrared thermal image sequences acquired by the infrared camera, and raw vibration signal sequences acquired by the MEMS sensor. The feature extraction module 220 is used to extract features from the original radio frequency signal sequence, the original infrared thermographic sequence, and the original vibration signal sequence to obtain physiological feature data, infrared thermodynamic feature data, and behavioral feature data. The feature fusion module 230 is used to fuse physiological feature data, infrared thermodynamic feature data and behavioral feature data based on spatiotemporal correlation to obtain global features. The health monitoring module 240 is used to input global feature data into the health detection model to obtain health monitoring results.

[0119] It should be understood that this device corresponds to the above-described vehicle health monitoring method embodiment and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0120] Please see Figure 3 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0121] Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a server or PC, or a virtual device, such as a virtual machine or virtualization container. Furthermore, electronic device 300 is not limited to a single device; it can be a combination of multiple devices or a cluster of numerous devices.

[0122] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0123] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0124] This application also provides a computer program product, including computer program instructions, which are executed by a processor to perform the method described above.

[0125] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0127] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A vehicle-mounted health monitoring method, characterized in that, include: Continuous raw data is collected by a multimodal acquisition module installed in the vehicle; the multimodal acquisition module includes a radar module, an infrared camera, and a MEMS sensor; the raw data includes the raw radio frequency signal sequence collected by the radar module, the raw infrared thermal image sequence collected by the infrared camera, and the raw vibration signal sequence collected by the MEMS sensor. Feature extraction is performed on the original radio frequency signal sequence, the original infrared thermographic sequence, and the original vibration signal sequence to obtain physiological feature data, infrared thermodynamic feature data, and behavioral feature data; The physiological feature data, infrared thermodynamic feature data, and behavioral feature data are fused based on spatiotemporal correlation to obtain global features; The global feature data is input into the health detection model to obtain health monitoring results; The physiological feature data, infrared thermodynamic feature data, and behavioral feature data are fused based on spatiotemporal correlation to obtain global features, including: Acquire the time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data; Using the time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data as nodes, a spatiotemporal correlation graph structure is constructed. This spatiotemporal correlation graph structure is used to characterize the temporal and / or spatial correlations between different types of characteristic data. The edges in the spatiotemporal correlation graph structure include spatial edges and temporal edges. Spatial edges are the edges connecting the corresponding time-synchronized physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data within the same node. Temporal edges are the edges connecting different nodes in chronological order. The spatial edges enable the health detection model to learn the interactions and constraints between the physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data at the same time, utilizing multimodal information to achieve consistent modeling and cross-validation. The temporal edges establish the causal relationships and evolution paths of the physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data over time, enabling the health detection model to perceive the dynamic trends of health status changes. The global features are obtained by performing feature fusion based on the spatiotemporal correlation graph structure. After acquiring continuous raw data via a multimodal acquisition module located in the vehicle, the method further includes: The data collected in each cycle of the original radio frequency signal sequence undergoes a Fast Fourier Transform (FFT) along the range dimension. The FFT transforms the original radio frequency signal from the time-sampling domain to the range-frequency domain, and the transformation result forms a "range profile." A peak appears at a specific range cell, and the distance corresponding to this peak is the distance between the radar and the target. A range gate containing information about chest cavity micro-movements is determined using the FFT. The signal phase information within the range gate is extracted. The extracted phase information from multiple consecutive cycles is arranged in chronological order to form a phase-time variation sequence. The sequence is demodulated and phase-unfolded to obtain a micro-Doppler timing signal. The micro-Doppler timing signal refers to a time-varying Doppler frequency shift signal caused by micro-movements in the human chest cavity, and the waveform fluctuations of the micro-Doppler timing signal correspond to the displacement changes in the chest cavity. Using the original vibration signal sequence that is time-synchronized with the original radio frequency signal sequence as reference noise, noise separation is performed on the micro-Doppler timing signal to obtain a purified micro-Doppler timing signal. The process of extracting features from the original radio frequency signal sequence to obtain physiological feature data includes: extracting features from the purified micro-Doppler time-series signal to obtain the physiological feature data.

2. The method according to claim 1, characterized in that, Using the original vibration signal sequence, which is time-synchronized with the original radio frequency signal sequence, as reference noise, noise separation is performed on the micro-Doppler timing signal to obtain a purified micro-Doppler timing signal, including: Based on the timestamp of the original radio frequency signal sequence, obtain the original vibration signal sequence that is time-synchronized with the original radio frequency signal sequence; The micro-Doppler time series signal is used as the main channel input of the adaptive filtering algorithm, and the time-synchronized original vibration signal sequence is used as the reference input channel of the adaptive filtering algorithm. The adaptive filtering algorithm learns the relationship between the main channel input and the reference input channel to obtain a predicted noise signal. The predicted noise signal is used to simulate the interference signal present in the micro-Doppler time series signal. Based on the predicted noise signal, noise separation is performed on the micro-Doppler time series signal to obtain the purified micro-Doppler time series signal.

3. The method according to claim 1, characterized in that, Feature extraction is performed on the original radio frequency signal sequence to obtain the physiological feature data, including: A short-time Fourier transform is performed on the original radio frequency signal sequence to transform it from the time domain to the time-frequency domain, generating a spectrum; the spectrum is used to reflect the distribution of the original radio frequency signal sequence over time and frequency. Based on the aforementioned spectrogram, the physiological characteristic data are obtained using peak detection algorithms and / or spectrum tracking algorithms; Feature extraction is performed on the original vibration signal sequence to obtain the behavioral feature data, including: The vehicle vibration signal is filtered out from the original vibration signal sequence to obtain the behavior signal; Feature extraction is performed on the behavioral signal to obtain the behavioral feature data.

4. The method according to claim 1, characterized in that, Feature extraction is performed on the original infrared thermal image sequence to obtain the infrared thermodynamic feature data, including: Target region identification is performed on each frame of the original infrared thermal image sequence to obtain at least one target region; Obtain the temperature data of the target region in each frame; A temperature waveform sequence is formed based on the temperature data of the same target area in each frame of the original infrared thermal image sequence; the temperature waveform sequence is used to characterize the waveform signal of the temperature change of the same target area over time. Thermodynamic features are extracted based on the temperature waveform sequence to obtain the infrared thermodynamic feature data.

5. The method according to claim 1, characterized in that, Feature fusion is performed based on the spatiotemporal correlation graph structure to obtain global features, including: The physiological characteristic data, infrared thermodynamic characteristic data, and behavioral characteristic data that are time-synchronized within the same node are aggregated to generate node features; The global feature is obtained by concatenating the features of multiple nodes in chronological order.

6. A computer program product, characterized in that, It includes computer program instructions that are executed by a processor to perform the method as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 5.

Images