Vehicle-mounted health monitoring method, interaction method, system and blood pressure estimation method

Abstract

The present application relates to the technical field of healthcare informatics, and specifically provides a vehicle-mounted health monitoring method, an interaction method, a system and a blood pressure estimation method.The method of the present application comprises: in response to a wake-up instruction of a user, acquiring a multi-dimensional data set; preprocessing the multi-dimensional data set to obtain a processed multi-dimensional data set; substituting the processed multi-dimensional data set into a fatigue and emergency identification module to obtain a health parameter and / or a fatigue parameter of the user; and substituting the health parameter and / or the fatigue parameter of the user into a risk prediction model to obtain a health medical recommendation for the user and an interaction depth matched with the current available cognitive resources of the user.Through multi-modal sensor fusion, physiological parameters and body composition data are synchronously collected, a gain adjustment compensation mechanism is set, the signal recognition accuracy is improved, the measurement accuracy is improved, visual fatigue monitoring is combined, the monitoring comprehensiveness and accuracy are improved, and overall evaluation of the health status is realized.

Description

Technical Field

[0001] This invention relates to the field of healthcare informatics technology, specifically to the field of vehicle-mounted health monitoring technology, and specifically provides a vehicle-mounted health monitoring method, interaction method, system, and blood pressure estimation method. Background Technology

[0002] As the level of automotive intelligence increases, users' demand for health management in in-vehicle scenarios is growing, especially for convenient and real-time health monitoring and risk warnings during driving. Existing in-vehicle health monitoring technologies face the following core problems: (1) Single monitoring dimension and poor data correlation: Existing vehicle health devices mostly monitor only single physiological parameters such as heart rate and blood oxygen, which cannot comprehensively assess the user's health status; and each parameter is collected independently, without establishing a collaborative analysis mechanism for multi-dimensional data, making it difficult to accurately identify potential health risks; (2) Insufficient adaptability to vehicle environment: Ordinary health monitoring equipment has problems such as low measurement accuracy and poor stability; (3) Insufficient remote collaboration and service extension: Most systems can only display monitoring data locally and cannot achieve efficient connection with cloud medical systems, making it difficult to obtain professional medical advice; they lack positioning, navigation and remote assistance functions in emergency situations, and cannot form a closed-loop service of "monitoring-early warning-rescue"; (4) Insufficient environmental adaptability and noise resistance: General filtering algorithms have difficulty distinguishing between signal fluctuations caused by vehicle bumps and real physiological abnormalities, resulting in a high false alarm rate under complex working conditions; (5) Static early warning logic: It often uses fixed thresholds for judgment, fails to capture the nonlinear phase transition characteristics of the human physiological system in a state of fatigue or disease, and lacks the ability to predict the trend of risk evolution.

[0003] Accordingly, there is a need in this field for a new in-vehicle multimodal health monitoring and intelligent interaction solution to address the above problems. Summary of the Invention

[0004] To overcome the above-mentioned shortcomings, this invention is proposed to provide an in-vehicle health monitoring method, interaction method, system, and blood pressure estimation method that solve or at least partially solve the technical problems of existing in-vehicle health monitoring technologies, such as single monitoring dimensions, poor data correlation, insufficient adaptability to the in-vehicle environment, insufficient remote collaboration and service extension, insufficient environmental adaptability and noise resistance, and static early warning logic.

[0005] In a first aspect, the present invention provides a vehicle-mounted multimodal health monitoring method, the method comprising: In response to the user's wake-up command, a multi-dimensional dataset is obtained, which includes at least health data and environmental data; Physiological gain adjustment was performed on the multi-dimensional dataset to obtain calibrated physiological feature parameters; Based on the multi-dimensional dataset, the confidence levels of each data source are obtained; The confidence levels of each data source and the calibrated physiological characteristic parameters are fused to obtain a processed multidimensional dataset, which is then used to obtain the user's multimodal health monitoring dataset.

[0006] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring method, the acquisition of multi-dimensional datasets includes: Acquire six parameters, including at least blood oxygen, blood pressure, respiration, pulse rate, electrocardiogram, and fingertip temperature; Acquire visual data, which includes at least facial data and eye data; Health data is obtained based on six-parameter data, body composition data, and visual data; Acquire environmental data, including at least in-vehicle temperature, humidity, traffic flow density, vehicle speed, road surface, illumination, and noise data, to obtain a multi-dimensional dataset.

[0007] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring method, the 18 body composition parameters include 18 data items: total cellular water, extracellular water, intracellular water, protein, inorganic salts, body fat, body fat percentage, skeletal muscle mass, muscle mass, basal metabolic rate, BMI, visceral fat level, waist-to-hip ratio, body age, standard weight, lean body mass, phase angle, and health score.

[0008] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring method, the step of adjusting the physiological gain of the multidimensional dataset to obtain calibrated physiological characteristic parameters includes: Physiological gain adjustment is performed on the body composition data in the multidimensional dataset to obtain the gain matrix. The diagonal elements of the gain matrix are the body composition compensation gain coefficients of the corresponding physiological characteristics, which are used to compensate for the influence of body composition differences on the measurement results of the corresponding physiological characteristics. Based on environmental parameters from a multi-dimensional dataset, determine the influence vector; The physiological feature parameters in the multidimensional dataset are calibrated based on the gain matrix, and the confidence of the calibrated physiological feature parameters is evaluated and cross-modal fusion is performed based on the influence vector to obtain the calibrated physiological feature parameters.

[0009] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring method, the method further includes: Based on the influence vector, the user's currently available cognitive resources are determined.

[0010] In a second aspect, the present invention provides a method for in-vehicle multimodal health monitoring and intelligent interaction, the method comprising: The user's multimodal health monitoring dataset in the vehicle-mounted multimodal health monitoring method described in the first aspect is substituted into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters; By inputting the user's health parameters and / or fatigue parameters into the risk prediction model, we can obtain health and medical advice for the user and the depth of interaction that matches the user's current available cognitive resources.

[0011] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring and intelligent interaction method, the step of substituting the user's multimodal health monitoring dataset into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters includes: A peripheral perfusion evaluation function is constructed based on six-parameter data from the user's multimodal health monitoring dataset. Based on the user's multimodal health monitoring dataset, determine the joint multiscale entropy function; By using the peripheral perfusion evaluation function and the joint multi-scale entropy function, the user's fatigue state and fatigue level are identified to obtain the user's fatigue parameters. And / or, Based on the user's multimodal health monitoring dataset, the user's health status parameters are obtained; Generate a health status assessment index for the user based on the user's health status parameters; Based on the user's health status assessment index, selectively generate and output the user's health parameters; The construction of the peripheral perfusion evaluation function includes: The temperature and humidity parameters in the environmental influence vector inside the vehicle cabin are acquired in real time. The temperature shift caused by the forced convection of the air conditioner is compensated by the thermal balance model to obtain the normalized fingertip temperature. The AC / DC amplitude ratio of the pulse wave signal was extracted, and combined with the pulse wave arrival time, normalized fingertip temperature, and blood oxygen saturation, a nonlinear evaluation function characterizing peripheral vascular resistance was constructed. The larger the value of the nonlinear evaluation function, the worse the peripheral perfusion and the higher the degree of sympathetic nerve stress activation.

[0012] In one technical solution of the aforementioned in-vehicle multimodal health monitoring and intelligent interaction method, the step of substituting the user's health parameters and / or fatigue parameters into the risk prediction model to obtain the user's health and medical advice and the interaction depth matching the user's currently available cognitive resources includes: The volume component data in the user's multimodal health monitoring dataset is mapped to a dynamic Bayesian network to obtain the initial state distribution parameters of the dynamic Bayesian network. Based on the initial state distribution parameters of the dynamic Bayesian network and the user's health parameters and / or fatigue parameters, multidimensional risk prediction data is obtained; and / or, the user's multimodal health monitoring dataset, body composition prior vector, and historical health state sequence are input into the deep neural network (DNN) risk prediction model to obtain multidimensional risk prediction data; wherein, the body composition prior vector is generated by preprocessing and vectorizing the body composition data collected by the body composition sensor, and the historical health state sequence is generated by the user's historical health state data stored in the vehicle's local memory and / or cloud server in chronological order; Personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-built medical knowledge graph to selectively generate medical suggestions, thus providing users with health and medical advice. Determine the depth of interaction with the user based on the user's currently available cognitive resources.

[0013] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring and intelligent interaction method, the deep neural network (DNN) risk prediction model is a cascaded structure of a temporal convolutional network (TCN) and a bidirectional long short-term memory (BiLSTM) network. It takes the user's multimodal health monitoring dataset, volume component prior vector, and historical health status sequence as input. The temporal convolutional network extracts local temporal features of the historical sequence, and the bidirectional long short-term memory network captures the long-term risk evolution trend, outputting the probability distribution of health risk level and the predicted value of the risk index at the next moment.

[0014] In one technical solution of the aforementioned in-vehicle multimodal health monitoring and intelligent interaction method, the personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-constructed medical knowledge graph to selectively generate medical suggestions, i.e., the user's health and medical advice includes: The multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data are mapped into personalized medical entity vectors through a modal encoder. Based on the semantic similarity between personalized medical entity vectors and knowledge graph nodes, and the in-vehicle scene weights determined by vehicle speed, traffic flow density, and environmental data, personalized subgraphs are retrieved and pruned. Multi-hop reasoning path scores are calculated on personalized subgraphs. The reasoning path with the highest score is selected and fused with clinical guidelines for similar cases before being input into a large language model to generate medical recommendations.

[0015] In one technical solution of the aforementioned in-vehicle multimodal health monitoring and intelligent interaction method, the personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-constructed medical knowledge graph to selectively generate medical suggestions, i.e., the user's health and medical advice includes: Map the multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data into personalized medical entity vectors; Based on personalized medical entity vectors, personalized subgraphs are obtained by retrieving and pruning from a pre-constructed medical knowledge graph according to semantic similarity and vehicle scene weight. Perform multi-hop path reasoning on the personalized subgraph to obtain the reasoning path and node confidence that match the user's current vehicle health status; If similar cases are found, then based on the personalized graph reasoning of the in-vehicle multimodal data, the knowledge graph reasoning path matching the user's current multimodal health monitoring dataset and the clinical guidelines corresponding to the similar cases are extracted. The knowledge graph reasoning path includes at least retrieving knowledge graph subgraphs associated with the user's body composition parameters, physiological characteristic parameters and environmental data, and LLM understanding the medical causes of the user's physiological characteristics. By integrating similar cases, knowledge graph reasoning paths, and corresponding clinical guidelines for similar cases, a fused medical dataset is obtained, and health and medical advice for users is generated based on the fused medical dataset. If no similar cases are found, the medical entity data is uploaded to the cloud server and the medical system, and medical advice is obtained from the medical system. Based on the medical advice provided by the medical system, health and medical advice for the user is generated.

[0016] In one technical solution of the above-mentioned vehicle-mounted multimodal health monitoring and intelligent interaction method, the method further includes: If the user's health parameters reach a preset emergency threshold, then the vehicle's location information is obtained; The vehicle's location information, along with the user's multi-dimensional dataset and health parameters, are uploaded to the cloud server and medical system. And send a remote distress signal to the medical system; Furthermore, the multi-dimensional datasets of each user and their health and medical advice are uploaded to the cloud server.

[0017] In a third aspect, the present invention provides an in-vehicle multimodal health monitoring and intelligent interaction system, the system comprising a perception layer, a communication layer, a processing layer, an interaction layer, and a cloud layer: The perception layer includes a multimodal sensor module, which is used to collect health data and environmental data, that is, to obtain the user's multidimensional dataset; The communication layer, including the communication module, is used for encrypted data transmission and command interaction; The processing layer includes a fatigue and emergency identification module, a risk prediction model, a wristband-less blood pressure estimation module, and a control module. These are used for data processing, user health risk prediction, and control between different layers of the system. The wristband-less blood pressure estimation module is used to estimate the user's blood pressure without a wristband. The control module includes a processor and a storage device. The storage device is adapted to store multiple lines of program code, which are adapted to be loaded and run by the processor to execute the vehicle-mounted multimodal health monitoring method described in any of the above technical solutions for the vehicle-mounted multimodal health monitoring method and the vehicle-mounted multimodal health monitoring and intelligent interaction method described in any of the above technical solutions for the vehicle-mounted multimodal health monitoring and intelligent interaction method. The interaction layer includes the in-vehicle interactive terminal module, which is used for interaction with users and information display; The cloud layer, including cloud servers and medical system modules, is used to enable information exchange between the system and the cloud server and medical system.

[0018] In a fourth aspect, the present invention provides a wristband-less blood pressure estimation method, applied to the wristband-less blood pressure estimation module in the aforementioned vehicle-mounted multimodal health monitoring and intelligent interaction system, the wristband-less blood pressure estimation method comprising: Acquire the user's pulse wave signal, wherein the pulse wave signal is obtained by a non-invasive vehicle-mounted sensor without a wristband; The pulse wave signal is preprocessed to extract pulse wave feature parameters, wherein the pulse wave feature parameters include at least the pulse wave arrival time, pulse wave amplitude ratio and pulse wave morphology features. The pulse wave feature parameters are input into a deep neural network (DNN) blood pressure model to obtain the user's blood pressure estimate. The blood pressure estimate is calibrated based on the user's body composition data to obtain the calibrated blood pressure value; The blood pressure model is trained based on pulse wave signal samples collected without a wristband. The deep neural network (DNN) blood pressure model integrates pulse wave feature parameters and body composition compensation vectors to eliminate the influence of individual body differences on blood pressure estimation.

[0019] In one technical solution of the above-mentioned wristband-less blood pressure estimation method, the deep neural network (DNN) blood pressure model is a cascaded structure of a one-dimensional convolutional neural network and a fully connected regression network. The one-dimensional convolutional neural network is used to extract local morphological features from the pulse wave feature sequence. The fully connected regression network maps the extracted features into a blood pressure estimation vector containing systolic and diastolic blood pressure. The ReLU activation function and Dropout regularization are used, and the blood pressure estimation vector is individually calibrated based on the body composition compensation mapping.

[0020] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects: By integrating multimodal sensors and simultaneously collecting physiological parameters and body composition data, combined with visual fatigue monitoring, the comprehensiveness and accuracy of monitoring have been improved, enabling a holistic assessment of health status. Multi-dimensional data collaborative analysis enhances the accuracy of risk identification and solves the problem of single monitoring dimensions in existing systems. Through automotive-grade hardware design and technologies such as ruggedization, electromagnetic interference resistance, and wide temperature range adaptation, the system is ensured to work stably in complex vehicle environments, making it highly adaptable to vehicle environments. It also adapts to vehicle power requirements with low-power management strategies, reducing energy consumption. By using voice interaction and a multimodal reminder mechanism, the number of touch operations during driving is reduced, the risk of distraction is lowered, and a balance is achieved between driving safety and interactive convenience. Through a multi-level warning design, abnormal information is ensured to be transmitted in a timely manner, taking into account both interactive efficiency and driving safety. By employing a communication data encryption mechanism, end-to-end encryption is achieved for data transmission and storage, ensuring data security and privacy compliance. Hierarchical access control and compliant design meet privacy protection requirements, addressing the data security deficiencies in existing systems. Through the collaborative architecture of "terminal-cloud-medical system", the system realizes the full-link service of health monitoring, early warning, assistance, and medical advice, achieving closed-loop service and scenario extension. The emergency positioning and remote assistance functions expand the system's emergency service capabilities and enhance users' sense of security. Through modular design, this system supports flexible replacement and upgrade of sensors and communication modules, achieving high compatibility and scalability. The open cloud API interface facilitates connection with more third-party medical services and expands the functional boundaries of the system. By setting up a gain adjustment compensation mechanism, the long-standing problem of vehicle-mounted electrode measurement being affected by the user's physique has been solved, improving signal recognition accuracy and measurement accuracy. By combining a deep neural network (DNN) blood pressure model with a wristband-less pulse wave blood pressure estimation algorithm, a 1D-CNN-FC cascade structure is used to achieve continuous estimation of wristband-less blood pressure. Individualized calibration is achieved by combining pulse wave feature parameters, which improves the accuracy and individual adaptability of continuous estimation of wristband-less blood pressure.

[0021] By calibrating the fingertip temperature ST using the temperature and humidity inside the vehicle, the false interference of the air conditioner on the peripheral temperature monitoring is eliminated, and the false judgment rate is reduced. This system achieves a complete in-vehicle closed-loop system from long-term physical fitness monitoring (BCA) to instantaneous physiological state (ECG / BP / SpO2), realizing comprehensive health management for users. By using medical knowledge graphs, the impact of the limitations of general large-scale models is reduced, the professional credibility of service recommendations is improved, and the recommendations provided by this system are supported by clinical medical records and knowledge graph logic, thereby enhancing the security of closed-loop services.

[0022] By using personalized graph reasoning based on in-vehicle multimodal data, physiological characteristics, body composition, and driving environment data are fused into personalized medical entity vectors. Weighted subgraph retrieval and multi-hop path reasoning are performed on the knowledge graph. This is different from medical question-answering systems that rely solely on general medical text queries, and improves the targeting and clinical relevance of medical advice in in-vehicle scenarios.

[0023] A DNN risk prediction model, which cascades a temporal convolutional network (TCN) and a bidirectional long short-term memory (BiLSTM) network, achieves deep learning prediction of health risk levels and evolution trends, forming a complementary protection with a dynamic Bayesian network. Attached Figure Description

[0024] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein: Figure 1 This is a schematic flowchart of the main steps of an in-vehicle multimodal health monitoring method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of the main steps of an in-vehicle multimodal health monitoring and intelligent interaction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the main structure of an in-vehicle multimodal health monitoring and intelligent interaction system according to an embodiment of the present invention. Detailed Implementation

[0025] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0027] Here we will first explain some of the terms involved in this invention.

[0028] NFC, Near Field Communication, is a short-range wireless communication technology. BMI, Body Mass Index; ECG, electrocardiogram; BP, blood pressure; PR, pulse rate; ST, fingertip temperature; DBN, Dynamic Bayesian Network; EHR, Electronic Health Record; LLM, Large Language Model; DNN, Deep Neural Network; TCN, Temporal Convolutional Network; BiLSTM, Bidirectional Long Short-Term Memory; 1D-CNN, One-Dimensional Convolutional Neural Network; CRC check, Cyclic Redundancy Check; NLB load balancing, network load balancing.

[0029] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of an in-vehicle multimodal health monitoring method according to an embodiment of the present invention. Figure 1 As shown, the vehicle-mounted multimodal health monitoring method in this embodiment of the invention mainly includes the following steps S101-S104.

[0030] Step S101: In response to the user's wake-up command, obtain a multi-dimensional dataset, wherein the multi-dimensional dataset includes at least health data and environmental data; Specifically, the user of this invention can be a driver.

[0031] Specifically, the user's wake-up command can be triggered by the user using a paired NFC device to perform an NFC touch, or by voice wake-up, or by pressing a button.

[0032] Specifically, obtaining the multi-dimensional dataset includes: Acquire six parameters, including at least blood oxygen, blood pressure, respiration, pulse rate, electrocardiogram, and fingertip temperature; Obtain body composition data, which includes at least 18 body composition parameters; Acquire visual data, which includes at least facial data and eye data; Health data is obtained based on six-parameter data, body composition data, and visual data; Acquire environmental data, which includes at least in-vehicle temperature, humidity, traffic flow density, vehicle speed, road surface, light intensity, and noise. To obtain a multi-dimensional dataset.

[0033] Specifically, the 18 body composition parameters include 18 data points: total cellular water, extracellular water, intracellular water, protein, inorganic salts, body fat, body fat percentage, skeletal muscle mass, muscle mass, basal metabolic rate, BMI, visceral fat level, waist-to-hip ratio, body age, standard weight, lean body mass, phase angle, and health score.

[0034] Step S102: Adjust the physiological gain of the multidimensional dataset to obtain the calibrated physiological feature parameters; Specifically, the physiological gain adjustment of the multi-dimensional dataset to obtain the calibrated physiological feature parameters includes: Physiological gain adjustment is performed on the body composition data in the multidimensional dataset to obtain the gain matrix. The diagonal elements of the gain matrix are the body composition compensation gain coefficients of the corresponding physiological characteristics, which are used to compensate for the influence of body composition differences on the measurement results of the corresponding physiological characteristics. Based on environmental parameters from a multi-dimensional dataset, determine the influence vector; The physiological feature parameters in the multidimensional dataset are calibrated based on the gain matrix, and the confidence of the calibrated physiological feature parameters is evaluated and cross-modal fusion is performed based on the influence vector to obtain the calibrated physiological feature parameters.

[0035] Specifically, the gain matrix It can be obtained through the following formula: In the formula, This refers to body composition analysis, which includes core parameters such as BMI, body fat percentage, and total cellular water content. Representing the The physical compensation gain coefficient corresponding to the physiological characteristics, where, This is used to compensate for the influence of physical differences on the measurement results of corresponding physiological characteristics, thereby normalizing the characteristics. It has physical adaptability, among which, Represents Body Mass Index. Represents body fat percentage. Represents total cellular water content and is used to assess skin conductivity. Representing the The preset nonlinear compensation mapping function corresponding to the dimensional physiological characteristics, and the calibrated 3rd... Dimensional physiological characteristics satisfy ; In this embodiment, the system uses body composition data (such as body fat percentage) to generate a gain matrix. This means that for users with higher body fat percentages, the original ECG signal may be weaker. BC-Gain automatically amplifies the signal gain, ensuring that the physiological characteristics entering the final fusion stage are data that have been "faired" and calibrated.

[0036] Specifically, the influence vector It can be obtained through the following formula: in, These represent normalized vehicle speed, longitudinal acceleration, and lateral acceleration, respectively, used to reflect the physical contribution of driving maneuvers to the heart rate baseline. Represents road surface curvature, used to identify stress-induced physiological fluctuations caused by driving on curves. The two terms, traffic flow density and the reciprocal of collision time, respectively, are used to characterize the external cognitive load of the current driving task. These represent the cabin lighting variations and background noise levels, respectively, used to correct the signal confidence levels of visual and acoustic sensors. Represents the cabin temperature. Represents humidity and is used to correct for the reference offset of fingertip temperature (ST) under thermodynamic equilibrium. In this embodiment, the measurement benchmarks of physiological indicators such as heart rate and blood oxygen are corrected by environmental parameters to eliminate environmental interference. By combining the reciprocal of the time to collision (TTC) and traffic flow density, dynamic assessment of driving risk level is achieved. The heterogeneous consistency of multi-parameter measurements is ensured by introducing a "volume composition mask" to adjust the gain of the original physiological vector.

[0037] Step S103: Based on the multi-dimensional dataset, obtain the confidence level of each data source; Specifically, the confidence levels obtained from each data source based on the multi-dimensional dataset include: Based on a multidimensional dataset, the confidence level of each data source in the multidimensional dataset under the current operating conditions is dynamically evaluated through a gating unit.

[0038] Specifically, the formula for the gating unit is: In the formula, Physiological feature vectors representing inclusion body components and six-parameter data adjusted by BC-Gain. Represents the Sigmoid activation function. , , representing the learnable weight matrix and bias term of the gated unit, respectively. The confidence vectors of each sensing mode output by the gating unit; This represents the concatenation result of the influence vector and the physiological feature vector. The output dimension is consistent with the dimension of the feature to be fused, and is used to dynamically evaluate the reliability of each sensing mode. In this embodiment, by Real-time assessment of the reliability of each sensing mode under current operating conditions suppresses noise interference in the data.

[0039] Step S104: The confidence scores of each data source and the calibrated physiological characteristic parameters are fused to obtain the processed multidimensional dataset, which is used to obtain the user's multimodal health monitoring dataset.

[0040] Specifically, the confidence levels of each data source and the calibrated physiological feature parameters are fused to obtain the processed multi-dimensional dataset, which includes: Data fusion is achieved using the following formula: In the formula, This refers to the confidence vectors of each sensing mode output by the gating unit in step S103. ⊙ represents element-wise multiplication, used to achieve dynamic fusion of the gating signal and the attention result. Represents a multi-head attention operator, through query items (Obtained through influence vectors) and perform deep alignment with physiological key values, i.e., physiological feature pairs. Represents the bond matrix. The representative value matrix is ​​used to construct cross-modal attention maps. Represents the calibrated physiological characteristic parameters, where, , Represents physiological characteristic parameters before calibration; In this embodiment, the signal gain is automatically adjusted for users with different body types using a gain matrix, which solves the measurement bias of traditional systems for users with high body fat. "Adaptive noise reduction" is achieved at the feature layer using body composition data, which solves the problem of large accuracy fluctuations of traditional formulas for users with different body types. The semantic association between driving environment parameters and physiological characteristics is achieved through a multi-head attention mechanism, which improves the accuracy of data fusion.

[0041] Specifically, the vehicle-mounted multimodal health monitoring method further includes: Based on the influence vector, the user's currently available cognitive resources are determined, wherein the types of the user's currently available cognitive resources include at least the voice interaction channel, the vibration interaction channel, and the restriction control channel; Furthermore, the different types of currently available cognitive resources for users, from highest to lowest priority, are: voice interaction channel, vibration interaction channel, and restrictive control channel.

[0042] See appendix Figure 2 , Figure 2 This is a schematic flowchart illustrating the main steps of an in-vehicle multimodal health monitoring and intelligent interaction method according to an embodiment of the present invention. Figure 2 As shown, the vehicle-mounted multimodal health monitoring and intelligent interaction method in this embodiment of the invention mainly includes the following steps S201-S202.

[0043] Step S201: Substitute the user's multimodal health monitoring dataset from the above vehicle-mounted multimodal health monitoring method into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters; Specifically, the step of inputting the user's multimodal health monitoring dataset into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters includes: A peripheral perfusion evaluation function is constructed based on six-parameter data from the user's multimodal health monitoring dataset. Based on the user's multimodal health monitoring dataset, determine the joint multiscale entropy function; By using the peripheral perfusion evaluation function and the joint multi-scale entropy function, the user's fatigue state and fatigue level are identified to obtain the user's fatigue parameters. And / or, Based on the user's multimodal health monitoring dataset, the user's health status parameters are obtained; Generate a health status assessment index for the user based on the user's health status parameters; Based on the user's health status assessment index, selectively generate and output the user's health parameters; The construction of the peripheral perfusion evaluation function includes: The temperature and humidity parameters in the environmental influence vector inside the vehicle cabin are acquired in real time. The temperature shift caused by the forced convection of the air conditioner is compensated by the thermal balance model to obtain the normalized fingertip temperature. The AC / DC amplitude ratio of the pulse wave signal was extracted, and combined with the pulse wave arrival time, normalized fingertip temperature, and blood oxygen saturation, a nonlinear evaluation function characterizing peripheral vascular resistance was constructed. The larger the value of the nonlinear evaluation function, the worse the peripheral perfusion and the higher the degree of sympathetic nerve stress activation.

[0044] Specifically, the peripheral infusion evaluation function is implemented through the following formula: In the formula, It represents the peripheral circulation resistance factor, used to quantify peripheral vascular perfusion status; This represents the normalized AC / DC amplitude ratio of the pulse wave. Represents the normalized arrival time of the pulse wave. This represents the normalized fingertip temperature after correction for the vehicle interior temperature. Represents normalized blood oxygen saturation; to These are non-negative weight coefficients, and their sum is 1. Each weight is determined through calibration data. The larger the value, the higher the peripheral circulation resistance and the worse the peripheral perfusion.

[0045] Specifically, when the system detects an increase in pulse rate (PR) and an abnormal decrease in fingertip temperature (ST), the system automatically identifies the user's fatigue state and fatigue level as "stress-induced vasoconstriction" and "high," which has a higher priority than conventional fatigue identification.

[0046] Specifically, the joint multiscale entropy function is achieved through the following formula: In the formula, Represents the complexity index of cardiopulmonary coordination; and These represent the pulse rate sequence and the respiratory rate sequence, respectively; τ represents the time scale factor. This represents the multivariate sample entropy calculated from the dual-channel sequences of pulse rate and respiratory rate at a time scale τ. The maximum time scale is denoted by m; m and r represent the embedding dimension and similarity tolerance, respectively. In this embodiment, m=2 and r=0.2SD, where SD is the standard deviation of the standardized dual-channel data.

[0047] In this embodiment, the temperature reference is corrected by the in-vehicle temperature to eliminate the influence of ambient temperature on physiological parameters and improve measurement reliability. The deep fatigue state is identified by assessing the "complexity of cardiopulmonary coordination", which improves the fatigue recognition accuracy and thus improves the system's early warning accuracy.

[0048] Step S202: Substitute the user's health parameters and / or fatigue parameters into the risk prediction model to obtain the user's health and medical advice and the interaction depth that matches the user's current available cognitive resources.

[0049] Specifically, the step of substituting the user's health parameters and / or fatigue parameters into the risk prediction model to obtain the user's health and medical advice and the depth of interaction matching the user's currently available cognitive resources includes: The volume component data in the user's multimodal health monitoring dataset is mapped to a dynamic Bayesian network to obtain the initial state distribution parameters of the dynamic Bayesian network. Based on the initial state distribution parameters of the dynamic Bayesian network and the user's health parameters and / or fatigue parameters, multidimensional risk prediction data is obtained; and / or, the user's multimodal health monitoring dataset, body composition prior vector, and historical health status sequence are input into the deep neural network (DNN) risk prediction model to obtain multidimensional risk prediction data. Personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-built medical knowledge graph to selectively generate medical suggestions, thus providing users with health and medical advice. Determine the depth of interaction with the user based on the user's currently available cognitive resources.

[0050] Specifically, before inputting the user's multimodal health monitoring dataset, body composition prior vector, and historical health status sequence into the deep neural network (DNN) risk prediction model, the method further includes: Obtain the body component prior vector and the historical health state sequence.

[0051] Specifically, the body composition prior vector is generated based on body composition data collected by a body composition sensor. Specifically, the body composition sensor collects the user's body composition data, which includes at least one or more of BMI, body fat percentage, total cellular water, muscle mass, and basal metabolic rate. The processing layer performs validity checks, fills in missing values, and normalizes the body composition data. Then, it maps the processed body composition data into a fixed-dimensional vector using a preset body composition encoding function or a fully connected mapping network to obtain the body composition prior vector.

[0052] Specifically, the historical health status sequence is generated based on the user's historical health status data. Specifically, after each multimodal health monitoring cycle, the system stores the corresponding health status parameters, health status assessment index, fatigue parameters, and / or risk level at the corresponding moment in the vehicle's local storage and / or cloud server, according to the user's identifier and timestamp. When performing DNN risk prediction, the processing layer reads the health status parameters, health status assessment index, fatigue parameters, and / or risk level from the L most recent sampling moments before the current moment and encodes them in chronological order into a historical health status sequence of length L. If the historical data length is less than L, it is padded using the user's current health status parameters, historical averages, or preset steady-state values.

[0053] Specifically, the volume component data in the user's multimodal health monitoring dataset is mapped to a dynamic Bayesian network, and the initial state distribution parameters of the dynamic Bayesian network are obtained, including: Mapping volume component data to the initial state distribution of DBN This results in users with high body fat and low metabolism being assigned a higher baseline cardiovascular risk weight.

[0054] Specifically, the multidimensional risk prediction data is obtained through the following formula: In the formula, This represents the driver's hidden health state at time t, which is one of the following: steady state, compensated state, or decompensated state. Sensor observation evidence representing time t, including instantaneous blood pressure and HRV; It represents the sequence of sensor observation evidence up to time t; The volume component prior vector at time t represents the volume component prior vector. Represents the prior sequence of volume components up to time t; Represents the observation likelihood function; Represents the state transition probability modulated a priori by the current volume component; This represents the posterior distribution at the previous time step. The posterior probabilities of the three health hidden states and their weighted risk values ​​obtained by weighting them according to state risk weights are used as multidimensional risk prediction data.

[0055] In this embodiment, the temporal evolution of a user's health status is modeled by combining the user's historical state with the current observed state; the state transition probability in the dynamic Bayesian network is affected by... Dynamic modulation enables continuous health risk prediction tailored to individual body constitutions. As a parallel prediction path, the deep neural network (DNN) risk prediction model employs a cascaded structure of a temporal convolutional network (TCN) and a bidirectional long short-term memory (BiLSTM) network: the TCN extracts historical health state sequences through dilated causal convolution. The local time series pattern is captured by BiLSTM, which bidirectionally encodes the TCN output to capture long-term risk evolution trends. Softmax outputs the probability distribution of health risk levels. Then Weighted according to the risk level quantification vector L, and compared with the current comprehensive risk index. By combining the results, we can obtain the predicted value of the comprehensive risk index for the next moment. .

[0056] Specifically, the DNN risk prediction model is implemented through the following formula: In the formula, Let be the probability distribution of health risk levels at time t, where the levels include at least low risk, medium risk, and high risk. A sequence of historical health states of length L; The multimodal observation vector at time t contains six-parameter data, volume composition data, and fatigue parameters; The square brackets represent the prior vectors of volume components obtained from the volume component data after validity verification, normalization, and vectorization mapping. The representation is feature concatenation; TCN(·) is a temporal convolutional network, and BiLSTM(·) is a bidirectional long short-term memory network; and For output layer parameters; L is the risk level quantization vector; This is the current comprehensive risk index. This is the predicted value of the comprehensive risk index for the next moment; To satisfy 0≤ A fusion weight of ≤1. This DNN path complements the dynamic Bayesian network path.

[0057] Specifically, the personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-built medical knowledge graph, selectively generating medical suggestions, i.e., obtaining health and medical advice for the user, including: The multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data are mapped into personalized medical entity vectors through a modal encoder. Based on the semantic similarity between personalized medical entity vectors and knowledge graph nodes, and the in-vehicle scene weights determined by vehicle speed, traffic flow density, and environmental data, personalized subgraphs are retrieved and pruned. Multi-hop reasoning path scores are calculated on personalized subgraphs. The reasoning path with the highest score is selected and fused with clinical guidelines for similar cases before being input into a large language model to generate medical recommendations.

[0058] or, Specifically, the personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-built medical knowledge graph, selectively generating medical suggestions, i.e., obtaining health and medical advice for the user, including: Map the multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data into personalized medical entity vectors; Based on personalized medical entity vectors, personalized subgraphs are obtained by retrieving and pruning from a pre-constructed medical knowledge graph according to semantic similarity and vehicle scene weight. Perform multi-hop path reasoning on the personalized subgraph to obtain the reasoning path and node confidence that match the user's current vehicle health status; If similar cases are found, then based on the personalized graph reasoning of the in-vehicle multimodal data, the knowledge graph reasoning path matching the user's current multimodal health monitoring dataset and the clinical guidelines corresponding to the similar cases are extracted. The knowledge graph reasoning path includes at least retrieving knowledge graph subgraphs associated with the user's body composition parameters, physiological characteristic parameters and environmental data, and LLM understanding the medical causes of the user's physiological characteristics. By integrating similar cases, knowledge graph reasoning paths, and corresponding clinical guidelines for similar cases, a fused medical dataset is obtained, and health and medical advice for users is generated based on the fused medical dataset. If no similar cases are found, the medical entity data is uploaded to the cloud server and the medical system, and medical advice is obtained from the medical system. Based on the medical advice provided by the medical system, health and medical advice for the user is generated.

[0059] Specifically, personalized graph reasoning based on in-vehicle multimodal data differs from general medical question-answering systems. Its core lies in using the user's real-time in-vehicle multimodal data as personalized input for graph retrieval and reasoning, rather than relying solely on general medical text queries. Personalized graph reasoning includes the following four steps: Step 1, Personalized Entity Mapping: Mapping physiological feature parameters from the multimodal health monitoring dataset. Body composition parameters and environmental data The corresponding modal encoders are input separately, and the resulting vectors are fused to obtain personalized medical entity vectors. .

[0060] Step 2, Personalized Subgraph Retrieval: Calculation Embedded vectors of each node in the knowledge graph Weighted semantic similarity, and weighted according to in-vehicle scenario. To correct this, nodes with similarity exceeding a threshold τ and their associated edges are retrieved, and the resulting subgraph is pruned to obtain a personalized subgraph. .

[0061] Step 3, Multi-hop Path Reasoning: In Starting from the abnormal sign node, the algorithm expands along the disease-symptom-intervention relationship edge in multiple hops, calculates the cumulative score of each reasoning path, and selects the K paths with the highest scores as the knowledge graph reasoning paths.

[0062] Step 4, LLM Fusion Generation: Integrating inference paths, clinical guidelines for similar cases, and... Concatenate the information into a context, input it into a large language model (LLM), and generate health and medical recommendations that include diagnostic conclusions, intervention strategies, and follow-up suggestions.

[0063] Specifically, personalized graph inference is achieved through the following formula: In the formula, Embed(·) is a modal encoder, and the encoding results of each modality have the same dimension; α, β, and γ are modal fusion weights, which satisfy α+β+γ=1; For knowledge graph nodes The embedding vector; For nodes Scene relevance weights under current vehicle speed, traffic flow density, and cabin environment conditions; τ is the similarity threshold; p is the multi-hop inference path starting from the abnormal node; Let (u, v) be the clinical association weight of the edge. Let node v be the vector of the current user entity. The degree of matching. This represents a personalized subgraph consisting of nodes with similarity exceeding a threshold and their associated edges.

[0064] Specifically, through the aforementioned personalized graph reasoning, the system performs differentiated medical knowledge retrieval and reasoning based on in-vehicle multimodal data, improving the matching degree between medical advice and the user's current in-vehicle health status.

[0065] Specifically, the pre-built medical knowledge graph is constructed through the following steps: Access to electronic health records and clinical guidelines; Extract triplet information (disease, manifestation, threshold) from electronic health records (EHRs) and clinical guidelines to construct a hierarchical medical knowledge graph.

[0066] Specifically, similar cases are retrieved from the EHR case database using the following formula: In the formula, This represents the current medical entity data. By querying the encoder The d-dimensional query vector obtained by mapping; This represents the EHR case document. Through document encoder The resulting d-dimensional case document vector; and Located in the same embedding space; sim(·) represents the semantic similarity score, which retrieves the top k similar cases in the EHR case database using cosine similarity. Retrieval of the medical knowledge graph is performed separately based on medical entity matching, subgraph retrieval, and path reasoning.

[0067] Specifically, the extraction of knowledge graph reasoning paths and clinical guidelines corresponding to similar cases includes: After identifying similar medical records, the system extracts related reasoning paths from the knowledge graph. For example, if it detects... If an anomaly is detected, the "autonomic nervous system-circulatory system" subgraph is automatically triggered to assist the subsequent large-scale language model LLM in understanding the medical causes of the user's state. Based on the identified similar cases, obtain the corresponding clinical guidelines.

[0068] Specifically, similar cases, knowledge graph reasoning paths, and corresponding clinical guidelines for similar cases are integrated to obtain a fused medical dataset. Based on this fused medical dataset, health and medical recommendations for users are generated, including: The retrieved clinical guidelines, similar medical records, and knowledge graph reasoning paths are fused together to obtain a fused medical dataset. Based on a fusion medical dataset, LLM is used to generate health and medical recommendations for users. These recommendations include "diagnostic conclusions, intervention strategies, recommended medications, and follow-up reminders," thereby reducing the risk of medical illusions through this system.

[0069] Specifically, determining the depth of interaction with a user based on their currently available cognitive resources includes: When the highest priority type of the user's currently available cognitive resources is the voice interaction channel, the interaction depth with the user is determined to be level one, which is silent display + voice prompt. When the highest priority type of the user’s currently available cognitive resources is the vibration interaction channel, the interaction depth with the user is determined to be level two, namely voice broadcast + seat vibration + steering wheel vibration. When the highest priority type of the user's currently available cognitive resources is the restrictive control channel, the interaction depth with the user is determined to be level three, namely voice broadcast + seat vibration + steering wheel vibration + automatic speed limit + navigation to the nearest service area and health risk assessment. This allows for dynamic adjustment of the intervention depth of the interaction layer.

[0070] Specifically, the method further includes: If the user's health parameters reach a preset emergency threshold, then the vehicle's location information is obtained; The vehicle's location information, along with the user's multi-dimensional dataset and health parameters, are uploaded to the cloud server and medical system. And send a remote distress signal to the medical system; Furthermore, the multi-dimensional datasets of each user and their health and medical advice are uploaded to the cloud server.

[0071] Specifically, the preset emergency threshold includes different data parameter types, and the threshold value corresponding to each data type can be set by those skilled in the art according to the actual situation.

[0072] Based on steps S101-S104 and S201-S202, multimodal sensor fusion is used to simultaneously collect physiological parameters and body composition data. Combined with visual fatigue monitoring, this improves the comprehensiveness and accuracy of monitoring, enabling a holistic assessment of health status. Voice interaction combined with a multimodal reminder mechanism reduces touch operations during driving, mitigating the risk of distraction and achieving a balance between driving safety and ease of interaction. A multi-level warning design ensures timely transmission of abnormal information, balancing interaction efficiency and driving safety. Through a collaborative architecture of "terminal-cloud-medical system," a full-link service encompassing health monitoring, early warning, assistance requests, and medical advice is achieved, realizing closed-loop service and scenario extension. The emergency location and remote assistance functions expand the system's emergency service capabilities and enhance user safety. By setting up a gain adjustment compensation mechanism, it solves the long-standing pain point that vehicle electrode measurements are affected by the user's physique, improves signal recognition accuracy, and enhances measurement accuracy. By using a medical knowledge graph, it reduces the impact of the limitations of general large model knowledge, improves the professional credibility of service suggestions, and enables the suggestions provided by this system to have clinical medical record support and knowledge graph logic, thereby improving the security of closed-loop services. It also solves the technical problems of existing vehicle health monitoring technologies, such as single monitoring dimensions, poor data correlation, insufficient adaptability to vehicle environment, insufficient remote collaboration and service extension, insufficient environmental adaptability and noise resistance, and static warning logic.

[0073] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0074] Furthermore, the present invention also provides an in-vehicle multimodal health monitoring and intelligent interaction system.

[0075] See appendix Figure 3 , Figure 3 This is a main structural block diagram of an in-vehicle multimodal health monitoring and intelligent interaction system according to an embodiment of the present invention. Figure 3 As shown, the in-vehicle multimodal health monitoring and intelligent interaction system in this embodiment of the invention mainly includes a perception layer, a communication layer, a processing layer, an interaction layer, and a cloud layer: The perception layer includes a multimodal sensor module, which is used to collect health data and environmental data, that is, to obtain the user's multidimensional dataset; The communication layer, including the communication module, is used for encrypted data transmission and command interaction; The processing layer includes a fatigue and emergency identification module, a risk prediction model, and a control module. It is used for data processing, user health risk prediction, and control between different layers of the system. The control module includes a processor and a memory. The memory can be configured to store programs for executing the vehicle-mounted multimodal health monitoring method and the vehicle-mounted multimodal health monitoring and intelligent interaction method described in the above-described method embodiments. The processor can be configured to execute the programs in the memory, including but not limited to programs for executing the vehicle-mounted multimodal health monitoring method and the vehicle-mounted multimodal health monitoring and intelligent interaction method described in the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This control module can be a control module device comprising various electronic devices. The interaction layer includes the in-vehicle interactive terminal module, which is used for interaction with users and information display; The cloud layer, including cloud servers and medical system modules, is used to enable information exchange between the system and the cloud server and medical system.

[0076] Specifically, the multimodal sensor module includes a six-parameter physiological sensor, a body composition sensor, and a visual sensor: The six-parameter physiological sensor can use custom metal electrode sheets, which can be snap-fit ​​design with welded connectors to achieve the detection of six parameters: blood oxygen, blood pressure, respiration, pulse rate, electrocardiogram, and fingertip temperature. Its power supply voltage is +5V, and it can use immersion gold process + three-proof reinforcement to adapt to the complex vehicle environment. The body composition sensor can use food-grade stainless steel electrodes with a contact area ≥5cm². 2 It is divided into excitation current electrode (CIO) and voltage drop measurement electrode (VIO). Its board is made of plexiglass cloth and its power supply voltage is +5V, so as to realize the detection of at least 18 volume composition parameters and to determine the validity of the data through verification. The visual sensor can use a camera to achieve facial recognition, which is user identity authentication and fatigue monitoring. It can determine the fatigue level by observing changes in pupil size and the frequency of eye closure. In addition, the lens is adapted to the lighting environment inside the vehicle and supports low-light shooting.

[0077] Specifically, the communication module can be a Bluetooth + mobile network communication module, responsible for encrypted data transmission and command interaction. Its core components are a Bluetooth communication board and a 4G module. The Bluetooth communication board can be a Bluetooth communication board that supports BLE Bluetooth communication, stable transmission within a 5-meter range, and communication latency with the PAD <100ms. It communicates with the six-parameter and volume composition sensor boards via serial ports to provide isolated power to ensure stable operation of the sensors. The 4G communication module can be a standard 4G module, which supports high-speed data interaction with the cloud server to enable health data uploading, medical advice reception and emergency location information transmission. Its communication protocol uses HTTPS encryption.

[0078] Specifically, the fatigue and emergency identification module, risk prediction model, and control module can employ integrated high-performance processors to achieve multimodal data fusion analysis. The fatigue and emergency identification module and risk prediction model can utilize running health analysis algorithms and anomaly detection models, supporting fatigue level classification: Level 1: voice reminder; Level 2: seat vibration + steering wheel vibration; Level 3: automatic speed limit + navigation to the nearest service area and health risk assessment. The control module is responsible for preprocessing sensor signals, including at least filtering, amplification, CRC check / checksum verification, to enable button and touch responses and coordinate the timing synchronization of various modules. The control module's memory can employ an integrated encrypted storage chip, using the AES algorithm for local backup of health data to support historical data retrieval. The memory also stores health status parameters, health status assessment indices, risk levels, and body composition data generated from each health monitoring session according to user identifiers and timestamps, for the risk prediction model to generate historical health status sequences and body composition prior vectors.

[0079] Specifically, the in-vehicle interactive terminal module can adopt an automotive-grade PAD or automotive-grade screen, which supports WiFi, 4G and Bluetooth communication, and integrates a camera, speaker and microphone to meet the needs of facial recognition and voice interaction. It runs a customized APP to realize account management, data display, voice interaction and multimodal reminders, and supports NFC wake-up, which improves the ease of use of the system.

[0080] Specifically, the cloud server and medical system module can adopt a three-layer architecture of "access layer - service layer - data layer": Access layer: Deploy NLB load balancer and WAF web application firewall to achieve secure communication between the terminal and the cloud via HTTPS and WebSocket protocols; Service layer: The business cluster is built based on Docker + Spring Boot, providing services such as authentication, data statistics and analysis, medical advice push, and emergency help response. It integrates Redis caching and RocketMQ message queue to improve system response speed. Data layer: Adopts MySQL master-slave architecture to achieve secure storage and backup of health data, and connects to third-party medical system APIs to achieve data sharing and professional diagnostic support.

[0081] Specifically, the system's process includes: System wake-up and identity authentication: Users wake up the system via NFC touch or voice, the camera collects facial information to complete identity authentication, and the APP automatically loads the user's health record; Multimodal data acquisition: The system simultaneously acquires physiological parameters and body composition data through a six-parameter sensor and a body composition sensor, while the camera monitors the user's fatigue state in real time; the sensor data is preprocessed by the acquisition board and then transmitted to the PAD via Bluetooth encryption. Data processing and analysis: The core board runs AI algorithms to integrate multi-dimensional data for health status assessment and fatigue level determination; if health parameters exceed the standard or fatigue status is detected, a multi-level reminder mechanism is triggered; Interaction and Service Output: The PAD APP displays real-time health data and analysis results, and delivers reminders through voice, screen pop-ups, and seat vibration; users can query historical data or health advice via voice. Data Upload and Cloud Collaboration: Health data is encrypted and uploaded to the cloud server via a 4G module. The cloud stores historical data and pushes personalized health advice. In emergencies, the system automatically uploads vehicle location information, initiates remote assistance, and connects to the medical system. Low-power sleep mode: After monitoring is completed, the system automatically controls the sensor module to enter sleep mode, keeping only the core communication components in standby mode to reduce vehicle power consumption.

[0082] In one implementation, the specific function can be described in steps S101-S104.

[0083] The aforementioned vehicle-mounted multimodal health monitoring and intelligent interaction system is used to perform Figure 1 The vehicle-mounted multimodal health monitoring method and Figure 2 The embodiments of the vehicle-mounted multimodal health monitoring and intelligent interaction method shown are similar in technical principle, the technical problems solved, and the technical effects produced. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the vehicle-mounted multimodal health monitoring and intelligent interaction system can be found in the embodiments of the vehicle-mounted multimodal health monitoring and intelligent interaction method, and will not be repeated here.

[0084] Furthermore, the present invention also provides a wristband-less blood pressure estimation method. The wristband-less blood pressure estimation method is applied to the wristband-less blood pressure estimation module in the aforementioned in-vehicle multimodal health monitoring and intelligent interaction system, and the method includes: Acquire the user's pulse wave signal, wherein the pulse wave signal is obtained by a non-invasive vehicle-mounted sensor without a wristband; The pulse wave signal is preprocessed to extract pulse wave feature parameters, wherein the pulse wave feature parameters include at least the pulse wave arrival time, pulse wave amplitude ratio and pulse wave morphology features. The pulse wave feature parameters are input into a deep neural network (DNN) blood pressure model to obtain the user's blood pressure estimate. The blood pressure estimate is calibrated based on the user's body composition data to obtain the calibrated blood pressure value; The blood pressure model is trained based on pulse wave signal samples collected without a wristband. The deep neural network (DNN) blood pressure model integrates pulse wave feature parameters and body composition compensation vectors to eliminate the influence of individual body differences on blood pressure estimation.

[0085] Specifically, the system collects the user's pulse wave signal through an in-vehicle non-invasive sensor, eliminating the need to wear a wrist-worn blood pressure monitor; the pulse wave signal is filtered and features are extracted to obtain the pulse wave arrival time (PAT), AC / DC amplitude ratio, and pulse wave morphology characteristics, which constitute the pulse wave feature vector (PW).

[0086] Specifically, the DNN blood pressure model estimates blood pressure using the following formula: In the formula, This is the calibrated blood pressure estimation vector. For systolic pressure, 1 represents diastolic blood pressure; PW is the pulse wave characteristic vector, which includes at least the pulse wave arrival time (PAT), AC / DC amplitude ratio, and morphological characteristics. This is the cascade mapping function between a one-dimensional convolutional neural network and a fully connected regression network; The volume component prior vector; θ represents the individualized blood pressure compensation term obtained from the body component prior vector; θ is the model learnable parameter.

[0087] The DNN blood pressure model and the wristband-free pulse wave blood pressure estimation algorithm constitute the wristband-free blood pressure estimation technical solution of this invention.

[0088] Specifically, the deep neural network (DNN) blood pressure model is a cascaded structure of a one-dimensional convolutional neural network (1D-CNN) and a fully connected regression network (FC). The 1D-CNN is used to extract local morphological features from the pulse wave feature sequence, and the FC network maps the extracted features into a blood pressure estimation vector containing systolic and diastolic blood pressure. ReLU activation and Dropout regularization are used, and the blood pressure estimation vector is individually calibrated based on volume component compensation mapping. The deep neural network (DNN) blood pressure model employs a cascaded structure of a one-dimensional convolutional neural network (1D-CNN) and a fully connected regression network (FC): the 1D-CNN contains multiple convolutional and pooling layers for extracting local morphological features from the pulse wave feature sequence; the FC network maps the convolutional features into a blood pressure estimation vector containing systolic blood pressure (SBP) and diastolic blood pressure (DBP). The model training uses the mean squared error loss function, with wristband-free pulse wave samples and their paired cuff-type blood pressure measurements as supervision labels.

[0089] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0090] Furthermore, the in-vehicle multimodal health monitoring and intelligent interaction system of the present invention also includes a computer-readable storage medium. In one embodiment of the present invention, the computer-readable storage medium can be configured to store a program for executing the in-vehicle multimodal health monitoring and intelligent interaction method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described in-vehicle multimodal health monitoring and intelligent interaction method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0091] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0092] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0093] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A vehicle-mounted multimodal health monitoring method, characterized in that, The method includes: In response to the user's wake-up command, a multi-dimensional dataset is obtained, which includes at least health data and environmental data; Physiological gain adjustment was performed on the multi-dimensional dataset to obtain calibrated physiological feature parameters; Based on the multi-dimensional dataset, the confidence levels of each data source are obtained; The confidence levels of each data source and the calibrated physiological characteristic parameters are fused to obtain a processed multidimensional dataset, which is then used to obtain the user's multimodal health monitoring dataset.

2. The vehicle-mounted multimodal health monitoring method according to claim 1, characterized in that, The acquisition of the multi-dimensional dataset includes: Acquire six parameters, including at least blood oxygen, blood pressure, respiration, pulse rate, electrocardiogram, and fingertip temperature; Obtain body composition data, which includes at least 18 body composition parameters; Acquire visual data, which includes at least facial data and eye data; Health data is obtained based on six-parameter data, body composition data, and visual data; Acquire environmental data, including at least in-vehicle temperature, humidity, traffic flow density, vehicle speed, road surface, illumination, and noise data, to obtain a multi-dimensional dataset.

3. The vehicle-mounted multimodal health monitoring method according to claim 2, characterized in that, The 18 body composition parameters include 18 data points: total cellular water, extracellular water, intracellular water, protein, inorganic salts, body fat, body fat percentage, skeletal muscle mass, muscle mass, basal metabolic rate, BMI, visceral fat level, waist-to-hip ratio, body age, standard weight, lean body mass, phase angle, and health score.

4. The vehicle-mounted multimodal health monitoring method according to claim 3, characterized in that, The physiological gain adjustment of the multi-dimensional dataset to obtain the calibrated physiological feature parameters includes: Physiological gain adjustment is performed on the body composition data in the multidimensional dataset to obtain the gain matrix. The diagonal elements of the gain matrix are the body composition compensation gain coefficients of the corresponding physiological characteristics, which are used to compensate for the influence of body composition differences on the measurement results of the corresponding physiological characteristics. Based on environmental parameters from a multi-dimensional dataset, determine the influence vector; The physiological feature parameters in the multidimensional dataset are calibrated based on the gain matrix, and the confidence of the calibrated physiological feature parameters is evaluated and cross-modal fusion is performed based on the influence vector to obtain the calibrated physiological feature parameters.

5. The vehicle-mounted multimodal health monitoring method according to claim 4, characterized in that, The method further includes: Based on the influence vector, the user's currently available cognitive resources are determined.

6. A method for vehicle-mounted multimodal health monitoring and intelligent interaction, characterized in that, The method includes: Substitute the user's multimodal health monitoring dataset from any one of claims 1 to 5 into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters; By inputting the user's health parameters and / or fatigue parameters into the risk prediction model, we can obtain health and medical advice for the user and the depth of interaction that matches the user's current available cognitive resources.

7. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 6, characterized in that, The step of inputting the user's multimodal health monitoring dataset into the fatigue and emergency identification module to obtain the user's health parameters and / or fatigue parameters includes: A peripheral perfusion evaluation function is constructed based on six-parameter data from the user's multimodal health monitoring dataset. Based on the user's multimodal health monitoring dataset, determine the joint multiscale entropy function; By using the peripheral perfusion evaluation function and the joint multi-scale entropy function, the user's fatigue state and fatigue level are identified to obtain the user's fatigue parameters. And / or, Based on the user's multimodal health monitoring dataset, the user's health status parameters are obtained; Generate a health status assessment index for the user based on the user's health status parameters; Based on the user's health status assessment index, selectively generate and output the user's health parameters; The construction of the peripheral perfusion evaluation function includes: The temperature and humidity parameters in the environmental influence vector inside the vehicle cabin are acquired in real time. The temperature shift caused by the forced convection of the air conditioner is compensated by the thermal balance model to obtain the normalized fingertip temperature. The AC / DC amplitude ratio of the pulse wave signal was extracted, and combined with the pulse wave arrival time, normalized fingertip temperature, and blood oxygen saturation, a nonlinear evaluation function characterizing peripheral vascular resistance was constructed. The larger the value of the nonlinear evaluation function, the worse the peripheral perfusion and the higher the degree of sympathetic nerve stress activation.

8. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 7, characterized in that, The step of substituting the user's health parameters and / or fatigue parameters into the risk prediction model to obtain the user's health and medical advice and the depth of interaction that matches the user's current available cognitive resources includes: The volume component data in the user's multimodal health monitoring dataset is mapped to a dynamic Bayesian network to obtain the initial state distribution parameters of the dynamic Bayesian network. Based on the initial state distribution parameters of the dynamic Bayesian network and the user's health parameters and / or fatigue parameters, multidimensional risk prediction data is obtained; and / or, the user's multimodal health monitoring dataset, body composition prior vector, and historical health state sequence are input into the deep neural network (DNN) risk prediction model to obtain multidimensional risk prediction data; wherein, the body composition prior vector is generated based on the body composition data in the multimodal health monitoring dataset; the historical health state sequence is generated based on the user's multiple health state parameters and / or health state assessment indices before the current moment, arranged in chronological order; Personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-built medical knowledge graph to selectively generate medical suggestions, thus providing users with health and medical advice. Determine the depth of interaction with the user based on the user's currently available cognitive resources.

9. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 8, characterized in that, The deep neural network (DNN) risk prediction model is a cascaded structure of a temporal convolutional network (TCN) and a bidirectional long short-term memory (BiLSTM) network. It takes the user's multimodal health monitoring dataset, volume component prior vector, and historical health status sequence as input. The temporal convolutional network extracts local temporal features of the historical sequence, and the bidirectional long short-term memory network captures the long-term risk evolution trend, outputting the probability distribution of health risk level and the predicted value of the risk index at the next moment.

10. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 9, characterized in that, The personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-constructed medical knowledge graph, selectively generating medical suggestions, i.e., obtaining health and medical advice for the user, including: The multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data are mapped into personalized medical entity vectors through a modal encoder. Based on the semantic similarity between personalized medical entity vectors and knowledge graph nodes, and the in-vehicle scene weights determined by vehicle speed, traffic flow density, and environmental data, personalized subgraphs are retrieved and pruned. Multi-hop reasoning path scores are calculated on personalized subgraphs. The reasoning path with the highest score is selected and fused with clinical guidelines for similar cases before being input into a large language model to generate medical recommendations.

11. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 9, characterized in that, The personalized graph reasoning based on in-vehicle multimodal data matches multidimensional risk prediction data with a pre-constructed medical knowledge graph, selectively generating medical suggestions, i.e., obtaining health and medical advice for the user, including: Map the multimodal health monitoring dataset, calibrated physiological characteristic parameters, and environmental data into personalized medical entity vectors; Based on personalized medical entity vectors, personalized subgraphs are obtained by retrieving and pruning from a pre-constructed medical knowledge graph according to semantic similarity and vehicle scene weight. Perform multi-hop path reasoning on the personalized subgraph to obtain the reasoning path and node confidence that match the user's current vehicle health status; If similar cases are found, then based on the personalized graph reasoning of the in-vehicle multimodal data, the knowledge graph reasoning path matching the user's current multimodal health monitoring dataset and the clinical guidelines corresponding to the similar cases are extracted. The knowledge graph reasoning path includes at least retrieving knowledge graph subgraphs associated with the user's body composition parameters, physiological characteristic parameters and environmental data, and LLM understanding the medical causes of the user's physiological characteristics. By integrating similar cases, knowledge graph reasoning paths, and corresponding clinical guidelines for similar cases, a fused medical dataset is obtained, and health and medical advice for users is generated based on the fused medical dataset. If no similar cases are found, the medical entity data is uploaded to the cloud server and the medical system, and medical advice is obtained from the medical system. Based on the medical advice provided by the medical system, health and medical advice for the user is generated.

12. The vehicle-mounted multimodal health monitoring and intelligent interaction method according to claim 9, characterized in that, The method further includes: If the user's health parameters reach a preset emergency threshold, then the vehicle's location information is obtained; The vehicle's location information, along with the user's multi-dimensional dataset and health parameters, are uploaded to the cloud server and medical system. And send a remote distress signal to the medical system; Furthermore, the multi-dimensional datasets of each user and their health and medical advice are uploaded to the cloud server.

13. A vehicle-mounted multimodal health monitoring and intelligent interaction system, characterized in that, The system comprises a perception layer, a communication layer, a processing layer, an interaction layer, and a cloud layer. The perception layer includes a multimodal sensor module, which is used to collect health data and environmental data, that is, to obtain the user's multidimensional dataset; The communication layer, including the communication module, is used for encrypted data transmission and command interaction; The processing layer includes a fatigue and emergency identification module, a risk prediction model, a wristband-less blood pressure estimation module, and a control module, which are used for data processing, user health risk prediction, and control between different layers of the system. The wristband-less blood pressure estimation module is used to estimate the user's blood pressure without a wristband. The control module includes a processor and a memory. The memory is adapted to store multiple program codes, which are adapted to be loaded and run by the processor to execute the vehicle-mounted multimodal health monitoring method according to any one of claims 1 to 5 and the vehicle-mounted multimodal health monitoring and intelligent interaction method according to any one of claims 6 to 12. The interaction layer includes the in-vehicle interactive terminal module, which is used for interaction with users and information display; The cloud layer, including cloud servers and medical system modules, is used to enable information exchange between the system and the cloud server and medical system.

14. A wristband-less blood pressure estimation method, applied to the wristband-less blood pressure estimation module in the vehicle-mounted multimodal health monitoring and intelligent interaction system of claim 13, characterized in that, The wristband-free blood pressure estimation method includes: Acquire the user's pulse wave signal, wherein the pulse wave signal is obtained by a non-invasive vehicle-mounted sensor without a wristband; The pulse wave signal is preprocessed to extract pulse wave feature parameters, wherein the pulse wave feature parameters include at least the pulse wave arrival time, pulse wave amplitude ratio and pulse wave morphology features. The pulse wave feature parameters are input into a deep neural network (DNN) blood pressure model to obtain the user's blood pressure estimate. The blood pressure estimate is calibrated based on the user's body composition data to obtain the calibrated blood pressure value; The blood pressure model is trained based on pulse wave signal samples collected without a wristband. The deep neural network (DNN) blood pressure model integrates pulse wave feature parameters and body composition compensation vectors to eliminate the influence of individual body differences on blood pressure estimation.

15. The wristband-less blood pressure estimation method according to claim 14, characterized in that, The deep neural network (DNN) blood pressure model is a cascaded structure of a one-dimensional convolutional neural network and a fully connected regression network. The one-dimensional convolutional neural network is used to extract local morphological features from the pulse wave feature sequence. The fully connected regression network maps the extracted features into a blood pressure estimation vector containing systolic and diastolic blood pressure. The ReLU activation function and Dropout regularization are used, and the blood pressure estimation vector is individually calibrated based on body composition compensation mapping.

Images