In-vehicle vital sign monitoring method based on millimeter wave radar and data fusion

Abstract

The invention discloses an in-vehicle vital sign monitoring method based on millimeter wave radar and data fusion, and relates to the technical field of automobile safety monitoring, and the method comprises the steps: collecting multi-modal environment data through an arranged sensor, and carrying out the preprocessing of the multi-modal environment data, and obtaining a processed vital sign signal, a carbon dioxide concentration change rate and an environment temperature value; inputting the processed vital sign signal into a pre-trained convolutional neural network model, and outputting a life entity existence probability; and fusing the life entity existence probability, the carbon dioxide concentration change rate and the environment temperature value by adopting a multi-sensor information fusion algorithm, carrying out risk grade judgment, and respectively executing graded early warning and an active intervention strategy according to a risk grade judgment result. According to the method, the detection accuracy can be improved, the environmental adaptability is enhanced, timely rescue response is realized, and the method has engineering application value in the field of passenger safety monitoring of intelligent networked automobiles.

Description

Technical Field

[0001] This invention relates to the field of automotive safety monitoring technology, and in particular to a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion. Background Technology

[0002] With the rapid development of intelligent connected vehicles and in-vehicle sensing technology, the focus of vehicle safety protection is gradually extending from traditional driving safety to the monitoring of the health and physiological status of occupants. Currently, in-vehicle vital sign monitoring has become an important component of intelligent cockpit safety management systems. Related technologies largely rely on infrared imaging sensors, pressure sensors, camera visual recognition, or single radar detection methods to identify the presence of occupants or detect vital signs such as breathing and heartbeat. In particular, the introduction of millimeter-wave radar technology enables non-contact monitoring even in complex environments such as low light, obstructions, or strong light, exhibiting strong robustness and penetration. However, single-sensor modes still have certain limitations in the actual in-vehicle environment. For example, complex electromagnetic reflections, multipath interference, temperature and humidity fluctuations, and changes in human posture can lead to radar signal distortion or unstable feature extraction, affecting the accuracy of vital sign detection. Furthermore, infrared or visual solutions are significantly limited by ambient light and privacy protection, making reliable operation at night or in obstructed conditions difficult, hindering all-weather, all-scenario occupant status monitoring.

[0003] Existing vital sign detection systems also have shortcomings in data fusion and intelligent recognition. Most studies infer the occupant's status solely through threshold judgments or based on single-parameter changes (such as respiratory fluctuations), failing to fully consider the correlation and dynamic uncertainty between multi-source heterogeneous data. This makes them prone to misjudgment in weak signal or high-noise environments. Especially in environments such as monitoring children left behind in vehicles, monitoring pets, and high-temperature confined spaces, traditional methods cannot accurately distinguish between real vital signs and environmental disturbance signals, lacking effective multi-dimensional feature fusion mechanisms and risk level assessment systems. Furthermore, current early warning mechanisms are mostly limited to single-level alarms, failing to develop tiered response strategies that match the degree of risk, and lacking a closed-loop control logic from "detection—judgment—intervention," making it difficult to achieve proactive safety intervention and intelligent response. Summary of the Invention

[0004] The purpose of this invention is to provide an in-vehicle vital sign monitoring method based on millimeter-wave radar and data fusion, addressing the problems of insufficient detection accuracy, poor environmental robustness, and limited early warning mechanisms in existing in-vehicle vital sign monitoring technologies. To solve the above technical problems, this invention provides the following technical solution:

[0005] In a first aspect, the present invention provides a method for monitoring vital signs in a vehicle based on millimeter-wave radar and data fusion, which includes collecting multimodal environmental data through arranged sensors and performing preprocessing to obtain processed vital sign signals, carbon dioxide concentration change rate and ambient temperature value.

[0006] The processed vital signs signal is input into a pre-trained convolutional neural network model, which outputs the probability of the existence of a living organism.

[0007] A multi-sensor information fusion algorithm is used to combine the probability of the presence of living organisms, the rate of change of carbon dioxide concentration, and the ambient temperature value to determine the risk level. Based on the risk level determination results, graded early warning and active intervention strategies are implemented respectively.

[0008] As a preferred embodiment of the in-vehicle vital signs monitoring method based on millimeter-wave radar and data fusion described in this invention, the sensors include millimeter-wave radar, carbon dioxide concentration sensor, and infrared temperature sensor; the multimodal environmental data includes millimeter-wave signals, in-vehicle carbon dioxide concentration values, and apparent temperature values ​​of each seating area.

[0009] The obtained processed vital sign signals include:

[0010] The distance to the target inside the vehicle and the micro-motion information of the chest cavity are calculated based on the frequency difference between the transmitted signal and the echo signal. The echo signal is bandpass filtered to remove high-frequency noise and low-frequency drift.

[0011] The static reflection background of the echo signal is obtained by mean modeling, and the dynamic signal is extracted by difference with the current signal.

[0012] Perform time-frequency analysis on dynamic signals to extract respiratory rate and heart rate as vital signs signals.

[0013] As a preferred embodiment of the in-vehicle vital sign monitoring method based on millimeter-wave radar and data fusion described in this invention, the convolutional neural network model includes:

[0014] The preprocessed vital signs signals were obtained by training with historical millimeter-wave vital signs data. The signals were then divided into segments according to a fixed time window and normalized.

[0015] Periodic features in the signal are extracted using a multi-layer convolutional structure to identify respiratory rate and heart rate features. The convolution operation is represented as follows:

[0016] ;

[0017] in, Indicates the first Feature maps of each convolutional channel For convolution kernel weights, For bias terms, For activation function, For signal segments;

[0018] A fully connected layer is used to map the features to the probability output space, outputting the probability value of the existence of a living organism, represented as:

[0019] ;

[0020] in, This represents the probability of the existence of a living organism. This is the result of a linear combination of the last layer of the neural network.

[0021] As a preferred embodiment of the in-vehicle vital sign monitoring method based on millimeter-wave radar and data fusion described in this invention, the multi-sensor information fusion algorithm includes:

[0022] Based on the DS evidence theory, which integrates the probability of the existence of living organisms, the rate of change of carbon dioxide concentration, and the ambient temperature value, confidence weights are calculated for millimeter-wave radar, carbon dioxide sensor, and infrared temperature sensor, respectively, and expressed as follows:

[0023] ;

[0024] ;

[0025] in: For sensors The weight, For sensors The confidence output, For statistical consistency indicators, For sensor health status, , and These are the weighting coefficients. For normalized sensors The weights;

[0026] For each sensor, a basic probability assignment function is constructed. Based on the regular merging rule of DS evidence theory, multiple basic probability assignment functions are combined to obtain the conservative confidence level of life existence and calculate the risk index.

[0027] As a preferred embodiment of the in-vehicle vital sign monitoring method based on millimeter-wave radar and data fusion described in this invention, the calculation formula for the risk index is:

[0028] ;

[0029] in: As a risk indicator, To maintain confidence in the existence of life. For temperature factor, It is the carbon dioxide concentration factor. and The amplification factor is used to control the intensity of the impact of temperature and carbon dioxide on the final risk, respectively.

[0030] By normalizing the risk indicators, the risk levels are mapped to the [0,1] interval for risk assessment, and divided into four levels: Level 0, Level 1, Level 2, and Level 3.

[0031] As a preferred embodiment of the in-vehicle vital sign monitoring method based on millimeter-wave radar and data fusion described in this invention, the step of implementing graded early warning and active intervention strategies according to the risk level judgment results includes:

[0032] When the risk level is Level 0, no active intervention is triggered, and the sensor remains in a periodic detection state.

[0033] When the risk level is Level 1, it enters monitoring mode, sends an alert to the car owner's app, and records the current sensor data;

[0034] When the risk level is Level 2, an audio-visual alert is triggered, and a notification is sent to the owner's APP along with emergency contact information. The cloud monitoring center is contacted to record the action execution results and vehicle status.

[0035] When the risk level is Level 3, the air conditioning ventilation system will be automatically turned on, an emergency notification will be sent to the owner's APP and confirmation will be requested. At the same time, the preset emergency contact will be called and the alarm information will be pushed to the cloud monitoring center.

[0036] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion.

[0037] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements the steps of a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion.

[0038] The beneficial effects of this invention are as follows: This method significantly reduces false alarms, enhances environmental adaptability, enables timely rescue response, and takes into account energy consumption and user privacy protection while improving detection accuracy and recall rate. Thus, it has substantial technological progress and engineering application value in the field of occupant safety monitoring of intelligent connected vehicles. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more readily understood, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. An embodiment appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment that selectively excludes other embodiments.

[0044] Reference Figure 1 This is the first embodiment of the present invention, which provides a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion, including:

[0045] S1: Collect in-vehicle environmental data through millimeter-wave radar, carbon dioxide concentration sensor and infrared temperature sensor, and preprocess the vital signs signal returned by the millimeter-wave radar to obtain the processed vital signs signal, carbon dioxide concentration change rate and ambient temperature value.

[0046] S2: Input the processed vital signs signal into the trained convolutional neural network model and output the probability of the existence of a living organism. The convolutional neural network model is trained with historical millimeter-wave radar data and can identify weak vital signs signals and output probabilistic results.

[0047] S3: Employing a multi-sensor information fusion algorithm, the algorithm integrates the probability of the existence of the organism, the rate of change of carbon dioxide concentration, and the ambient temperature value to determine the risk level and output the risk level determination result. The risk level determination result includes Level 0, Level 1, Level 2, and Level 3. Based on the risk level determination result, a graded early warning and proactive intervention strategy is executed. The multi-sensor information fusion algorithm is based on the DS evidence theory, dynamically weights the data from each sensor, and outputs a graded risk level through evidence combination and uncertainty processing.

[0048] Specifically, millimeter-wave radar is installed in the center of the vehicle's interior roof or near the rearview mirror. By measuring the frequency difference between the transmitted signal and the echo signal, the distance to the target and information on the micro-movements of the chest cavity can be obtained.

[0049] The acquired millimeter-wave signal was preprocessed, and the received echo signal was bandpass filtered to remove high-frequency noise and low-frequency drift components, retaining the frequency band corresponding to the micro-movements of the human chest cavity. By averaging the echo data from multiple time periods, the static reflection background was extracted and subtracted from the current signal to obtain a signal containing only dynamic components. Time-frequency analysis was performed on the dynamic signal to extract key characteristic parameters reflecting the periodic movement of the occupant's chest cavity, including respiratory rate and heart rate, as vital signs signals.

[0050] Carbon dioxide concentration sensors and infrared temperature sensors installed inside the vehicle simultaneously collect the carbon dioxide concentration value inside the vehicle and the apparent temperature value of each seating area.

[0051] The rate of change of carbon dioxide concentration per unit time is calculated based on the carbon dioxide concentration obtained from continuous sampling and is used as the carbon dioxide concentration change rate.

[0052] The apparent temperature values ​​of each seat area output by the infrared temperature sensor are averaged over time to obtain the ambient temperature value inside the vehicle.

[0053] The preprocessed vital signs signals from millimeter-wave radar, extracted vital characteristic parameters, carbon dioxide concentration change rate, and ambient temperature are uniformly formatted and time-synchronized to output a structured dataset.

[0054] The trained convolutional neural network (CNN) model is used to intelligently analyze the preprocessed vital signs signals, thereby determining the probability of the existence of living organisms and comprehensively assessing their vital signs status.

[0055] The vital signs signals are standardized and divided into time windows. The signals are divided into continuous segments according to a fixed time window length. Then, each segment is normalized to keep the amplitude range of the input signal consistent.

[0056] A trained convolutional neural network model takes signal segments as input and extracts vital sign patterns from time series through its multi-layer convolutional structure. The model's convolutional layers extract local features from the signal through convolutional kernels, identifying waveform patterns with periodic changes, such as respiratory rhythm and heartbeat oscillations.

[0057] The convolution operation is represented as:

[0058] ;

[0059] in, Indicates the first Feature maps of each convolutional channel For convolution kernel weights, For bias terms, For activation function, This is a signal segment.

[0060] Multi-scale, time-related feature representations are extracted stepwise through multi-layer convolution and pooling operations. Fully connected layers map these features to a probability output space, representing the confidence level of the presence of a living organism in the current time period. The final output of the convolutional neural network is the probability of the presence of a living organism, indicating the likelihood of detecting vital signs in the current observation segment. This is normalized using the Sigmoid function and expressed as:

[0061] ;

[0062] in, This represents the probability of the existence of a living organism. This is the result of a linear combination of the last layer of the neural network.

[0063] When the probability of the presence of a living being is close to 1, it means that obvious signs of life have been detected; when the probability of the presence of a living being is close to 0, it means that no obvious signs of life have been detected inside the vehicle.

[0064] To dynamically reflect the recent status and reliability of the sensors, a weighted index is calculated for each sensor, and a composite confidence index for each sensor is calculated, expressed as:

[0065] ;

[0066] ;

[0067] in: For sensors The weight, For sensors The confidence output, For statistical consistency indicators, The sensor health status (e.g., whether the sensor is online, whether there is a fault indicator; if the sensor is online and there is no fault indicator, the sensor health status is 1, and otherwise it is 0). , and These are the weighting coefficients. For normalized sensors The weights;

[0068] The basic probability assignments for each sensor are constructed separately. Using the canonical merging rule of the DS evidence theory, the basic probability assignments of the three sensors are merged sequentially. The conservative confidence level of life existence is derived using the merged life existence probability, expressed as:

[0069] ;

[0070] in: To maintain confidence in the existence of life. The probability of existence of the merged life form;

[0071] By combining the obtained life confidence scores, ambient temperature values, and carbon dioxide concentration change rates, risk indicators are calculated. The risk indicators are defined as follows:

[0072] ;

[0073] in: As a risk indicator, For temperature factor, It is the carbon dioxide concentration factor. and The amplification factor controls the intensity of the impact of temperature and carbon dioxide on the final risk, respectively.

[0074] By normalizing the risk indicators, the risk levels are mapped to the [0,1] interval for risk assessment, and divided into four levels: Level 0, Level 1, Level 2, and Level 3.

[0075] If the vehicle is in motion, physical interventions that may affect driving safety are prohibited. Instead, a remote notification is sent, and physical measures are implemented only when it is safe to do so. The risk level-driven system triggers the following action combinations based on priority and safety constraints: notification, audio-visual alerts, environmental interventions, low-power monitoring, and cloud reporting.

[0076] If the risk level is Level 0, no active intervention will be triggered, and the sensor will be periodically woken up to detect changes in status.

[0077] If the risk level is Level 1, it will immediately enter short-term enhanced monitoring mode, send a reminder to the car owner's APP, archive the current sensor data locally, and attach a cloud-backed alarm summary if the vehicle is in a movable state.

[0078] If the risk level is Level 2, an audio-visual alert will be triggered, and a notification will be sent to the owner's app with emergency contact information. The cloud monitoring center will be contacted to enter enhanced monitoring mode and record the results of the actions and the vehicle status.

[0079] If the risk level is Level 3, physical interventions to improve the environment will be implemented, the air conditioning / ventilation system will be automatically turned on, the start-up status and power consumption will be recorded, the external safety status will be monitored, and a remote notification will be triggered to send an emergency notification to the owner's APP and request confirmation. At the same time, the preset emergency contact will be called and the alarm information will be pushed to the cloud monitoring center.

[0080] For each physical action, the system will re-detect changes in vital signs and adjust subsequent actions after the action is completed. Before triggering a risk level, the system requires that the risk level be maintained for several consecutive predetermined time windows before the action can be executed. After sending an APP notification, if the car owner confirms that they are aware of / cancel the alarm through the APP within the specified time, the system will select to downgrade the monitoring based on the confirmation type.

[0081] This embodiment also provides a computer device applicable to a method for monitoring in-vehicle vital signs based on millimeter-wave radar and data fusion, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement all or part of the steps of the method described in the above embodiments of the present invention.

[0082] This embodiment also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, it performs the method in any optional implementation of the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0083] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0084] In summary, this method significantly reduces false alarms, enhances environmental adaptability, enables timely rescue response, and balances energy consumption and user privacy protection while improving detection accuracy and recall rate. Therefore, it has substantial technological advancement and engineering application value in the field of occupant safety monitoring of intelligent connected vehicles.

[0085] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for vital sign monitoring in a vehicle based on millimeter wave radar and data fusion, characterized in that: include, Multimodal environmental data is collected by deployed sensors and preprocessed to obtain processed vital signs signals, carbon dioxide concentration change rate, and ambient temperature value. The processed vital signs signal is input into a pre-trained convolutional neural network model, which outputs the probability of the existence of a living organism. A multi-sensor information fusion algorithm is used to combine the probability of the presence of living organisms, the rate of change of carbon dioxide concentration, and the ambient temperature value to determine the risk level. Based on the risk level determination results, graded early warning and active intervention strategies are implemented respectively.

2. The method for vital sign monitoring of in-vehicle occupants based on millimeter wave radar and data fusion as claimed in claim 1, wherein: The sensors include millimeter-wave radar, carbon dioxide concentration sensor, and infrared temperature sensor; the multimodal environmental data includes millimeter-wave signals, in-vehicle carbon dioxide concentration values, and apparent temperature values ​​for each seating area. The obtained processed vital sign signals include: The distance to the target inside the vehicle and the micro-motion information of the chest cavity are calculated based on the frequency difference between the transmitted signal and the echo signal. The echo signal is bandpass filtered to remove high-frequency noise and low-frequency drift. The static reflection background of the echo signal is obtained by mean modeling, and the dynamic signal is extracted by difference with the current signal. Perform time-frequency analysis on dynamic signals to extract respiratory rate and heart rate as vital signs signals.

3. The method for vital sign monitoring inside a vehicle based on millimeter wave radar and data fusion as claimed in claim 1, wherein: The convolutional neural network model includes: The preprocessed vital signs signals were obtained by training with historical millimeter-wave vital signs data. The signals were then divided into segments according to a fixed time window and normalized. Periodic features in the signal are extracted using a multi-layer convolutional structure to identify respiratory rate and heart rate features. The convolution operation is represented as follows: ； wherein, represents the feature map of the th convolutional channel, is a convolution kernel weight, is a bias term, is an activation function, is a signal segment; A fully connected layer is used to map the features to the probability output space, outputting the probability value of the existence of a living organism, represented as: ； wherein, is the probability of existence of the living body, is the linear combination result of the last layer of the neural network.

4. The method for vital sign monitoring inside a vehicle based on millimeter wave radar and data fusion as claimed in claim 1, wherein: The multi-sensor information fusion algorithm includes: Based on the DS evidence theory, which integrates the probability of the existence of living organisms, the rate of change of carbon dioxide concentration, and the ambient temperature value, confidence weights are calculated for millimeter-wave radar, carbon dioxide sensor, and infrared temperature sensor, respectively, and expressed as follows: ； ； wherein: is a weight of the sensor , is a confidence output of the sensor , is a statistical consistency indicator, is a sensor health status, , and are weight coefficients, is a weight of the normalized sensor . For each sensor, a basic probability assignment function is constructed. Based on the regular merging rule of DS evidence theory, multiple basic probability assignment functions are combined to obtain the conservative confidence level of life existence and calculate the risk index.

5. The method for vital sign monitoring inside a vehicle based on millimeter wave radar and data fusion as claimed in claim 4, wherein: The formula for calculating the risk indicator is as follows: ； wherein: is a risk indicator, is a conservative life presence confidence, is a temperature factor, is a carbon dioxide concentration factor, and are amplification factors, respectively controlling the strength of the influence of temperature and carbon dioxide on the final risk. By normalizing the risk indicators, the risk levels are mapped to the [0,1] interval for risk assessment, and divided into four levels: Level 0, Level 1, Level 2, and Level 3.

6. The method for vital sign monitoring inside a vehicle based on millimeter wave radar and data fusion as claimed in claim 1, wherein: The implementation of tiered early warning and proactive intervention strategies based on risk level assessment results includes: When the risk level is Level 0, no active intervention is triggered, and the sensor remains in a periodic detection state. When the risk level is Level 1, it enters monitoring mode, sends an alert to the car owner's app, and records the current sensor data; When the risk level is Level 2, an audio-visual alert is triggered, and a notification is sent to the owner's APP along with emergency contact information. The cloud monitoring center is contacted to record the action execution results and vehicle status. When the risk level is Level 3, the air conditioning ventilation system will be automatically turned on, an emergency notification will be sent to the owner's APP and confirmation will be requested. At the same time, the preset emergency contact will be called and the alarm information will be pushed to the cloud monitoring center. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The computer program is executed by the processor to implement the steps of the method for monitoring vital signs of a living body in a vehicle based on millimeter wave radar and data fusion according to any one of claims 1-7.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method for monitoring vital signs of a living body in a vehicle based on millimeter wave radar and data fusion according to any one of claims 1-7.

Images