Biometric Data Capture and Analysis Using a Hybrid Sensing System

Abstract

Devices and methods detect a user's health condition, which can be evaluated based on thermal sensor signals and / or radar sensor signals. One or more synthetic biometric vectors can be generated from biometric vectors based on the thermal and radar signals, wherein the synthetic biometric vectors contain synthetic information about one or more biometric characteristics of the user. Hazard information about the user is obtained from the one or more synthetic biometric vectors, wherein the hazard information indicates a health event of the user. Accordingly, appropriate actions can be performed on behalf of the user to improve the health condition. The one or more synthetic biometric vectors can include additional biometric characteristics and / or a time series of the synthetic biometric vectors to enhance hazard prediction. Additionally, the devices and methods can support the user in different environments including homes, enterprises, or vehicles.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims priority to U.S. Non - Provisional Application No. 16 / 797,071, filed on February 21, 2020, entitled "Biometric Data Capturing and Analysis Using a Hybrid Sensing Systems", which is a continuation - in - part of U.S. Non - Provisional Application No. 16 / 559,814, filed on September 4, 2019, entitled "Biometric Data Capturing and Analysis", the content of which is hereby incorporated by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure relate to extracting biometric data from thermal and radio frequency (RF) sensors and to performing appropriate actions. Background Art

[0004] Image sensors are commonly used in home applications. Examples include those for baby monitors, Internet Protocol (IP) cameras, security cameras, etc. Other image sensors include thermal cameras and thermal sensor arrays. Expanding the effective applications of image sensors will increase their popularity.

[0005] In May 2019, the Chicago Sun Times reported an article about a man in Illinois who had a heart - related event while driving and crashed and died, which highlighted the need to expand the applications of sensors (e.g., thermal sensors). The man experienced a "heart - related event" while driving, lost consciousness, and crashed his vehicle into a utility pole. After hitting the utility pole, his car hit another vehicle. Preventive measures to address such terrifying events would surely benefit the general public. Summary of the Invention

[0006] A device detects a user's health condition, which can be evaluated based on thermal sensor signals and / or radar sensor signals. One or more synthetic biometric vectors can be generated from biometric vectors based on thermal and radar signals, where the synthetic biometric vectors contain synthetic information about one or more biometric characteristics associated with the user.

[0007] On the other hand, hazard information about a user is obtained from one or more synthetic biometric vectors, where the hazard information indicates a health event associated with the user. Accordingly, appropriate actions can be performed on behalf of the user to improve the health condition. The one or more synthetic biometric vectors can include additional biometric features and / or a time series of synthetic biometric vectors to enhance hazard prediction.

[0008] On the other hand, when the thermal signal is unavailable, only the first synthetic information about the first biometric feature is extracted from the second information containing the second biometric vector from the radar signal.

[0009] On the other hand, a thermal feature identifying the user is extracted from the thermal signal. When a health event is detected, the device sends a message indicating the hazard level of the user to a medical institution.

[0010] On the other hand, the device downloads the health record of the user. The biometric features can be weighted differently based on the health record.

[0011] On the other hand, the first and second trained machine learning models can transform the thermal signal and the radar signal respectively. Then the transformed signals are used to obtain biometric vectors. In addition, the thermal features and / or motion vectors provided by the first machine learning model to the second machine learning model can assist in transforming the radar signal.

[0012] On the other hand, the machine learning model can be downloaded by the device from a cloud server. The model can be updated at the cloud server such that the device can receive updated model information to update the downloaded model.

[0013] On the other hand, the device provides an assessment of a vehicle driver. The device obtains a thermal signal and a radar signal about the vehicle driver from a thermal sensor and a radar sensor respectively. The device generates biometric vectors from the thermal signal and the radar signal, extracts synthetic information about one or more biometric features, and generates synthetic biometric vectors from the synthetic information. Then, when a health event occurs, the device determines hazard information from the synthetic biometric vectors. Based on the hazard information, the device identifies appropriate actions and performs appropriate actions on behalf of the vehicle driver.

[0014] On the other hand, the device downloads the event health record and behavior record of the vehicle driver and identifies appropriate actions based on the event health and behavior records.

[0015] On the other hand, the device applies a first weight to the first synthetic information about the first biometric feature and a second weight to the second synthetic information about the second biometric feature based on the event health and behavior records of the vehicle driver.

[0016] On the other hand, when a health event regarding the vehicle driver is detected, the device sends the logbook health and behavior records to the emergency services to prepare for the arrival of the vehicle driver.

[0017] On the other hand, the device uses a thermal sensor for biometric data extraction and tracking in smart home applications. Applications such as health condition analysis, motion estimation (e.g., fall estimation or motion trajectory), casual prediction (e.g., the heartbeat is slowing down to a dangerous level), hazard detection (e.g., abnormal body position tolerance such as lying down on the floor for a long time, lying on the side on the couch, or head down), learning personal profiles, and system adaptation according to personal preferences.

[0018] On the other hand, the parameters of the thermal sensor can be enhanced to allow extraction of as much data as possible. Examples include but are not limited to: increasing the number of sensing elements (i.e., resolution), frame rate, sensitivity, and / or signal-to-noise ratio level.

[0019] On the other hand, signal processing techniques extract biometric data from the thermal image.

[0020] On the other hand, an analysis model is used for hazard prediction and associated actions taken subsequently.

[0021] On the other hand, hazard analysis is performed through a deep learning model. Actions are taken based on the hazard coefficient and the associated confidence level estimated from the model.

[0022] On the other hand, the model will recommend taking actions based on the input data sequence with the associated confidence level.

[0023] On the other hand, the model can be trained to predict the hazard coefficient and corresponding actions if necessary with the corresponding confidence level based on previously occurred events.

[0024] On the other hand, for applications with less time urgency, the model can reside in a cloud server instead of a local processing unit.

[0025] On the other hand, the parameters of the smart device are configured differently based on the thermal characteristics of the detected person.

[0026] On the other hand, based on the detected condition detected by the first application, the executed application is changed from the first application to the second application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above summary of the present invention and the following detailed description of the exemplary embodiments of the present invention will be better understood when read in conjunction with the accompanying drawings, which are included by way of example and not as a limitation of the claimed invention.

[0028] Figure 1Shows a thermal sensor located in a room according to an embodiment.

[0029] Figure 2 Shows a device interfacing with one or more thermal sensors and one or more associated intelligent devices according to an embodiment.

[0030] Figure 3a shows a device for processing information from one or more thermal sensors according to an embodiment.

[0031] Figure 3b shows a device for processing information from one or more thermal sensors using a deep learning model to estimate a hazard coefficient according to another embodiment.

[0032] Figure 3c shows a device for processing information from one or more thermal sensors using a deep learning model to suggest actions according to another embodiment.

[0033] Figure 4 Shows a process for identifying a user from thermal sensor information and applying a corresponding profile according to an embodiment.

[0034] Figure 5 Shows a flowchart for executing multiple applications according to an embodiment.

[0035] Figure 6 Shows a flowchart for configuring an intelligent device based on one of multiple parameter sets of detected thermal features according to an embodiment.

[0036] Figure 7 Shows a vehicle system for continuously monitoring the physical health of a vehicle operator according to an embodiment.

[0037] Figure 8 Shows a process for performing one or more actions based on the detected physical condition of a vehicle driver according to an embodiment.

[0038] Figure 9 Shows a device interfacing with a radar sensor and a thermal sensor according to an embodiment.

[0039] Figure 10 Shows a computing system for processing thermal signals and radar signals according to an embodiment.

[0040] Figure 11 Shows a computing system for processing thermal signals and radar signals according to an embodiment.

[0041] Figure 12 Shows a flowchart for performing feature analysis according to an embodiment.

[0042] Figure 13 Shows a preprocessing of a thermal signal according to an embodiment.

[0043] Figure 14 Shows a sensing system for a vehicle according to an embodiment.

[0044] Figure 15 Shows a computing system for processing biometric signals in a vehicle according to an embodiment.

[0045] Figure 16 Shows a flowchart of a decision logic block according to an embodiment. Detailed Description

[0046] According to one aspect of an embodiment, the device can detect the health condition of a user that can be evaluated from a thermal sensor signal and / or a radar sensor signal. One or more synthetic biometric vectors can be generated from the biometric vectors based on the thermal and radar signals, where the synthetic biometric vectors contain synthetic information about one or more biometric characteristics of the user. Hazard information about the user is obtained from the one or more synthetic biometric vectors, where the hazard information indicates a health event of the user. Accordingly, appropriate actions can be taken on behalf of the user to improve the health condition. The one or more synthetic biometric vectors can include additional biometric characteristics and / or a time series of the synthetic biometric vectors to enhance hazard prediction.

[0047] According to another aspect of an embodiment, the performance metrics (e.g., resolution, frame rate, and sensitivity) of a thermal sensor or a thermal sensor array can be increased to support applications such as authentication, biometric data extraction, and health condition analysis. Prediction can be performed by monitoring a time series of thermal images, and thus an early warning of the health condition can be generated.

[0048] According to another aspect of an embodiment, the frame rate of the thermal sensor can be increased to a determined level to capture the change over time of minute details of the thermal radiation from the human body, such as the detailed change of the thermal radiation from the human body.

[0049] According to another aspect of an embodiment, the thermal image of the blood flow through the skin can be converted into a time signal for pulse rate extraction. Further signal processing techniques can be applied to additional biometric data of an individual to analyze the health condition. The image signal can be processed to identify multiple objects from the content and track the associated biometric data.

[0050] According to another aspect of an embodiment, the application can determine the position of the human body within the image signal, as well as motion tracking from a previous image, for fall detection. Motion estimation can be applied to predict whether there is any hazard to the individual within the image signal.

[0051] According to another aspect of the embodiment, a profile can be associated with an individual. The device can track and learn the behavior of the individual from the history of the image signal. Further, when an individual is detected in a scene, the device can adapt. For example, when a person is detected entering the living room in summer, the set temperature of the air conditioner in the living room can be adapted to the person's preference.

[0052] According to another aspect of the embodiment, the ambient temperature can be controlled based on the body surface temperature of the individual and other parameters such as relative humidity and outdoor temperature, etc., to reach an overall comfort zone through machine learning.

[0053] According to another aspect of the embodiment, the accuracy of the analysis is determined by the resolution, sampling frequency and sensitivity of the thermal sensor, the signal processing technology for extracting biometric data from the image signal, and the analysis / learning algorithm.

[0054] According to another aspect of the embodiment, the application of the thermal sensor can be extended to home applications.

[0055] According to another aspect of the embodiment, the analysis model consists of a trained model. The model is trained from a database of reference thermal image signals and associated target vectors, which can represent a series of settings of a smart home device. Reinforcement learning can be deployed to allow the model to adapt to new target vectors. For example, a user can change the temperature setting of a room between summer and winter.

[0056] According to another aspect of the embodiment, instead of applying training to the analysis model, learning is performed over time from a sequence of target vectors associated with thermal features. For example, when a new thermal feature associated with a new user is detected, the default settings of the smart device are applied. When a user changes the settings of an individual device, the new settings are recorded to retrain the model.

[0057] Figure 1 A thermal sensing camera 101 located in room 100 according to an embodiment is shown. The camera 101 can generate a thermal image (thermogram) 102 of an individual not explicitly shown.

[0058] In some embodiments, the thermal sensing camera 101 includes a lens that focuses infrared or far-infrared radiation on visible objects. The focused light is scanned by a thermal sensor that includes a plurality of infrared detector elements (e.g., 24×32 pixels). The detector elements can generate a very detailed temperature pattern (e.g., thermogram 102).

[0059] In some embodiments, the camera 101 may take one-hundredth of a second of the detector array to obtain sensor information to obtain a thermal image. The sensor information can be obtained periodically from thousands of points in the field of view of the thermal sensor to form a sequence of thermal images.

[0060] The thermal spectrogram 102 generated by the detector element of the thermal sensor can be converted into electrical pulses. The pulses are then sent to a signal processing unit (e.g., device 300 as shown in Figure 3), which can be implemented as a circuit board with a dedicated chip that converts the sensor information into biometric data.

[0061] The thermal camera 101 may also include tracking capabilities such that the direction of the camera 101 can be changed to track a moving object such as person 102 moving in room 100.

[0062] Although Figure 1 one thermal sensor is depicted, some embodiments may interface with multiple thermal sensors. For example, an array of thermal sensors may be located at different rooms and / or entry points of a dwelling.

[0063] Figure 2 Device 200 is shown interfacing with thermal sensors 204 and / or 205 via sensor interface 206 and with smart devices 202 and / or 203 via smart device interface 209, according to an embodiment.

[0064] Thermal sensors 204 and 205 are generally used for access control and presence detection. In some embodiments, in order for processor 201 to extract biometric data from the sensor information, the performance of thermal sensor 204 can be extended by increasing the sample frequency (e.g., frame rate) of the captured image signal, identifying and tracking individuals from the image signal, and analyzing the detailed changes of the thermal image over time. Processor 201 can convert the sensor information (signals) into biometric data such as heart rate, body position, health condition, etc. Device 200 can also support prediction of future health events and / or support system personalization by processing the image signal.

[0065] In some embodiments, processor 201 can process the sensor information to detect the thermal signature of a user. When the thermal signature of a specific individual is detected, processor 201 can apply the profile (e.g., temperature settings) of that individual to a smart device 202 (e.g., air conditioner).

[0066] Processor 201 can support one or more health applications that process and / or analyze biometric data, and can generate notifications regarding the biometric data to an external entity (e.g., a doctor) via interface 210 over communication channel 251. For example, a health application may detect from the biometric data that the user is at risk of a heart attack; thus, an emergency notification about the event is sent to the user's doctor.

[0067] Reference Figure 2, A computing system environment may include a computing device in which the processes discussed herein (e.g., process 300 shown in FIG. 3) may be implemented. The computing device may include a processor 201 for controlling the overall operation of the computing device and its associated components (including RAM, ROM, communication module, and first memory device 207). The computing device generally includes various computer-readable media. Computer-readable media can be any available media accessible by the computing device and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may comprise a combination of computer storage media and communication media.

[0068] Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device.

[0069] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. A modulated data signal is a signal whose one or more characteristics are set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0070] In some embodiments, the processor 201 may execute computer-executable instructions stored at the memory 207 and access archival data stored at the memory 208.

[0071] In some embodiments, the memory devices 207 and 208 may be physically implemented within a single memory device.

[0072] FIG. 3a shows a device 300a for processing information from one or more thermal sensors 301 according to an embodiment.

[0073] By using a higher-quality thermal sensor 301 (e.g., having a frame rate of at least 100 frames per second, a resolution of at least 24×32 pixels, good sensitivity, and low noise), biometric data 351 can be extracted via an analog front end 302, an analog-to-digital converter (ADC) 303, and a feature extractor 305 via appropriate signal processing techniques. The biometric data 351 may include pulse rate, body surface temperature, temperature distribution pattern, body contour, and posture, etc. By tracking the changes in the biometric data, the health status of an individual can be analyzed by an analyzer 306, and a warning signal 352 and 353 can be generated by the analyzer 306 and an action generator 307 respectively by further processing the biometric data 351.

[0074] The application can utilize the installed home thermal camera for fall detection by tracking posture changes. For example, when the posture changes from upright to horizontal within a short period of time, a possible fall can be detected, and thus an associated alarm can be generated. In addition, changes in posture, body surface temperature, temperature distribution pattern, and heart rate can be tracked to estimate the hazard level, and an associated action 353 can be taken.

[0075] Hazard prediction from the biometric data 351 can also be supported. For example, when a person's body surface temperature continues to drop and his / her posture sways, the possibility of falling may be higher (as indicated by the hazard level 352), and thus an alarm can be generated (triggered) before the fall occurs.

[0076] Block 302 can perform signal amplification and non-recursive band-pass filtering, where the analog signal corresponding to the thermal image is processed for DC offset removal, noise reduction, and frequency limitation before being processed by the ADC 303. (In the case of some embodiments, block 303 may include a 16-bit ADC, where the sampling frequency (e.g., 200 Hz) is set high enough to capture the details of the object's temperature change.)

[0077] In the feature extraction block 305, image processing is applied to identify valid objects, track the thermal distribution of individual objects over time, and extract parameters from the thermal distribution to form a feature vector. Examples of the parameters of the feature vector include cycle time, change in cycle time, certain time constants within each cycle, and their changes over time, etc. The analysis model 306 obtains the feature vector and compares it with the trained model. Using deep learning algorithms, such as deep neural networks, the model is pre-trained with a large number of general feature vectors. Reinforcement learning can be employed to allow the model to learn from mistakes. A hazard level can be provided for the identified object. In block 307, the action list can be predefined and can be triggered based on the associated hazard level.

[0078] Figure 3b shows a device 300b that processes information from one or more thermal sensors 301 according to another embodiment. Using a model trained with a deep learning model 308, such as a convolutional neural network with supervised learning, a hazard coefficient 322 with an associated confidence level 321 is estimated. An action 324 is determined from an action list 323 and can be based on the hazard coefficient 322 and confidence level 321 provided by the model 308. The model 308 can initially support hazard levels but then identify different hazards with more empirical data such as abnormal heart rate, body surface temperature drop, fall detection, etc.

[0079] The model 308 in the device 300b can also be trained to predict hazards based on a training sequence starting from a fairly early time rather than estimating hazards.

[0080] Figure 3c shows a device 300c that processes information from one or more thermal sensors 301 according to a third embodiment, where an action 332 and an associated confidence level 331 are estimated by a trained model 309. Similarly, the model 309 in the device 300c can also be trained to predict any required action.

[0081] The image processing techniques that can be used depend on system complexity, including the number of thermal sensors, the resolution of each thermal sensor, the list of hazards and actions, system computing power, available memory, etc.

[0082] For the embodiments shown in Figures 3a, 3b, and 3c, depending on the criticality of the response time, the analysis model can be implemented locally or in a cloud server.

[0083] Figure 4 A process 400 for identifying a user from thermal sensor information and applying a corresponding profile according to an embodiment is shown.

[0084] The process (application) 400 supports human presence detection and thermal signature verification at block 401. If a human object is detected at block 401 and the thermal signature matches a known entity, all supported smart devices (e.g., air conditioner, smart TV, or smart lighting) can be adjusted 403 according to a profile database stored at block 402.

[0085] If any adjustment is made to the applied profile 405, the adjustment data 453 can be sent to a profile adaptation unit 406, which can include new settings in the profile. If an adjustment is needed, the profile database is updated 451 by the profile adaptation unit 406.

[0086] To add a new user, the profile adaptation unit 406 sends the thermal signature of the new user 453 to the user identifier unit 401 and sends the associated profile, which can be a default profile, to the profile database unit 402.

[0087] The file adaptation unit 406 may include a deep learning model trained using reinforcement learning.

[0088] Figure 5 A flowchart 500 for sorting through multiple applications executed by the device 200 according to an embodiment is shown. The device 200 may execute one of the multiple applications based on the detected situation. For example, a first health application may monitor general health measurements of the user (e.g., activity level and temperature). If one or more measurements are abnormal, the device 200 may initiate a different health application based on the detected situation.

[0089] Reference Figure 5 ,the device 200 configures the thermal sensors 204 and 205 according to a first set of sensor parameters at block 501 to execute a first application at block 502.

[0090] If an abnormal situation is detected at block 503, the device 200 initiates an appropriate application. For example, at blocks 504 to 505 and 506 to 507, the device may switch to a second application to monitor fall prediction or a third application to monitor the user's heart rate, respectively. When executing the second or third application, the device 200 may configure the thermal sensors 204 and 205 differently to obtain different biometric data.

[0091] In another embodiment, different configuration parameters may be applied to each sensor for each application.

[0092] In a third embodiment, different sets of configuration parameters are applied to the sensors one by one to extract all biometric data before running the application.

[0093] In a fourth embodiment, a set of the most comprehensive configuration parameters is used for all sensors and applications. All sensors may be set to an optimal configuration group, such as but not limited to the highest image resolution, number of bits, frame rate, sensitivity, signal-to-noise ratio (SNR), computing power, power consumption, etc.

[0094] Figure 6FIG. 600 is a flowchart according to an embodiment, where device 200 configures an intelligent device based on a detected user using one of multiple parameter sets. Device 200 may monitor sensor data from thermal sensors 204 and / or 205 to detect thermal characteristics of one or more users. For example, thermal sensor 204 may be located at an entry point of a residence. Based on the sensor information obtained from sensor 204, device 200 may identify users entering and leaving the residence. In some embodiments, the device may detect thermal characteristics from the front (corresponding to a person entering the residence) or from the back (corresponding to a person leaving the residence). Based on the detected thermal characteristics, the intelligent device may be configured with different parameter sets (e.g., temperature settings of an air conditioner).

[0095] At block 601, device 200 trains to detect thermal characteristics of different users from sensor data. For example, the distinguishing characteristics may be stored in memory 208. When the thermal characteristics of two users are detected at block 602, only user A is detected at block 604, or only user B is detected at block 606, the intelligent device may be configured according to the first set of intelligent device parameters at block 603, the second set of intelligent device parameters at block 605, or the third set of intelligent device parameters at block 607, respectively. In some embodiments, the first set (when two users are detected) may be a compromise between the second set and the third set (when only one user is detected). Otherwise (when no user is detected), at block 608, the intelligent device may be configured according to the default settings of the intelligent device parameters.

[0096] Embodiments may support the following capabilities.

[0097] The device uses thermal sensors for biometric data extraction and tracking in smart home applications. Applications such as health status analysis, motion estimation (e.g., fall estimation), accident prediction (e.g., heart rate slowing down to a hazardous level), hazard detection (e.g., lying down for a long time), learning personal profiles, and system adaptation according to personal preferences, etc.

[0098] The parameters of the thermal sensors may be enhanced to allow extraction of as much data as possible. Examples include but are not limited to:

[0099] a. Increase the resolution, frame rate, sensitivity, and signal-to-noise ratio level for heart rate monitoring, for example.

[0100] b. Increase the resolution, sensitivity, and signal-to-noise ratio level for distance detection, etc.

[0101] c. Increase the resolution of the number of objects being tracked.

[0102] Signal processing techniques extract biometric data from thermal images.

[0103] An analytical model for hazard estimation and associated actions taken subsequently.

[0104] An analytical model for action estimation.

[0105] An analytical model for hazard and / or action prediction.

[0106] A model for learning the individual behavior of an intelligent device based on biometric data extracted from a thermal sensor.

[0107] Parameters for configuring an intelligent device based on detected different personnel.

[0108] Changing from a first health application to a second health application based on a condition detected by the first health application. The configuration parameter sets for the respective sensors of the proactive health application can be the same or different.

[0109] Before running a health application, extracting all biometric data using different configuration parameter sets.

[0110] Using a set of comprehensive configuration parameters for all sensors and health applications.

[0111] Obtaining thermal sensor data to detect the thermal characteristics of a person's front or back.

[0112] Capable of increasing the sampling frequency of thermal sensors (including IP cameras, thermal cameras, and thermal sensors) to capture minute changes in color content caused by thermal radiation from the human body.

[0113] Capable of improving the resolution and sensitivity of thermal sensors to span the detection range.

[0114] Exemplary clauses:

[0115] 1. A device supporting at least one intelligent device, the device comprising:

[0116] An intelligent device interface;

[0117] A thermal sensor interface configured to obtain sensor information from a first thermal sensor;

[0118] A processor for executing computer-executable instructions;

[0119] A memory storing the computer-executable instructions, which when executed by the processor cause the device to perform:

[0120] Detecting detected thermal characteristics of a detected user from the sensor information;

[0121] When the detected user is the first user, obtain a first profile corresponding to the first user, wherein the first profile includes a first set of smart device parameters; and

[0122] When the detected user is the first user, configure a first smart device based on the first set of smart device parameters through the smart device interface.

[0123] 2. The apparatus according to clause 1, wherein the memory stores computer-executable instructions that, when executed by the processor, further cause the apparatus to perform:

[0124] When the detected user is the second user, obtain a second profile corresponding to the second user, wherein the second profile includes a second set of smart device parameters, and wherein the second set is different from the first set; and

[0125] When the detected user is the second user, configure the first smart device based on the second set of smart device parameters through the smart device interface.

[0126] 3. An apparatus for supporting at least one smart application, the apparatus comprising:

[0127] A thermal sensor interface configured to obtain sensor information from a first thermal sensor;

[0128] A processor for executing computer-executable instructions;

[0129] A memory storing the computer-executable instructions that, when executed by the processor, cause the apparatus to perform:

[0130] When executing a first application:

[0131] Configure the thermal sensor according to a first set of sensor parameters;

[0132] When the thermal sensor is configured with the first set of parameters, extract biometric data from the sensor information; and

[0133] When a first condition is detected from the biometric data, start a second application; and

[0134] When executing the second application,

[0135] Configure the thermal sensor according to a second set of parameters, wherein the first set of parameters and the second set of parameters differ by at least one parameter; and

[0136] When the thermal sensor is configured with the second set of sensor parameters, extract the biometric data from the sensor information.

[0137] 4. A device that supports at least one smart application, the device comprising:

[0138] A thermal sensor interface configured to obtain sensor information from a thermal sensor and configure the thermal sensor according to a set of most comprehensive sensor parameters for all applications;

[0139] A processor for executing computer-executable instructions;

[0140] A memory that stores the computer-executable instructions, the computer-executable instructions when executed by the processor cause the device to perform:

[0141] Extract biometric data from the sensor information;

[0142] Execute the first application;

[0143] Execute the second application.

[0144] 5. A device that supports at least one smart application with more than one thermal sensor, the device comprising:

[0145] A first thermal sensor interface configured to obtain sensor information from a first thermal sensor;

[0146] A second sensor interface configured to obtain sensor information from a second thermal sensor;

[0147] A processor for executing computer-executable instructions;

[0148] A memory that stores the computer-executable instructions, the computer-executable instructions when executed by the processor cause the device to perform:

[0149] Configure the first thermal sensor according to the first set of sensor parameters;

[0150] Configure the second thermal sensor according to the second set of sensor parameters;

[0151] Extract biometric data from the sensor information from all the sensors;

[0152] Execute the first application:

[0153] Execute the second application.

[0154] In some embodiments, the set of configuration parameters for all sensors can be the same, in other words, all sensors can be configured with the most comprehensive set of parameters for all applications. The optimal sensor configuration can include but is not limited to the highest image resolution, number of bits, frame rate, sensitivity, and signal-to-noise ratio (SNR).

[0155] The following relates to an embodiment of continuous health monitoring of a vehicle operator.

[0156] Referring again to Figure 1 , while the embodiment supports using a thermal sensor to evaluate the health of a person in a room, the embodiment can utilize thermal sensor data to evaluate the health of a person in other types of enclosed spaces (such as a vehicle). The parameters used can include, but are not limited to, heart rate, respiratory rate, body surface temperature, posture (especially head position), and the trajectory of such data over time, etc.

[0157] The physical health condition of a vehicle operator (a vehicle driver) is crucial for the safety of the operator, passengers, and the vehicle itself. If an emergency occurs unexpectedly, the status of the vehicle operator's condition can determine the output of the situation.

[0158] In traditional methods, there are various ways to monitor the physical health of a vehicle operator through wearable devices. However, wearable devices are specific to the individual wearing the device rather than the vehicle, and may not ensure the safe monitoring of information or data on the health of the vehicle operator during vehicle use.

[0159] According to one aspect of the embodiment, the monitoring of the driver and / or the vehicle can be performed in a non-invasive and precise manner, which is continuously activated during vehicle operation. Thus, the health condition of anyone driving the vehicle can be evaluated. Through this method, biometric information about the driver is used for accident prevention, accident alerts, critical health warnings, and post-event analysis.

[0160] Figure 7 A vehicle system 700 for continuously monitoring the physical health of a vehicle operator according to an embodiment is shown.

[0161] Referring to Figure 2 , as discussed previously, this embodiment obtains thermal sensor data from a thermal sensor 204 via a thermal sensor interface 206.

[0162] The thermal sensor 204 is typically installed at a fixed position in front of the vehicle operator (driver), for example, mounted against the top windshield corner in front of the driver.

[0163] The processor 703 configures the thermal sensor by referring to the method in Figure 5 .

[0164] The processor 703 extracts biometric information contained in the sensor data 750. For example, once the driver sits in the driver's seat, the processor 703 can continuously monitor the driver's heart rate and head posture. Additionally, the driver's health record can be loaded into the processor 703 from a remote database server via a wireless device 704.

[0165] The processor 703 may determine that additional biometric data is needed based on the driver's health record. For example, if the driver's BMI exceeds a certain value, changes in heart rate, changes in body surface temperature, and changes in head posture over time may also be monitored.

[0166] As will be discussed in further detail, the processor 703 detects one or more current physical conditions of the driver and performs one or more actions to address the detected physical conditions.

[0167] The processor 703 may report the detected physical conditions to the driver, doctor, emergency contact, etc. by executing an application via the wireless device 704 (such as a smartphone), initiating a call to 911, generating an email sent to a designated person, etc.

[0168] The processor 703 may also initiate an action in response to the detected physical condition. For example, if the processor 703 determines that the driver is having a heart attack, the processor may instruct the autonomous driving interface 704 to route the vehicle to the nearest hospital.

[0169] As will be further discussed, the biometric information may be stored in the storage device 706 for subsequent analysis of the health condition of the vehicle driver. Although the storage device 706 is shown as a separate device, the storage device 706 may be integrated within the wireless device 704.

[0170] Figure 8 A process 800 for performing one or more actions based on the detected physical condition of the vehicle driver according to an embodiment is shown.

[0171] At block 801, the processor 703 extracts the biometric information contained in the sensor data 750. At block 802, the processor 703 processes the information transmitted in the signal 750 to extract measurements of one or more biometric characteristics of the vehicle driver. Biometric characteristics may include, but are not limited to, heart rate, respiratory rate, and deviation from the average heart rate (e.g., the degree of irregular heartbeat).

[0172] Measurements of biometric characteristics may be stored in the storage device 706 for analysis of the health condition of the vehicle driver at a later time. For example, the stored data may be evaluated by the driver's doctor to determine if treatment is needed.

[0173] At block 803, process 800 obtains measurements of biometric characteristics (e.g., heart rate and respiration rate of a vehicle driver) and determines whether a health profile is applicable to the driver. Multiple health profiles may be specified, where a first health profile maps to the driver's normal vital functions (in other words, no health event is detected), a second health profile maps to a heart attack event, a third health profile maps to the driver falling asleep, a fourth health profile maps to excessive alcohol consumption, etc.

[0174] If an abnormal health is detected at block 804 based on the determined health profile, process 800 detects at blocks 805 through 809 whether a specific health event has occurred. Based on the specific health event, process 800 performs an appropriate action. Exemplary actions include but are not limited to:

[0175] · Sleep event (block 805 - driver falls asleep): emit a loud warning sound via the vehicle radio or wireless device to warn the driver

[0176] · Heart attack event (block 806): instruct the autonomous driving interface to drive the vehicle to the nearest hospital

[0177] · Excessive alcohol consumption (block 807): prevent the vehicle driver from starting the vehicle or safely stop the vehicle if the car is in motion

[0178] · Arrhythmia event (block 808 - irregular heartbeat or arrhythmia): generate an alert to the driver via a wireless device

[0179] · Stroke event (block 809): instruct the autonomous driving interface to drive the vehicle to the nearest hospital

[0180] According to one aspect of the embodiment, the processing unit continuously monitors and analyzes the heartbeat of the vehicle driver to generate an alert regarding any irregularities. The processing unit may use a unique algorithm to provide this capability.

[0181] According to one aspect of the embodiment, the processing unit may identify the detected irregularities to correspond to one of multiple events of the vehicle driver, including but not limited to falling asleep, having a heart attack, excessive alcohol consumption, etc.

[0182] According to one aspect of the embodiment, data regarding the heartbeat of the vehicle driver may be stored in a storage device. The data may be retrieved at a later time to analyze whether an abnormal health event has occurred.

[0183] As previously discussed (e.g., as Figure 2The device 200 shown captures biometric data using thermal sensors 204 and 205. However, when the thermal sensor signals are blocked (e.g., by furniture in a smart home application), to enhance the robustness of the biometric system, one or more RF sensors (e.g., radar sensors) can be added to form a hybrid sensing system. Although it may be difficult to identify a user using only a radar sensor, the thermal signature of the user can be obtained from the thermal sensor array and correlated with the radar signal. Thus, as will be discussed, the use of both thermal and radar sensors can complement each other to improve the accuracy of hazard / action estimation.

[0184] Figure 9 Device 900 is shown interfacing with radar sensor 902 and thermal sensor 903 through radar sensor interface 904 and thermal sensor interface 905, respectively.

[0185] As will be discussed in more detail, radar sensor 904 can include a transmitter that emits radio frequency (RF) signals in the radar spectrum (e.g., operating at 20 GHz or 60 GHz) and one or more receivers that detect the reflected radar signals. Device 900 can then extract biometric data (e.g., respiration rate and motion vectors) from the detected reflected radar signals.

[0186] Thermal sensor 903 can be used for access control and presence detection. In some embodiments, to enable processor 901 to extract biometric data from the thermal sensor information, the performance of thermal sensor 903 can be extended by increasing the sample frequency (e.g., frame rate) of the captured image signal, identifying and tracking individuals from the image signal, and analyzing the detailed changes in the thermal image over time. Computing device 901 can convert the sensor information (signals) into biometric data such as heart rate, body position, health status, etc.

[0187] As will be discussed further in detail, computing device 901 can utilize the thermal signature derived from the thermal sensor data and the associated motion vectors to assist in processing the radar sensor data.

[0188] Device 900 can also support the prediction of future health events by processing the sensor signals and / or personalizing the support system.

[0189] In some embodiments, computing device 901 can process the thermal and radar sensor information to detect biometric data about the user. When the thermal signature of a specific individual is detected, computing device 901 can apply the profile (e.g., temperature settings) of that individual to a smart device (e.g., air conditioner) through output interface 909.

[0190] In some embodiments, computing device 901 may support a radio frequency (RF) sensor. The RF sensor may operate in the radar spectrum (5 - 60 GHz).

[0191] Computing device 901 may support one or more health applications that process and / or analyze biometric data and may generate notifications regarding the biometric data to external entities (e.g., a doctor) via interface 908 over communication channel 951. For example, a health application may detect a possible heart attack in a user from the biometric data; thus, an emergency notification regarding the event is sent to the user's doctor. In some embodiments, if the user is driving a vehicle, the health application may stop the vehicle (via output interface 909) and report to an emergency health service (via communication interface 908), and / or initiate an autonomous driving function (via output interface 909) to take the user to a hospital.

[0192] Device 900 may also interact with cloud server 910 to enable computing device 901 to access data from a remote database (e.g., the user's health records). For example, computing device 901 may change its decision based on the user's health records. Additionally, device 900 may continuously stream sensor data to cloud server 910 for storing or real-time analysis of the user's health condition.

[0193] Reference Figure 9 , a computing system environment may include a computing device in which the processes discussed herein (e.g., Figure 10 processes 1003 - 1006 as shown) may be implemented. Each process may correspond to a block of computer-readable instructions executed by computing device 901. The computing system may include computing device 901 for controlling the overall operation of the computing device and its associated components (including RAM, ROM, communication module, and first memory device 906). Computing devices generally include various computer-readable media. Computer-readable media may be any available media accessible by the computing device and include volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may comprise a combination of computer storage media and communication media.

[0194] Computer storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic tape cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by a computing device.

[0195] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery medium. A modulated data signal is a signal whose one or more characteristics are set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0196] In some embodiments, computing device 901 may execute computer-executable instructions stored at memory 906 and access archive data stored at memory 907.

[0197] In some embodiments, memory devices 906 and 907 may be physically implemented within a single memory device.

[0198] Figure 10 A hybrid sensing system 1000 is shown. Thermal and radar sensor data are captured separately by thermal sensor 1001 and radar sensor 1002, respectively. Depending on the complexity of the application, thermal sensor 1001 may comprise a 32×32 thermopile array, and radar sensor 1002 may operate at 20 GHz with 1 transmit antenna and 3 receive antennas.

[0199] Thermal signal 1051 and radar signal 1052 are processed by corresponding analysis models 1003 and 1004, respectively, to obtain biometric vectors with associated confidence levels.

[0200] Model 1003 may be a neural network model pre-trained using pre-processed thermal images and the resulting feature vectors. Model 1004 may be another neural network model pre-trained using pre-processed RF signals and the resulting feature vectors. Referring Figure 11 , model 1103 may contain models 1003 and 1004 as a first-stage process and additional neural network layers as a second-stage process. The additional neural network layers may be trained using the feature vectors as input and the synthetic hazards and confidence levels as output.

[0201] As will be discussed, the feature vectors 1053 and 1054 are obtained from the biometric vectors.

[0202] The biometric data extracted from the thermal sensor 1001 can include, but is not limited to, respiration rate, heart rate, body surface temperature, thermal signature, and motion vector. The biometric data extracted from the radar sensor 1002 can include, but is not limited to, respiration rate and motion vector. During the sensor data acquisition phase, the thermal signature and the associated motion vector 1055 from the thermal sensor model 1003 can be provided to the radar sensor model 1004 to assist in its data analysis and processing.

[0203] The feature vectors 1053 and 1054 are then passed to the feature analysis block 1005, where the vectors 1053 and 1054 are compared to obtain a composite feature vector 1056. The composite feature vector 1056 is then passed to the decision logic block 1006 for risk analysis to provide a hazard level 1057 and a corresponding confidence level 1058.

[0204] A hypothetical example of the device 1000 is discussed below, where the biometric data vector V r and V t convey one or more features. In addition, V t can convey additional features not conveyed by V r .

[0205] The biometric data vector V r from the radar sensor model 1004 can be V r = [B rb , C rb , B rx , B ry , C rm , where B rb is the measured respiration rate, and C rh is the confidence level of the measured respiration rate, ranging from 0.0 to 1.0, with increasing confidence level. B rx and B ry are the components of the motion vector in X and Y, respectively, with a confidence level of C rm . A confidence level of 0 indicates an uncertain measurement result.

[0206] Meanwhile, the biometric data vector V t from the thermal sensor 1001 can be V t = [B tb , C tb , B tx , B ty , C tv , where B tb and Ctb is the measured respiration rate and the associated confidence level, and B tx , B ty and C tm are the motion vectors in X and Y and the associated confidence level C tm .

[0207] Biometric vector V r and V t include measurements of one or more biometric characteristics and the corresponding confidence levels. For example, for the example shown above, V r and V t span the respiration rate and motion characteristics.

[0208] Feature vectors 1053 and 1054 are then processed by feature analysis block 1005. An exemplary flowchart 1200 of feature analysis logic block 1005 is shown in Figure 12 . The same type of biometric characteristic (e.g., respiration rate) can be passed through a decision matrix to obtain a synthetic feature vector V1056.

[0209] For example, when the biometric vectors at time t1 are V r1 = [25,0.90,0,0,0.80] and V t1 = [23,0.80,0,0,0.90], the respiration rate vectors 1053 and 1054 are equal to [25,0.9] and [23,0.8] respectively.

[0210] Referring to Figure 12 the flowchart 1200 shown, process 1200 determines the synthetic feature vector (V) 1056, where V = [B,C], B is the measured feature value, and C is the corresponding confidence level. For example, the biometric characteristic can be the respiration rate.

[0211] Feature vector 1053 V t = [B t ,C t and feature vector 1054 V r = [B r ,C r . For flowchart 1200, C min = 0.6. C min is the minimum confidence level of the measurements to be considered. The confidence level can be selected based on empirical results. It can also be made adaptive based on the quality of the sensor signals.

[0212] In step 1201, if C t and C r are both less than C min= 0.6, the result is indeterminate. Otherwise, in step 1202, process 1200 will discard the eigenvectors with a confidence level less than C min = 0.6. In other words, in step 1203, if C r < 0.6, then V = V t . Otherwise, V = V r .

[0213] In step 1204, process 1200 determines B diff and B th , where B diff = |B t - B r |, and B th = 0.1 * (0.5 * (B t + B r ))

[0214] If B diff < B th , as determined in step 1205, then in step 1206, V = [B a , C s , where B a = 0.5 * (B t + B r ) and C s = 0.5 * (C t + C r )

[0215] Otherwise, if Bdiff >= Bth, if Ct > Cr, as determined in step 1207, then V = [Bt, Ct]. Otherwise, V = [Br, Cr].

[0216] For example, let the biometric vector V t = [25, 0.9] and V r = [23, 0.8]. According to process 1200, by processing V t and V r through the feature analysis logic block 1005, the synthetic eigenvector [24, 0.85] is determined, corresponding to a respiration rate of 24 and a confidence level of 0.85

[0217] If the biometric vector spans more than one feature (e.g., respiration rate and movement), then each feature can be processed separately by process 1200. For example, by processing the biometric vectors V r1 = [25, 0.90, 0, 0, 0.80] and V t1 = [23, 0.80, 0, 0, 0.90], the synthetic biometric vector feature V will be: V1 = [24, 0.85, 0, 0, 0.85]

[0218] When there is only one biometric vector V r or V t contains the feature information (B i , C i ) of the i-th feature (e.g., heart rate), when Ci >= C min , the feature analysis logic block 1005 can use the feature information (B i , C i ) to construct a synthetic biometric vector.

[0219] Return reference Figure 10 , the decision logic block 1006 determines the hazard level 1057 and its associated confidence level 1058. For the above example of V1, since the respiratory rate is normal, the potential hazard is low, with a confidence level of, for example, 0.7 (high), although no movement is detected for the identified subject (user). However, the value of the confidence level can be adjusted based on the empirical results associated with the embodiment. The decision logic block 1006 continuously monitors the synthetic feature vector 1056 and its evolution over time to determine the potential hazard measure (the corresponding hazard warning 1057). For example, the synthetic biometric vectors for eight consecutive measurements can be: V1 = [24, 0.85, 0, 0, 0.85], V2 = [23, 0.87, 0, 0, 0.88], V3 = [21, 0.89, 0, 0, 0.82], V4 = [18, 0.85, 0, 0, 0.85], V5 = [14, 0.87, 0, 0, 0.90], V6 = [9, 0.86, 0, 0, 0.95], V7 = [5, 0.83, 0, 0, 0.90], and V8 = [0, 0.85, 0, 0, 0.90].

[0220] The decision logic block 1006 can determine that the hazard level is high and the confidence level (0.6) is medium at the 5th measurement because the subject's respiratory rate is decreasing and no movement is detected. However, the decision logic block 1006 can determine that the hazard level is high and the confidence level (0.8) is very high at the 7th measurement because the subject's respiratory level is rapidly dropping to a dangerous level and no movement is detected in seven consecutive measurements.

[0221] If more biometric data (corresponding to additional features) is obtained from the sensors 1001 and / or 1002, the detection accuracy can be improved. For example, the heart rate and thermal features can also be obtained from the thermal sensor 1001. For example, V t = [B tb , C tb , B th , C th , B tt , C tt , B ts1,C ts1 ,B tx ,B ty ,C tv , where B tb ,C tb corresponds to the measured respiratory rate, B th ,C th corresponds to the measured heart rate, B tt ,C tt corresponds to the measured body surface temperature, B ts1 ,C ts1 corresponds to the thermal signature of an associated entity (e.g., registered user 1), while B tx ,B ty and C tm correspond to the motion vectors in X and Y, with a confidence level of C tm .

[0222] Referring to the previous example, assume the biometric vector is V r1 = [25, 0.90, 0, 0, 0.80] and V t1 = [23, 0.80, 80, 0.85, 37.0, 0.90, 1, 0.90, 0, 0, 0.90]. The synthesized biometric vector (determined by the feature analysis block 1005 according to the process 1200) is V1 = [24, 0.85, 80, 0.85, 37.0, 0.90, 1, 0.90, 0, 0, 0.85]. From the decision logic block 1006, the hazard level will be low, with a confidence level of 0.8 (high), because all the biometric data of the identified subject (user 1) are normal.

[0223] For example, the time series of synthetic biometric vectors can be V1 = [24, 0.85, 80, 0.85, 37.0, 0.90, 1, 0.90, 0, 0, 0.85], V2 = [23, 0.87, 77, 0.90, 37.1, 0.90, 1, 0.90, 0, 0, 0.88], V3 = [21, 0.89, 72, 0.87, 37.0, 0.87, 1, 0.90, 0, 0, 0.82], V4 = [18, 0.85, 66, 0.85, 37.0, 0.90, 1, 0.90, 0, 0, 0.85], V5 = [14, 0.87, 60, 0.86, 36.9, 0.89, 1, 0.90, 0, 0, 0.90], V6 = [9, 0.86, 55, 0.88, 36.9, 0.90, 1, 0.90, 0, 0, 0.95], V7 = [5, 0.83, 50, 0.86, 36.7, 0.90, 1, 0.90, 0, 0, 0.90], and V8 = [0, 0.85, 45, 0.87, 36.7, 0.90, 1, 0.90, 0, 0, 0.90].

[0224] For the above time series, the decision logic block 1006 can estimate the hazard level as high with a confidence level of 0.8 (high) for the identified subject (User 1) after the 5th measurement because both the respiratory rate and the heart rate are decreasing together. Additionally, since User 1 is detected via thermal characteristics, more valuable information can be provided, such as reporting the hazard level and the confidence level to User 1's registered hospital and preparing User 1's medical care before arriving at the hospital.

[0225] Although not explicitly shown in Figure 10 , in some embodiments, the hazard prediction can also be done separately by using the machine learning models 1003 and 1004 alone.

[0226] For Figure 11 the embodiment shown, an alternative approach is to directly feed the outputs 1151 and 1152 from the sensors 1101 and 1102 into the analysis model 1103. The analysis model 1103 can be an artificial neural network pre-trained with artificial vectors extracted from clinical data. The analysis model 1103 can be fine-tuned (re-trained) with empirical data directly obtained from the field.

[0227] In some embodiments, the combination of the thermal sensor and the radar sensor can be installed by an associated entity (e.g., a house or an enterprise) to ensure the desired coverage at the best system cost in smart home / enterprise applications. For example, the thermal sensor and the radar sensor can be installed in the living room and each bedroom. However, only the thermal sensor array can be installed near the toilet and in the kitchen and / or garage.

[0228] In the case of another smart home application, two thermal sensors and one radar sensor may be installed in the living room. Signals 1351 and 1352 from the two thermal sensors 1301 and 1302 can be preprocessed by a signal preprocessing unit 1303, as Figure 13 shown, to obtain a synthetic thermal sensor signal 1353, which can be provided to Figure 10 or Figure 11 the system shown. In some embodiments, the signal preprocessing may include high-pass filtering the image signal over time and selecting the image signal with a higher residual. In some embodiments, the signal preprocessing unit 1303 may include stitching the two thermal sensor signals 1351 and 1352 to form a better thermal sensor signal. Alternatively, the machine learning models 1003 and 1004 (as Figure 10 shown) or the machine learning model 1103 (as Figure 11 shown) can be retrained for a specific combination of thermal and radar sensors without signal preprocessing.

[0229] In some embodiments, the hybrid sensing system can be inside a vehicle to increase the accuracy of monitoring the physical health of the vehicle operator (the vehicle driver), thereby reducing the probability of false detection. The state of the vehicle operator's condition can determine the action plan when an emergency occurs.

[0230] Referring to Figure 14 the hybrid sensing system 1400 continuously monitors the physical health of the vehicle operator for vehicle applications as shown. The hybrid sensing system 1400 includes a radar sensor 902 and a thermal sensor 903.

[0231] The radar sensor 902 includes an RF transmitter 1405, which is typically mounted in a fixed position in front of the vehicle operator (driver) (e.g., mounted at the top windshield corner in front of the driver such that the signal from the transmitter is reflected by the driver's body to the receiver without being blocked by any non - interesting moving objects such as the steering wheel). The RF transmitter 1405 generates a fixed - frequency signal according to the RF characteristics of the vehicle, e.g., between 20 - 30 GHz and with a power level between 0.1 - 0.5 watts.

[0232] The RF signals received by the receivers 1406 - 1408 are processed by an embedded microcontroller unit (MCU) 1410 to obtain a radar signal 1451, which is suitable for the corresponding machine learning model.

[0233] The thermal sensor 903 is typically physically located next to the radar sensor 902 to capture thermal signals 1452 from the driver's head and body via the thermal sensor array 1411. The thermal signals 1452 are processed by the embedded MCU 1412 to output a signal 1453, which is suitable for the corresponding machine learning model.

[0234] The core processor 1401 executes computer-readable instructions stored at the device 1402 to support the machine learning models of the thermal sensor 903 and the radar sensor 902, the feature analysis block 1005, and the decision logic block 1006, as Figure 10 shown. In some embodiments, the core processor 1401 may have machine learning models implemented for both sensor types, as Figure 11 shown.

[0235] When the core processor 1401 detects any dangerous physical condition of the driver, it can perform one or more actions to address the detected physical condition. The list of actions includes but is not limited to reporting the detected physical condition to the driver, doctor, or emergency contact via the embedded wireless device 1403 (e.g., LTE module) installed in the hybrid sensing system 1400, initiating a phone call to 911, and generating an email message to a designated person.

[0236] The wireless device 1403 may allow the driver's health records to be loaded into the core processor 1401 from a remote database server via the wireless device. The parameters of the decision logic supported by the core processor 1401 can be changed based on the driver's health records.

[0237] In addition, the wireless device 1403 may also allow sensor data to be continuously streamed to the cloud server 1413 for storing or real-time analysis of the driver's health condition (if the analysis model is selected to be executed in the cloud instead of locally or both locally and in the cloud for cross-checking purposes). The cloud server 1413 can use the data from the hybrid sensor system 1400 to fine-tune the analysis model. In addition, the system 1400 can train a new model based on the new sensor combination. The retrained or new analysis model can be downloaded from the cloud server 1413 to the core processor 1401 to continuously improve the accuracy of the analysis model.

[0238] The information downloaded from the cloud server 1413 may include the user (driver)'s chronological health records and behavior records for reference. For example, if the downloaded behavior records indicate that the user previously drove while intoxicated, the system 1400 can extract relevant feature information that may indicate intoxication so that appropriate actions can be taken.

[0239] In addition, if the system 1400 detects a hazard, the system 1400 can send a record of the driver's behavior to a control center to assist in the decision-making process. When a hazard occurs, a log of health records and the driver's behavior records can be sent to emergency services to allow for better preparation when the user arrives at the hospital.

[0240] All downloaded information (e.g., sensor data and system outputs) can be temporarily stored in the storage device 1402 (which can also store computer-readable instructions as discussed previously). The data may be retained until cleared by an authorized person, e.g., after backing up the data at the end of the working day. Additionally, the data may be retrieved by an authorized person under certain conditions, e.g., when an accident involving the user occurs.

[0241] The core processor 1401 can store biometric data (e.g., sensor data, biometric vectors, and / or synthetic biometric vectors) at the storage device 1402. The stored biometric data can subsequently be retrieved to reconstruct the health events that occurred or provide data for legal evidence.

[0242] When computing power is available, the core processor 1401 can implement advanced functions. For example, if the core processor 1401 determines that the user (driver) is experiencing a heart attack, it can instruct the activation of the autonomous driving unit via the interface 1404 to route the vehicle to the nearest hospital.

[0243] As previously discussed, in some cases, the thermal signal (e.g., from the thermal sensor 903) may be blocked (and thus unavailable); however, the received RF signal (e.g., from the radar sensor 902) can be processed on its own to evaluate the physical health of the vehicle operator.

[0244] Figure 15 An embodiment of the processing that the system 1400 can support is shown. The sequence of actions is similar to the sequence of actions discussed with respect to Figure 10 The driver's health record 1551 and an overview of the associated actions for each hazard / each detected hazard 1552 can be loaded into the decision logic block 1504. By presenting the feature vector to the decision logic, the block 1504 is able to detect a specific hazard and execute the associated action 1554 based on the presented feature vector 1553.

[0245] Figure 16Flowchart 1600 shows decision logic block 1504. Based on the driver's health record, different weights can be applied to the elements in the synthetic feature vector (e.g., corresponding to different feature information). For example, if the driver is over a certain age (e.g., 55 years old) and obese (e.g., 85 kg and 1.68 m), more weight can be applied to the change in heart rate over time and the respiratory rhythm. As another example, if the user (driver) has a record of being a careless driver, more weight can be applied to the motion vector. Then the weighted feature vector is mapped to a hazard list. An exemplary hazard list can be:

[0246]

[0247] When a match for a hazard type is found, an associated action list can be executed. For example, if it is determined that the driver is tired and prone to driving while asleep, the alarm system in the vehicle can be activated to wake up the driver. As another example, if abnormal critical biometric data is detected (e.g., abnormal change in heart rate), the alarm system can instruct the driver to pull the vehicle over to the side of the road and can send a message to the control center to request a backup driver to take over the driving. As another example, if the driver is detected to be unconscious, the alarm system in the vehicle can stop the vehicle. In addition, the alarm system can automatically request emergency health services. If the vehicle can drive autonomously, the vehicle can drive itself to the nearest emergency health service department.

[0248] As those skilled in the art can understand, a computer system having an associated computer-readable medium containing instructions for controlling the computer system can be used to implement the exemplary embodiments disclosed herein. The computer system can include at least one computer, such as a microprocessor, a digital signal processor, and associated peripheral electronic circuits.

Claims

1. An apparatus for supporting biometric data of a user, the apparatus comprising: A thermal sensor interface configured to obtain a thermal signal from a thermal sensor; A radar sensor (RF) interface configured to obtain a radar signal from a radar sensor; A processor for executing computer-executable instructions; A memory storing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the apparatus to perform: Generate a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric characteristic, and wherein the first biometric vector contains third information about a second biometric characteristic; Generate a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric characteristic, and wherein the second biometric vector contains fourth information about the second biometric characteristic; Extract first composite information about the first biometric characteristic from the first information and the second information; Extract second composite information about the second biometric characteristic from the third information and the fourth information; Generate a first composite biometric vector, the first composite biometric vector containing: The first composite information about the first biometric characteristic; And The second composite information about the second biometric characteristic; Apply a first weight to the first composite information about the first biometric characteristic and a second weight to the second composite information about the second biometric characteristic based on the user's health record, wherein the first weight and the second weight are different; And Determine hazard information from the first composite biometric vector, wherein the hazard information includes a hazard level and a confidence level, and wherein the hazard level indicates the occurrence of a health event of the user and the confidence level indicates the degree of certainty of the hazard level.

2. The device according to claim 1, wherein The computer-executable instructions when executed by the processor further cause the apparatus to perform: When the thermal signal is unavailable, extract the first composite information about the first biometric characteristic only from the second information containing the second biometric vector from the radar signal.

3. The apparatus according to claim 1, comprising: A communication interface, Among them, The computer-executable instructions when executed by the processor further cause the apparatus to perform: Extract a thermal feature from the thermal signal, wherein the thermal feature identifies the user; And When the health event is detected, send a message indicating the hazard level about the user through the communication interface.

4. The apparatus according to claim 3, wherein the hazard information includes a hazard type, and wherein the computer-executable instructions when executed by the processor further cause the apparatus to perform: Download and store the user's health record; Include at least a portion of the health record in the message; Identify an action mapped to the hazard type; and Perform the identified action on behalf of the user.

5. The device according to claim 1, wherein, The computer-executable instructions, when executed by the processor, further cause the device to perform: Determine a hazard type from the first synthetic biometric vector; Identify an action mapped to the hazard type; and Perform the identified action on behalf of the user.

6. The device according to claim 5, wherein the hazard type is one of a plurality of hazard types, and wherein the memory stores computer-executable instructions, which, when executed by the processor, further cause the device to perform: Group the plurality of hazard types having different importance levels; and Generate an alarm warning based on the different importance levels.

7. The device according to claim 5, wherein The computer-executable instructions, when executed by the processor, further cause the device to perform: Generate a time series of synthetic biometric vectors, wherein the time series includes the first synthetic biometric vector and the second synthetic biometric vector corresponding to a first time instance and a second time instance, respectively; and Determine the hazard level from the time series of synthetic biometric vectors.

8. The apparatus according to claim 1, wherein, The computer-executable instructions, when executed by the processor, further cause the device to perform: Train a first model based on previously occurring thermal signals; Transform the thermal signals according to the first model; Obtain the first biometric vector from the transformed thermal signals; Train a second model based on previously occurring radar signals; Provide thermal features and motion vectors from the first model to the second model; Transform the radar signals according to the second model, the thermal features, and the motion vectors; and Obtain the second biometric vector from the transformed radar signals.

9. The device according to claim 8, wherein the first model includes a first neural network model, wherein the second model includes a second neural network model, and wherein the memory stores computer-executable instructions, which, when executed by the processor, further cause the device to perform: Pre-train the first neural network model using a preprocessed image and a first synthetic feature vector; and Pre-train the second neural network model using a preprocessed RF signal and a second synthetic feature vector.

10. The device according to claim 9, wherein the combined neural network model includes a first neural stage and a second neural stage, and wherein the memory stores computer-executable instructions, which, when executed by the processor, further cause the device to perform: Apply the first neural network model and the second neural network model to the first neural stage; and Train the second neural stage with the synthetic feature vector as an input and a synthetic hazard and a confidence level as outputs.

11. The device according to claim 8, wherein, The computer-executable instructions, when executed by the processor, further cause the device to perform: Download the first model and the second model from a cloud server; Receive updated model information from the cloud server; and Update at least one of the first model and the second model based on the updated model information.

12. The device according to claim 11, wherein, The computer-executable instructions, when executed by the processor, further cause the device to perform: Send biometric information about the first biometric vector and the second biometric vector to the cloud server.

13. A method for assessing a user's health condition, the method comprising: Obtain a thermal signal from a thermal sensor; Obtain a radar signal from a radar sensor; Generate a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric characteristic, and wherein the first biometric vector contains third information about a second biometric characteristic; Generate a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric characteristic, and wherein the second biometric vector contains fourth information about the second biometric characteristic; Extract first synthetic information about the first biometric characteristic from the first information and the second information; Extract second synthetic information about the second biometric characteristic from the third information and the fourth information; Generate a first synthetic biometric vector, the first synthetic biometric vector containing: The first synthetic information about the first biometric characteristic; And The second synthetic information about the second biometric characteristic; Apply a first weight to the first synthetic information about the first biometric characteristic and a second weight to the second synthetic information about the second biometric characteristic based on the user's health record, wherein the first weight and the second weight are different; And Determine hazard information from the first synthetic biometric vector, wherein the hazard information includes a hazard level and a confidence level, and wherein the hazard level indicates the occurrence of a health event of the user and the confidence level indicates the degree of certainty of the hazard level.

14. The method according to claim 13, comprising: When the thermal signal is unavailable, extract the first synthetic information about the first biometric characteristic only from the second information containing the second biometric vector from the radar signal.

15. The method according to claim 13, comprising: Extract a thermal feature from the thermal signal, wherein the thermal feature identifies the user; and When the health event is detected, send a message indicating the hazard level about the user.

16. The method according to claim 15, wherein the hazard information includes a hazard type, the method comprising: Download and store the user's health record; Include at least a portion of the health record in the message; Identify an action mapped to the hazard type; and Perform the identified action on behalf of the user.

17. The method according to claim 13, comprising: Send the first biometric vector and the second biometric vector to a cloud server; and Receive the hazard information from the cloud server.

18. The method according to claim 13, comprising: Determine a hazard type from the first synthetic biometric vector; Identify an action mapped to the hazard type; and Perform an action for recognition on behalf of the user.

19. The method according to claim 13, wherein the first biometric vector contains fifth information about a third biometric characteristic, and the second biometric vector does not contain any information about the third biometric characteristic, the method comprising: Extract second synthetic information about the third biometric characteristic only from the fifth information.

20. A device for providing an assessment of a vehicle driver of a vehicle, the device comprising: A thermal sensor interface configured to obtain a thermal signal from a thermal sensor; A radar sensor (RF) interface configured to obtain a radar signal reflected by the vehicle driver from a radar sensor; A processor for executing computer-executable instructions; A memory storing the computer-executable instructions, the computer-executable instructions, when executed by the processor, cause the device to perform: Generate a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric characteristic, and wherein the first biometric vector contains third information about a second biometric characteristic; Generate a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric characteristic, and wherein the second biometric vector contains fourth information about the second biometric characteristic; Extract first synthetic information about the first biometric characteristic from the first information and the second information; Extract second synthetic information about the second biometric characteristic from the third information and the fourth information; Generate a first synthetic biometric vector, the first synthetic biometric vector containing: The first synthetic information about the first biometric characteristic; And The second synthetic information about the second biometric characteristic; Download the medical history health record and behavior record of the vehicle driver; Based on the medical history health or the behavior record of the vehicle driver, weight the first synthetic biometric vector by applying a first weight to the first synthetic information about the first biometric characteristic and applying a second weight to the second synthetic information about the second biometric characteristic, wherein the first weight and the second weight are different; Determine a hazard type from the first synthetic biometric vector weighted based on the medical history health record or the behavior record, wherein the hazard type indicates the occurrence of a health event of the vehicle driver; Identify an action mapped to the hazard type; And Perform the identified action on behalf of the vehicle driver.

21. The device according to claim 20, wherein the identified action indicates a switch to an autonomous driving function, the device comprising: An autonomous driving interface, wherein the computer-executable instructions, when executed by the processor, further cause the device to perform: Through the autonomous driving interface, instruct the vehicle to autonomously drive to an emergency health service department.

22. The device according to claim 20, wherein, The computer-executable instructions, when executed by the processor, further cause the device to perform: When the health event is detected, send the event health and behavior record to the emergency service department.

Images