Biometric data capture and analysis

Abstract

The invention discloses biometric data capture and analysis. The health condition of the person may be evaluated from the thermal sensor signals. By increasing performance metrics (e.g., resolution, frame rate, sensitivity) of the thermal camera, operations may be extended to authentication, biometric data extraction, health analysis, and the like. The prediction may be performed by monitoring a time series of thermal images, and thus an early warning of the health condition may be provided. The apparatus may be used, but not limited to, to personalize smart home devices through supervision and reinforcement learning. The device can be applied to, but not limited to, smart home, smart building, smart vehicle and the like.

Description

[0001] This application is a divisional application of an application with a filing date of September 5, 2019, application number 201980059733.3, and invention name “Biometric Data Capture and Analysis”.

[0002] This patent application claims priority to U.S. Provisional Patent Application Serial No. 62 / 730,160, filed on September 12, 2018, entitled “BIOMETRIC DATA CAPTURE AND ANALYSIS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] Aspects of the present disclosure relate to extracting biometric data from a thermal sensor image sequence.A thermal sensor may include an array of thermal sensitive elements to improve performance. Background Art

[0004] Image sensors are commonly used in home applications. Examples include those used in baby monitors, Internet Protocol (IP) cameras, security cameras, etc. Other image sensors include thermal cameras and thermal sensor arrays. Expanding the effective applications of image sensors will increase their popularity.

[0005] The need for expanded use of sensors, such as thermal sensors, was highlighted in a May 2019 Chicago Sun Times article about an Illinois man who suffered a cardiac-related event while driving and crashed his car and died. The man suffered a “cardiac-related event” while driving, lost consciousness, and crashed his vehicle into a utility pole. After hitting the pole, his car crashed into another vehicle. Preventive measures to deal with such horrific events would certainly benefit the general public. Summary of the invention

[0006] A device uses a thermal sensor for biometric data extraction and tracking for smart home applications. Applications include health status analysis, motion estimation (e.g., fall estimation), casual prediction (e.g., heart rate is slowing to a hazardous level), hazard detection (e.g., lying down for a long time), learning personal profiles, and system adaptation based on personal preferences.

[0007] On the other hand, the parameters of the thermal sensor can be enhanced to allow the extraction of as much data as possible. Examples include, but are not limited to: increasing the number of sensing elements (ie, resolution), frame rate, sensitivity, and / or signal-to-noise levels.

[0008] On the other hand, signal processing techniques extract biometric data from thermal images.

[0009] On the other hand, analytical models are used for hazard prediction and the associated actions taken subsequently.

[0010] On the other hand, hazard analysis is performed through deep learning models. Actions are taken based on the hazard coefficient and the associated confidence level estimated from the model.

[0011] On the other hand, the model will recommend actions to be taken with an associated confidence level based on the input data sequence.

[0012] On the other hand, the model can be trained to predict the hazard factor and corresponding actions when necessary based on previously occurring events with corresponding confidence levels.

[0013] On the other hand, for less time-critical applications, the model can reside in a cloud server instead of a local processing unit.

[0014] On the other hand, the parameters of the smart device are configured differently based on the thermal signature of the detected person.

[0015] On the other hand, based on the detected condition detected by the first application, the executed application is changed from the first application to the second application.

[0016] This application provides the following: 1). A device supporting at least one smart device, the device comprising: Smart device interface; a thermal sensor interface configured to obtain sensor information from a first thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: detecting a detected thermal signature of a detected user from the sensor information; When the detected user is a first user, obtaining a first profile corresponding to the first user, wherein the first profile includes a first set of smart device parameters; and When the detected user is the first user, a smart device is configured based on the first set of smart device parameters through the smart device interface.

[0017] 2). The device according to 1), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When the detected user is a second user, obtaining a second profile corresponding to the second user, wherein the second profile includes a second set of smart device parameters, and wherein the second set is different from the first set; and When the detected user is the second user, the smart device is configured based on the second set of smart device parameters through the smart device interface.

[0018] 3). The device according to 2), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: trained to detect a first thermal signature of the first user and a second thermal signature of the second user; and In response to the training, the first user and the second user are distinguished based on the received thermal signature.

[0019] 4). The device according to 2), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When both the first user and the second user are detected, the smart device is configured at a compromise setting of the first setting and the second setting of the first user and the second user, respectively.

[0020] 5). A device supporting at least one smart application, the device comprising: a thermal sensor interface configured to obtain sensor information about the person from a thermal sensor; a processor for executing computer-executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: When the first application is executed, configuring the thermal sensor according to a first set of sensor parameters; extracting biometric data from the sensor information when the thermal sensor is configured with the first set of parameters; and When a first condition is detected from the biometric data, launching a second application; and When the second application is executed, configuring the thermal sensor according to a second set of parameters, wherein the first set of parameters and the second set of parameters differ by at least one parameter; and The biometric data is extracted from the sensor information when the thermal sensor is configured with the second set of parameters.

[0021] 6). The device according to 5), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: A third application is launched when a second condition is detected from the biometric data, wherein the second condition is different from the first condition.

[0022] 7). The device according to 5), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: When the first condition indicates that the person is experiencing an abnormal health event, a notification regarding the first condition is generated.

[0023] 8). The device according to 5), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: An occurrence of a health event for the person is predicted based on the biometric data.

[0024] 9). The device according to 6), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: Determine the level of hazard of a health event; and An action is generated from a plurality of actions based on the risk level.

[0025] 10). A method for assessing the health status of a vehicle driver, the method comprising: generating, by a thermal sensor, a continuous image signal reflected by a driver of the vehicle; extracting, by a processing unit, a first measurement of a first biometric characteristic of the vehicle driver from the continuous image signal; mapping the first measurement to a first health profile among a plurality of health profiles; and Detect whether the first health record indicates a first abnormal health event.

[0026] 11). The method according to 10), further comprising: The heartbeat characteristics of the vehicle driver are monitored.

[0027] 12). The method according to 11), further comprising: detecting irregularities from the heartbeat characteristics of the vehicle driver; and In response to the detecting, an alert is generated regarding the irregularity.

[0028] 13). The method according to 12), further comprising: Distinguish one irregularity from multiple irregularities.

[0029] 14). The method according to 11), further comprising: Data about the heartbeat characteristics is stored.

[0030] 15). The method according to claim 10, further comprising: When the first abnormal health event is detected, a first action is initiated by the processing unit.

[0031] 16). The method according to 10), further comprising: extracting, by the processing unit, a second measurement of a second biometric characteristic of the vehicle driver; mapping the first measurement and the second measurement to a second health profile among the plurality of health profiles; detecting whether the second health record indicates a second abnormal health event; and When the second abnormal health event is detected, a second action is initiated by the processing unit.

[0032] 17). The method according to 10, further comprising: obtaining a thermal sensor signal from the thermal sensor; and Biometric data is extracted from the thermal sensor signal by the processing unit.

[0033] 18). An apparatus for providing an assessment of a vehicle driver of a vehicle, the apparatus comprising: a thermal sensor configured to receive a continuous sequence of images reflected by a driver of the vehicle; a processing unit for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processing unit, cause the apparatus to perform: extracting, by the processing unit, a first measurement of a first biometric characteristic of the vehicle driver from the image signal; mapping the first measurement to a first health profile among a plurality of health profiles; detecting whether the first health record indicates a first abnormal health event; and When the first abnormal health event is detected, a first action is initiated by the processing unit.

[0034] 19). The device according to 18), further comprising: Autonomous driving interface; and The memory stores the computer executable instructions, which when executed by the processing unit further cause the device to perform: The vehicle is instructed to drive to an appropriate destination through the autonomous driving interface.

[0035] 20). The device according to 18), wherein the memory stores the computer executable instructions, which when executed by the processing unit further cause the device to perform: extracting, by the processing unit, a second measurement of a second biometric characteristic of the vehicle driver; mapping the first measurement and the second measurement to a second health profile among the plurality of health profiles; detecting whether the second health record indicates a second abnormal health event; and When the second abnormal health event is detected, a second action is initiated by the processing unit.

[0036] 21). A device supporting at least one smart application, the device comprising: a thermal sensor interface configured to obtain sensor information from the thermal sensor array and configure the thermal sensors according to the most comprehensive set of sensor parameters for all applications; a processor for executing computer-executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: extracting biometric data about the person from the sensor information; executing a first application; The second application is executed.

[0037] 22). The device according to 21), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: A third application is launched when a second condition is detected from the biometric data regarding the person, wherein the second condition is different from a previous condition regarding the person.

[0038] 23). The device according to 22), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: When the previous condition indicates that the person is experiencing an abnormal health event, a notification regarding the previous condition is generated.

[0039] 24). The device according to 21), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: An occurrence of a health event for the person is predicted based on the biometric data.

[0040] 25). The device according to 22), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: Determine the level of hazard of a health event; and An action is generated from a plurality of actions based on the risk level.

[0041] 26). A device supporting at least one smart application having more than one sensor to form a sensor array, the device comprising: a first thermal sensor interface configured to obtain first sensor information about a person from a first thermal sensor; a second sensor interface configured to obtain second sensor information about the person from a second thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: configuring the first thermal sensor according to a first set of sensor parameters; configuring the second thermal sensor according to a second set of sensor parameters, wherein the first set of sensor parameters and the second set of sensor parameters are the same; extracting biometric data from sensor information from all of said sensors; executing a first application; and The second application is executed.

[0042] 27). The device according to 26), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: A third application is launched when a second condition is detected from the biometric data, wherein the second condition is different from a previous condition regarding the person.

[0043] 28). The device according to 27), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: When executing the first application: When the previous condition indicates that the person is experiencing an abnormal health event, a notification regarding the previous condition is generated.

[0044] 29). The device according to 26), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: An occurrence of a health event for the person is predicted based on the biometric data.

[0045] 30). The device according to 27), wherein the memory stores computer executable instructions, which when executed by the processor further cause the device to perform: Determine the level of hazard of a health event; and An action is generated from a plurality of actions based on the risk level. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The foregoing summary of the invention, as well as the following detailed description of exemplary embodiments of the invention, will be better understood when read in conjunction with the accompanying drawings, which are included by way of example and not as limitations of the invention as claimed.

[0047] Figure 1 A thermal sensor located in a room is shown according to an embodiment.

[0048] Figure 2 An apparatus interfacing with one or more thermal sensors and one or more associated smart devices is shown according to an embodiment.

[0049] Figure 3a An apparatus for processing information from one or more thermal sensors according to an embodiment is shown.

[0050] Figure 3b An apparatus for processing information from one or more thermal sensors using a deep learning model to estimate a hazard factor according to another embodiment is shown.

[0051] Figure 3c An apparatus for processing information from one or more thermal sensors using a deep learning model to suggest actions is shown according to another embodiment.

[0052] Figure 4 A process of identifying a user from thermal sensor information and applying a corresponding profile according to an embodiment is shown.

[0053] Figure 5 A flow chart for executing multiple applications according to an embodiment is shown.

[0054] Figure 6A flow chart for configuring a smart device based on one of a plurality of parameter groups of a detected thermal signature is shown according to an embodiment.

[0055] Figure 7 A vehicle system for continuously monitoring the physical health of a vehicle operator according to an embodiment is shown.

[0056] Figure 8 A process of performing one or more actions based on a detected physical condition of a vehicle driver according to an embodiment is shown. DETAILED DESCRIPTION

[0057] According to one aspect of the embodiment, the performance indicators (e.g., resolution, frame rate, and sensitivity) of a thermal sensor or thermal sensor array can be increased to support applications such as identity authentication, biometric data extraction, and health status analysis. Predictions can be performed by monitoring a time series of thermal images, and thus early warnings of health conditions can be generated.

[0058] According to another aspect of the embodiment, the frame rate of the thermal sensor may be increased to a determined level to capture minute details of thermal radiation from a human body that change over time, such as detailed changes in thermal radiation from a human body.

[0059] According to another aspect of the embodiment, thermal images of blood flow through the skin can be converted into a temporal signal for pulse rate extraction. Further signal processing techniques can be applied to additional biometric data of an individual to analyze health conditions. Image signals can be processed to identify multiple objects from the content and track associated biometric data.

[0060] According to another aspect of the embodiment, the application may determine the position of a human body within the image signal, along with motion tracking from previous images, for fall detection. Motion estimation may be applied to predict if there is any hazard to the individual within the image signal.

[0061] According to another aspect of the embodiment, a profile can be associated with an individual. The device can track and learn the behavior of the individual from the history of the image signal. In addition, when an individual is detected in a scene, the device can adapt. For example, when an individual is detected entering a living room in the summer, the set temperature of the air conditioner in the living room can be adapted to the individual's preferences.

[0062] According to another aspect of the embodiment, the ambient temperature can be controlled based on the individual's body temperature and other parameters (such as relative humidity and outdoor temperature, etc.) to achieve an overall comfort zone through machine learning.

[0063] According to another aspect of the embodiment, the accuracy of the analysis is determined by the resolution, sampling frequency and sensitivity of the thermal sensor, the signal processing techniques to extract the biometric data from the image signal, and the analysis / learning algorithm.

[0064] According to another aspect of the embodiment, the application of the thermal sensor can be extended to home applications.

[0065] According to another aspect of the embodiment, the analysis model consists of a training model. The model is trained from a database of reference thermal image signals and associated target vectors, which can represent a range of settings for smart home devices. Reinforcement learning can be deployed to allow the model to adapt to new target vectors. For example, a user can change the temperature setting of a room between summer and winter.

[0066] According to another aspect of the embodiment, training is not applied to the analytical model, but rather learning is done over time from a sequence of target vectors associated with thermal signatures. For example, when a new thermal signature associated with a new user is detected, the default settings of the smart device are applied. When a user changes the settings of a single device, the new settings are recorded to retrain the model.

[0067] Figure 1 A thermal camera 101 is shown located in a room 100 according to an embodiment. The camera 101 may generate a thermal image (thermogram) 102 of an individual not explicitly shown.

[0068] In some embodiments, thermal camera 101 includes a lens that focuses infrared or far infrared radiation onto visible objects. The focused light is scanned by a thermal sensor that includes multiple infrared detector elements (e.g., 24×32 pixels). The detector elements can produce a very detailed temperature pattern (e.g., thermal spectrogram 102).

[0069] In some embodiments, camera 101 may require one hundredth of a second of the detector array to acquire sensor information to obtain a thermal image.Sensor information may be acquired periodically from thousands of points in the field of view of the thermal sensor to form a sequence of thermal images.

[0070] The thermogram 102 generated by the detector element of the thermal sensor can be converted into electrical pulses. The pulses are then sent to a signal processing unit (e.g., Figure 3a , 3b 3 and 3c), the signal processing unit can be implemented as a circuit board with a dedicated chip that converts the sensor information into biometric data.

[0071] The thermal camera 101 may also include tracking capabilities such that the direction of the camera 101 may be changed to track a moving object such as a person 102 moving in the room 100 .

[0072] Although Figure 1 One thermal sensor is depicted, but some embodiments may interface with multiple thermal sensors. For example, thermal sensor arrays may be located in different rooms and / or entry points of a residence.

[0073] Figure 2 Shown is apparatus 200 interfacing with thermal sensors 204 and / or 205 via sensor interface 206 and with smart devices 202 and / or 203 via smart device interface 209 in accordance with an embodiment.

[0074] Thermal sensors 204 and 205 are typically used for access control and presence detection. In some embodiments, in order for the processor 201 to extract biometric data from the sensor information, the performance of the thermal sensor 204 can be expanded by increasing the sampling frequency (e.g., frame rate) of capturing image signals, identifying and tracking individuals from image signals, and analyzing the detailed changes of thermal images over time. The processor 201 can convert the sensor information (signals) into biometric data, such as heart rate, body position, health status, etc. The device 200 can also support the prediction of future health events and / or support system personalization by processing image signals.

[0075] In some embodiments, the processor 201 can process the sensor information to detect the thermal signature of the user. When the thermal signature of a particular individual is detected, the processor 201 can apply the profile (eg, temperature setting) of the individual to the smart device 202 (eg, air conditioner).

[0076] Processor 201 may support one or more health applications that process and / or analyze biometric data, and may generate notifications about the biometric data to an external entity (e.g., a physician) via interface 210 over communication channel 251. For example, a health application may detect from the biometric data that the user is having a possible heart attack; therefore, an emergency notification about the event is sent to the user's physician.

[0077] refer to Figure 2 , the computing system environment may include one in which the processes discussed herein may be implemented (e.g., Figure 4 The computing device may include a processor 201 for controlling the overall operation of the computing device and its associated components (including RAM, ROM, communication module, and first memory device 207). The computing device typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device, and includes volatile and non-volatile media, removable and non-removable media. As an example and not limitation, computer-readable media may include a combination of computer storage media and communication media.

[0078] Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computing device.

[0079] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0080] In some embodiments, processor 201 may execute computer-executable instructions stored at memory 207 and access archival data stored at memory 208 .

[0081] In some embodiments, memory devices 207 and 208 may be physically implemented within a single memory device.

[0082] Figure 3a An apparatus 300a is shown that processes information from one or more thermal sensors 301 according to an embodiment.

[0083] By using a high-quality thermal sensor 301 (e.g., with a frame rate of at least 100 frames per second, a resolution of at least 24×32 pixels, good sensitivity, and low noise), biometric data 351 can be extracted via an analog front end 302, an analog-to-digital converter (ADC) 303, and a feature extractor 305 via appropriate signal processing techniques. The biometric data 351 may include pulse rate, body temperature, temperature distribution pattern, body contour and posture, etc. By tracking changes in the biometric data, the health status of the individual can be analyzed by the analyzer 306, and warning signals 352 and 353 can be generated by the analyzer 306 and the action generator 307, respectively, by further processing the biometric data 351.

[0084] The application can utilize an installed home thermal camera for fall detection by tracking posture changes. For example, when posture changes from upright to horizontal within a short period of time, a possible fall can be detected and an associated alarm can be generated accordingly. In addition, changes in posture, body temperature, temperature distribution pattern, and heart rate can be tracked to estimate the hazard level and associated actions can be taken 353.

[0085] It is also possible to support hazard prediction from biometric data 351. For example, when a person's body temperature continues to drop and his / her posture is shaky, the likelihood of falling may be higher (as indicated by hazard level 352), and thus an alarm can be generated (triggered) before a fall occurs.

[0086] Block 302 may perform signal amplification and non-recursive bandpass filtering, where an analog signal corresponding to a thermal image is processed for DC offset removal, noise reduction, and frequency limiting before being processed by ADC 303. (In some embodiments, block 303 may include a 16-bit ADC with a sampling frequency (e.g., 200 Hz) set high enough to capture details of temperature variations of the object.)

[0087] In feature extraction block 305, image processing is applied to identify valid objects, track the thermal distribution of each object over time, and extract parameters from the thermal distribution to form a feature vector. Examples of parameters of the feature vector include cycle time, changes in cycle time, certain time constants within each cycle and their changes over time, etc. Analysis model 306 obtains the feature vector and compares it with the trained model. Using a deep learning algorithm, such as a deep neural network, the model is pre-trained with a large number of common feature vectors. Reinforcement learning can be used to allow the model to learn from errors. Hazard levels can be provided for the identified objects. In block 307, an action list can be predefined and can be triggered based on the associated hazard level.

[0088] Figure 3b A device 300b is shown that processes information from one or more thermal sensors 301 according to another embodiment. A hazard factor 322 with an associated confidence level 321 is estimated using a model trained using a deep learning model 308, such as a convolutional neural network with supervised learning. Actions 324 are determined from an action list 323 and may be based on the hazard factor 322 and confidence level 321 provided by the model 308. The model 308 may initially support a hazard level, but then identify different hazards with more empirical data (e.g., abnormal heart rate, decreased body temperature, fall detection, etc.).

[0089] The model 308 in the device 300b may also be trained to predict hazards, rather than estimate hazards, based on training sequences from a relatively early time.

[0090] Figure 3c A device 300c is shown for processing information from one or more thermal sensors 301 according to a third embodiment, wherein an action 332 and an associated confidence level 331 are estimated by a trained model 309. Likewise, the model 309 in the device 300c may also be trained to predict any action desired.

[0091] The image processing techniques that can be used depend on the system complexity, including the number of thermal sensors, the resolution of each thermal sensor, the list of hazards and actions, the system computing power and available memory, etc.

[0092] for Figure 3a , 3b As with the embodiment shown in 3c, the analytical model may be implemented locally or in a cloud server, depending on the criticality of the response time.

[0093] Figure 4 A process 400 of identifying a user from thermal sensor information and applying a corresponding profile is shown according to an embodiment.

[0094] The process (application) 400 supports human presence detection and thermal signature verification at block 401. If a human object is detected at block 401 and the thermal signature matches a known entity, all supported smart devices (e.g., air conditioner, smart TV, or smart lighting) can be adjusted 403 according to the profile database stored at block 402.

[0095] If any adjustments are made to the applied profile 405, adjustment data 453 may be sent to the profile adaptation unit 406, which may include the new settings in the profile. If adjustments are required, the profile database is updated 451 by the profile adaptation unit 406.

[0096] To add a new user, the profile adaptation unit 406 sends the thermal signature of the new user 453 to the user identifier unit 401 and sends the associated profile, which may be a default profile, to the profile database unit 402 .

[0097] The profile adaptation unit 406 may include a deep learning model trained using reinforcement learning.

[0098] Figure 5 A flow chart 500 for sorting through multiple applications executed by a device 200 according to an embodiment is shown. The device 200 can execute one of the multiple applications based on the detected condition. For example, a first health application can monitor general health measurements of the user (e.g., activity level and temperature). If one or more measurements are abnormal, the device 200 can launch a different health application based on the detected condition.

[0099] refer to Figure 5, the device 200 configures the thermal sensors 204 and 205 according to a first set of sensor parameters at box 501 in order to execute a first application at box 502 .

[0100] If an abnormal condition is detected at block 503, the device 200 launches an appropriate application. For example, at blocks 504 to 505 and 506 to 507, the device may switch to a second application to monitor fall prediction or a third application to monitor the user's heart rate, respectively. When executing the second or third application, the device 200 may configure the thermal sensors 204 and 205 differently to obtain different biometric data.

[0101] In another embodiment, different configuration parameters may be applied to each sensor for each application.

[0102] In a third embodiment, different sets of configuration parameters are applied to the sensors one by one to extract all biometric data before running the application.

[0103] In a fourth embodiment, a most comprehensive set of configuration parameters is used for all sensors and applications. All sensors can be set to the best configuration set, such as but not limited to the highest image resolution, number of bits, frame rate, sensitivity, signal-to-noise ratio (SNR), computing power, power consumption, etc.

[0104] Figure 6 A flow chart 600 is shown according to an embodiment, in which the device 200 configures a smart device with one of a plurality of parameter groups based on a detected user. The device 200 may monitor sensor data from thermal sensors 204 and / or 205 to detect thermal signatures of one or more users. For example, thermal sensor 204 may be located at an entry point of a residence. Based on the sensor information obtained from sensor 204, the device 200 may identify users entering and leaving the residence. In some embodiments, the device may detect thermal signatures from the front (corresponding to a person entering the residence) or from the back (corresponding to a person leaving the residence). Based on the detected thermal signatures, the smart device may be configured with different parameter groups (e.g., temperature settings for an air conditioner).

[0105] At block 601, the device 200 is trained to detect thermal signatures of different users from sensor data. For example, the distinguishing signatures may be stored in the memory 208. When thermal signatures of two users are detected at block 602, only user A is detected at block 604, or only user B is detected at block 606, the smart device may be configured according to a first set of smart device parameters at block 603, a second set of smart device parameters at block 605, or a third set of smart device parameters at block 607, respectively. In some embodiments, the first set (when two users are detected) may be a compromise between the second set and the third set (when only one user is detected). Otherwise (when no user is detected), at block 608, the smart device may be configured according to the default settings of the smart device parameters.

[0106] Embodiments may support the following capabilities.

[0107] A device uses a thermal sensor for biometric data extraction and tracking for smart home applications. Applications include health status analysis, motion estimation (e.g., fall estimation), accident prediction (e.g., heart rate slowing to a hazardous level), hazard detection (e.g., lying down for a long time), learning personal profiles, and system adaptation based on personal preferences.

[0108] The parameters of the thermal sensor can be enhanced to allow for the extraction of as much data as possible. Examples include, but are not limited to:

[0109] a. Increase resolution, frame rate, sensitivity and signal-to-noise levels, e.g. for heart rate monitoring.

[0110] b. Increase the resolution, sensitivity, signal-to-noise ratio, etc. used to detect distance.

[0111] c. Increase the resolution of the number of tracked objects.

[0112] Signal processing techniques extract biometric data from thermal images.

[0113] Analytical models for hazard estimation and associated actions subsequently taken.

[0114] Analytical models for motion estimation.

[0115] Analytical models for hazard and / or action prediction.

[0116] Model for learning individual behavior of smart devices based on biometric data extracted from thermal sensors.

[0117] The parameters of the smart device are configured based on the different persons detected.

[0118] Changing from the first health application to the second health application based on the condition detected by the first health application. The configuration parameter sets for the various sensors of the active health application may be the same or different.

[0119] Before running the health application, all biometric data are extracted using different sets of configuration parameters.

[0120] Use a comprehensive set of configuration parameters for all sensors and health applications.

[0121] Thermal sensor data is obtained to detect the thermal signature of the front or back of a person.

[0122] Ability to increase the sampling frequency of thermal sensors (including IP cameras, thermal cameras, and thermal sensors) to capture small changes in color content due to thermal radiation from the human body.

[0123] The resolution and sensitivity of thermal sensors can be increased to span the detection range.

[0124] Model clauses: 1. A device supporting at least one smart device, the device comprising: Smart device interface; a thermal sensor interface configured to obtain sensor information from a first thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: detecting a detected thermal signature of a detected user from the sensor information; When the detected user is a first user, obtaining a first profile corresponding to the first user, wherein the first profile includes a first set of smart device parameters; and When the detected user is the first user, a first smart device is configured based on the first set of smart device parameters through the smart device interface.

[0125] 2. The apparatus of clause 1, wherein the memory stores computer executable instructions that, when executed by the processor, further cause the apparatus to perform: When the detected user is a second user, obtaining a second profile corresponding to the second user, wherein the second profile includes a second set of smart device parameters, and wherein the second set is different from the first set; and When the detected user is the second user, the first smart device is configured based on the second set of smart device parameters through the smart device interface.

[0126] 3. A device supporting at least one smart application, the device comprising: a thermal sensor interface configured to obtain sensor information from a first thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: When executing the first application: configuring the thermal sensor according to a first set of sensor parameters; extracting biometric data from the sensor information when the thermal sensor is configured with the first set of parameters; and When a first condition is detected from the biometric data, launching a second application; and When the second application is executed, configuring the thermal sensor according to a second set of parameters, wherein the first set of parameters and the second set of parameters differ by at least one parameter; and The biometric data is extracted from the sensor information when the thermal sensor is configured with the second set of sensor parameters.

[0127] 4. A device supporting at least one smart application, the device comprising: a thermal sensor interface configured to obtain sensor information from a thermal sensor and configure the thermal sensor according to a most comprehensive set of sensor parameters for all applications; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: extracting biometric data from said sensor information; executing the first application; The second application is executed.

[0128] 5. A device supporting at least one smart application having more than one thermal sensor, the device comprising: a first thermal sensor interface configured to obtain sensor information from a first thermal sensor; a second sensor interface configured to obtain sensor information from a second thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: configuring the first thermal sensor according to the first set of sensor parameters; configuring the second thermal sensor according to the second set of sensor parameters; extracting biometric data from said sensor information from all of said sensors; Execute the first application: The second application is executed.

[0129] In some embodiments, the configuration parameter set of all sensors may be the same, in other words, all sensors may be configured with the most comprehensive parameter set for all applications. The optimal sensor configuration may include, but is not limited to, the highest image resolution, number of bits, frame rate, sensitivity, and signal-to-noise ratio (SNR).

[0130] The following relates to embodiments of continuous physical health monitoring of a vehicle operator.

[0131] Reference again Figure 1 Although embodiments support the use of thermal sensors to assess the health of people in a room, embodiments may utilize thermal sensor data to assess the health of people in other types of confined spaces (such as vehicles). The parameters used may include, but are not limited to, heart rate, breathing rate, body temperature, posture (particularly head position), and the trajectory of these data over time.

[0132] The physical health of the vehicle operator (vehicle driver) is critical to the safety of the operator, passengers, and the vehicle itself. If an emergency situation unexpectedly arises, the state of the vehicle operator's condition can determine the outcome of the situation.

[0133] In conventional methods, there are many ways to monitor the physical health of vehicle operators through wearable devices. However, wearable devices are specific to the individual wearing the device rather than the vehicle, and may not ensure that information or data on the health of the vehicle operator is safely monitored during vehicle use.

[0134] According to one aspect of the embodiment, monitoring of the driver and / or vehicle can be performed in a non-invasive and accurate manner, which is always activated when the vehicle is in operation. Therefore, the health status of anyone driving the vehicle can be assessed. In this way, biometric information about the driver is used for accident prevention, accident alerts, critical health warnings, and post-event analysis.

[0135] Figure 7 A vehicle system 700 for continuously monitoring the physical health of a vehicle operator is shown according to an embodiment.

[0136] refer to Figure 2 As previously discussed, this embodiment obtains thermal sensor data from thermal sensor 204 via thermal sensor interface 206 .

[0137] The thermal sensor 204 is typically mounted in a fixed location in front of the vehicle operator (driver), such as against a top windshield corner in front of the driver.

[0138] Processor 703 refers to Figure 5 Configure the thermal sensor using the method in .

[0139] Processor 703 extracts biometric information contained in sensor data 750. For example, once the driver sits in the driver's seat, processor 703 can continuously monitor the driver's heart rate and head posture. In addition, the driver's health record can be loaded into processor 703 from a remote database server via wireless device 704.

[0140] The processor 703 may decide that additional biometric data is needed based on the driver's health record. For example, if the driver's BMI exceeds a certain value, changes in heart rate, changes in body temperature, and changes in head posture over time may also be monitored.

[0141] As will be discussed in further detail, processor 703 detects one or more current physical conditions regarding the driver and performs one or more actions to address the detected physical conditions.

[0142] The processor 703 can execute an application via a wireless device 704 (eg, a smartphone), initiate a call to 911, generate an email to a designated person, etc., to report the detected physical condition to the driver, doctor, emergency contact, etc.

[0143] The processor 703 can also initiate actions in response to a detected physical condition. For example, if the processor 703 determines that the driver is experiencing a heart attack, the processor can instruct the autonomous driving interface 704 to route the vehicle to the nearest hospital.

[0144] As will be discussed further, the biometric information may be stored in the storage device 706 for subsequent analysis regarding the health of the vehicle driver. Although the storage device 706 is shown as a separate device, the storage device 706 may be integrated within the wireless device 704.

[0145] Figure 8 A process 800 is shown for performing one or more actions based on a detected physical condition of a vehicle driver, according to an embodiment.

[0146] At block 801, processor 703 extracts biometric information contained in sensor data 750. At block 802, processor 703 processes the information conveyed in signal 750 to extract measurements of one or more biometric characteristics of the vehicle driver. Biometric characteristics may include, but are not limited to, heart rate, respiration rate, and deviation from average heart rate (e.g., degree of irregularity of the heartbeat).

[0147] The measurements of the biometric features may be stored in the storage device 706 for use in analyzing the health of the vehicle driver at a later time. For example, the stored data may be evaluated by the driver's doctor to determine whether treatment is needed.

[0148] At block 803, process 800 obtains measurements of biometric characteristics (e.g., heart rate and respiration rate of the vehicle driver) and determines whether a health profile is applicable to the driver. Multiple health profiles may be specified, with a first health profile mapping to normal vital functions of the driver (in other words, no health event detected), a second health profile mapping to a heart attack event, a third health profile mapping to the driver falling asleep, a fourth health profile mapping to excessive drinking, etc.

[0149] If abnormal health is detected based on the determined health profile at block 804, process 800 detects whether a specific health event has occurred at blocks 805 to 809. Based on the specific health event, process 800 performs appropriate actions. Exemplary actions include, but are not limited to:

[0150] Sleep event (block 805 - driver falls asleep): a loud warning sound is emitted via the vehicle radio or wireless device to alert the driver Heart attack event (block 806): Instruct the autonomous driving interface to drive the vehicle to the nearest hospital ● Excessive drinking (block 807): Prevent the vehicle driver from starting the vehicle or safely parking it if the vehicle is moving • Arrhythmia event (block 808 - irregular heartbeat or arrhythmia): generate an alert to the driver via wireless device ● Stroke event (block 809): Instruct the autonomous driving interface to drive the vehicle to the nearest hospital According to one aspect of the embodiment, the processing unit continuously monitors and analyzes the heartbeat of the vehicle driver to generate an alert about any irregularities. The processing unit may use a unique algorithm to provide this capability.

[0151] According to one aspect of the embodiment, the processing unit may identify the detected irregularity to correspond to one of a plurality of events of the vehicle driver, including but not limited to falling asleep, having a heart attack, drinking excessively, and the like.

[0152] According to one aspect of the embodiment, data about the heartbeat of the vehicle driver can be stored in a storage device. The data can be retrieved at a later time to analyze whether an abnormal health event has occurred.

[0153] As those skilled in the art will appreciate, a computer system with an associated computer readable medium containing instructions for controlling the computer system can be used to implement the exemplary embodiments disclosed herein. The computer system can include at least one computer, such as a microprocessor, a digital signal processor, and associated peripheral electronic circuits.

Claims

1. A device supporting at least one smart device, the device comprising: Smart device interface; a thermal sensor interface configured to obtain sensor information from a first thermal sensor; a processor for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processor, cause the device to perform: detecting a detected thermal signature of a detected user from the sensor information; When the detected user is a first user, a first profile corresponding to the first user is obtained, wherein: The first file includes a first set of smart device parameters; as well as When the detected user is the first user, a smart device is configured based on the first set of smart device parameters through the smart device interface.

2. The device according to claim 1, wherein The memory stores computer executable instructions that, when executed by the processor, further cause the device to: When the detected user is a second user, obtaining a second profile corresponding to the second user, wherein the second profile includes a second set of smart device parameters, and wherein the second set is different from the first set; and When the detected user is the second user, the smart device is configured based on the second set of smart device parameters through the smart device interface.

3. The device according to claim 2, wherein: The memory stores computer executable instructions that, when executed by the processor, further cause the device to: training to detect a first thermal signature of the first user and a second thermal signature of the second user; as well as In response to the training, the first user and the second user are distinguished based on the received thermal signature.

4. The device according to claim 2, wherein: The memory stores computer executable instructions that, when executed by the processor, further cause the device to: When both the first user and the second user are detected, the smart device is configured at a compromise setting of the first setting and the second setting of the first user and the second user, respectively.

5. A method for assessing the health status of a vehicle driver, the method comprising: generating, by a thermal sensor, a continuous image signal reflected by a driver of the vehicle; extracting, by a processing unit, a first measurement of a first biometric characteristic of the vehicle driver from the continuous image signal; mapping the first measurement to a first health profile of a plurality of health profiles, wherein the plurality of health profiles include at least two specific diseases of the vehicle driver; as well as Detect whether the first health record indicates a first abnormal health event.

6. The method according to claim 5, further comprising: The heartbeat characteristics of the vehicle driver are monitored.

7. The method according to claim 6, further comprising: detecting irregularities from the heartbeat characteristics of the vehicle driver; and In response to the detecting, an alert is generated regarding the irregularity.

8. The method according to claim 7, further comprising: Distinguish one irregularity from multiple irregularities.

9. The method according to claim 6, further comprising: Data about the heartbeat characteristics is stored.

10. The method according to claim 5, further comprising: When the first abnormal health event is detected, a first action is initiated by the processing unit.

11. The method according to claim 5, further comprising: extracting, by the processing unit, a second measurement of a second biometric characteristic of the vehicle driver; mapping the first measurement and the second measurement to a second health profile among the plurality of health profiles; detecting whether the second health record indicates a second abnormal health event; as well as When the second abnormal health event is detected, a second action is initiated by the processing unit.

12. The method according to claim 5, further comprising: obtaining a thermal sensor signal from the thermal sensor; and Biometric data is extracted from the thermal sensor signal by the processing unit.

13. An apparatus for providing an assessment of a vehicle driver of a vehicle, the apparatus comprising: a thermal sensor configured to receive a continuous sequence of images reflected by a driver of the vehicle; a processing unit for executing computer executable instructions; a memory storing the computer executable instructions which, when executed by the processing unit, cause the apparatus to perform: extracting, by the processing unit, a first measurement of a first biometric characteristic of the vehicle driver from the image signal; mapping the first measurement to a first health profile among a plurality of health profiles; detecting whether the first health record indicates a first abnormal health event; When the first abnormal health event is detected, the processing unit initiates a first action; identifying a driver of the vehicle; downloading health records and thermal signatures of identified vehicle drivers; determining, based on the downloaded health record, whether additional biometric data regarding the driver of the vehicle is required; as well as Responsive to the determination, the additional biometric data is monitored.

14. The apparatus of claim 13, further comprising an autonomous driving interface; and in, The memory stores the computer executable instructions, which when executed by the processing unit further cause the apparatus to perform: The vehicle is instructed to drive to an appropriate destination through the autonomous driving interface.

15. The apparatus according to claim 13, wherein: The memory stores the computer executable instructions, which when executed by the processing unit further cause the apparatus to perform: extracting, by the processing unit, a second measurement of a second biometric characteristic of the vehicle driver; mapping the first measurement and the second measurement to a second health profile among the plurality of health profiles; detecting whether the second health record indicates a second abnormal health event; as well as When the second abnormal health event is detected, a second action is initiated by the processing unit.

16. The apparatus according to claim 13, wherein: The memory stores the computer executable instructions, which when executed by the processing unit further cause the apparatus to perform: identifying a driver of the vehicle; downloading health records and thermal signatures of identified vehicle drivers; determining, based on the downloaded health record, whether additional biometric data regarding the driver of the vehicle is required; as well as Responsive to the determination, the additional biometric data is monitored.

17. A method for assessing the health status of a vehicle driver, the method comprising: generating, by a thermal sensor, a continuous image signal reflected by a driver of the vehicle; extracting, by a processing unit, a first measurement of a first biometric characteristic of the vehicle driver from the continuous image signal; mapping the first measurement to a first health profile among a plurality of health profiles; as well as detecting whether the first health record indicates a first abnormal health event; identifying a driver of the vehicle; downloading the health records of the identified vehicle drivers; determining, based on the downloaded health record, whether additional biometric data regarding the driver of the vehicle is required; as well as Responsive to the determination, the additional biometric data is monitored.

18. The method according to claim 17, wherein: The monitoring includes: downloading a downloaded thermal signature of said identified vehicle operator; obtaining a detected thermal signature of the vehicle driver from the thermal sensor; comparing the downloaded thermal signature to the detected thermal signature; and In response to the comparing, the detected thermal signature is matched to a known entity.

19. The method according to claim 17, wherein: The identification includes: authenticating the vehicle driver prior to downloading the health record of the vehicle driver; When the authentication is successful, downloading of the authenticated vehicle driver's health record is achieved.

20. The method according to claim 17, wherein: The determination includes: selecting one of a plurality of biometric features when a body mass index (BMI) of the vehicle driver exceeds a predetermined amount, wherein the one biometric feature is different from the first biometric feature; and The one biometric feature of the plurality of biometric features is measured.

21. One or more non-transitory computer-readable media storing instructions that, when executed by a computing system comprising at least one processor, a thermal sensor, and a memory, cause the computing system to: obtaining a continuous image signal reflected by a vehicle driver from the thermal sensor; extracting a first measurement of a first biometric characteristic of the vehicle driver from the continuous image signal; mapping the first measurement to a first health profile among a plurality of health profiles; detecting whether the first health record indicates a first abnormal health event; identifying a driver of the vehicle; downloading the health records of the identified vehicle drivers; determining, based on the downloaded health record, whether additional biometric data regarding the driver of the vehicle is required; as well as Responsive to the determination, the additional biometric data is monitored.

Images