Multi sensor array smart plug for health and habit tracking

Abstract

Described herein is a smart home monitoring device and smart home monitoring system. The smart device is a device that plugs into an electrical outlet in the home environment. The smart device includes a sensor array and does not use video and audio to monitor the home environment, thus providing the monitored subject a sense of privacy. The smart device wirelessly communicated with a data server executing a machine learning model to track deviations from normal behavior. This smart home monitoring system can be used to monitor habits and behavioral changes in (elder) people, infants, and pets.

Description

NONPROVISIONAL PATENT APPLICATIONMULTI SENSOR ARRAY SMART PLUG FOR HEALTH AND HABITTRACKINGInventors: ZMRZLI, RobertKNIGGE, BernhardAssignee: Wellnuo LLC14585 Big Basin Way Saratoga, CA 95070Multi Sensor Array Smart Plug for Health and Habit TrackingPRIORITY CLAIM

[0001] This application claims priority under 35 USC 119(e) to U.S. Provisional Patent Application Ser. No. 63 / 512,289, filed July 7th, 2023, all of which is incorporated herein by reference in their entirety.BACKGROUND

[0002] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0003] A large percentage of the elderly population prefer to live at home alone for the peace of mind that it provides since the home environment is one in which the elderly person feels comfortable and safe in. However, family members and loved ones may be concerned about the safety and wellbeing of the elderly person. With no one else in the home, it would be difficult to determine whether the elderly person living alone became injured or if the home environment became an unsafe environment. Thus, there is a need for a non-intrusive solution to monitor the elderly person and his or her surroundings in the home environment without intruding on their privacy.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 illustrates an exemplary schematic of a smart device according to some embodiments.

[0005] Figure 2 illustrates the sensor array within the smart device according to some embodiments.

[0006] Figure 3 illustrates a system diagram according to some embodiments.

[0007] Figure 4 illustrates a flow diagram for a machine learning model according to some embodiments.

[0008] Figure 5 illustrates another flow diagram for the machine learning model according to some embodiments.

[0009] Figure 6 illustrates how sensor data is processed according to some embodiments.

[0010] Figure 7 illustrates the classification of events that a deep neural network (DNN) can accomplish according to some embodiments.

[0011] Figures 8a, 8b, 8c, and 8d illustrate different views of an exemplary smart device according to some embodiments.

[0012] Figure 9 depicts a simplified block diagram of an example computer system 900, which can be used to implement some of the techniques described in the foregoing disclosure.DETAILED DESCRIPTION

[0013] Described herein are methods and apparatuses to non-intrusively monitor one or more persons or animals in a home environment. While the description below will focus on the usage model of one person living alone in the home, it is to be understood by those skilled in the art that the solution can also apply to scenarios where multiple people are living in the home. A smart sensor array plug device (“smart device”) is described which can be used as both a human monitoring device and also as a home environment monitoring device. The smart device can include 5 to 10 different sensors. The sensor array includes as a minimum of the following 5 sensors: 1) Temperature, 2) smell (volatile organic compounds - VOC, Volatile sulfur compounds - VSC), 3) Presence Radar, 4) CO2 gas and 5) light sensor. The preferred embodiment includes the additional 2 sensors: 6) Humidity and 7) pressure (Barometer). Additional sensors yield small incremental benefit but could include 8) (passive) infrared or thermopile, 9) CO gas 10) particle counter (i.e. PM2.5). The sensor array may capture data related to the ambient environment surrounding the smart device. In other embodiments, the smart device may be used in other applications to monitor the ambient environment. For example, the smart device may be used by plant growers to monitor the environment of the greenhouse. Output from the smart device can be used to optimally control the greenhouse environment to create an optimal environment to grow plants. The alerts generated by the smart device that indicate the greenhouse environment is not optimal may be transmitted to a botanist or farmer who in turn may manually or automatically correct the environment.

[0014] The sensor array is connected to a CPU to allow data acquisition and to WIFI and / or Thread connection to wirelessly send sensor data to cloud-based storage system / and or local memory chip. In one embodiment, the smart device with the sensor array may store thevarious sensor traces in the local memory chip and apply a machine learning model or sensor fusion algorithms to create use cases and indicate deviations from normal that would then alert the user and family members or caregivers. Alternatively, data analysis can occur in a data center or data server in the cloud. In yet other embodiments, a first portion of the data analysis may happen on the smart device while a second portion of the data analysis may happen external to the smart device. This is shown in figure 7 where certain input data for the deep neural network running external to the smart device such as sleep quality and moving patterns may be generated by the smart device from analyzing sensor data (sensor data may be generated from the smart device or generated from another smart device in the home) while other input data such as light sensors may be generated by sensors on the smart device and transmitted to deep neural network to be used as input data. The smart device has a series of LED light indicators to show the status of various events and it could also serve as a night light. Each room in the house could have a plug with an integrated sensor array to track door openings and movements.

[0015] The smart device purposefully does not include sensors that are used for audio or video recording to respect and maintain privacy in the home environment. However, since the smart device has Bluetooth connectivity external audio and / or video devices can be optionally connected to the smart device via Bluetooth. The smart device is capable of tracking and monitoring the habits and behavioral changes in people, infants, or pets within the home. The person or animal that we wish to monitor at home herein is called the monitored subject. In some embodiments, the smart device may be combined with a machine learning model so that deviations from normal behaviors can result in an alert to a person monitoring the home environment such as a caregiver or a family member. The machine learning model may be incorporated into the smart device or alternatively running on a server or data center in the cloud. In one example, the sensor array monitors home air quality and movements. In another example, the sensor array can inform the monitored subject or person monitoring of changes in sleep patterns, bathroom activity, kitchen activity. In another example, the sensor array can also monitor for fire and burglary / home intrusion and alert caregivers if there is a problem. In one embodiment, the smart device is powered by a wall outlet, so there are no batteries to worry about. This may be advantageous since people, especially elderly people, sometimes forget to replace batteries when the voltage becomes low, thus resulting in a device that is not in operation if powered by batteries. In another embodiment, the smart device may be powered by both a rechargeable battery and a wall outlet. When there is power available from the wall outlet, the smart device may use thepower to operate the device and to charge the rechargeable battery. When power is not available from the wall outlet such as during a power outage or when transporting the smart device to another location in the home, the smart device may be powered by the rechargeable battery. This may be advantageous so that the smart device is still operational during transport and during power outages, thus providing constant monitoring even during these times. The smart device can be installed in each room and it can also function well even when hidden behind contents in the home such as furniture.

[0016] Figure 1 illustrates an exemplary schematic of a smart device according to some embodiments. As shown, the smart includes housing 100, sensor array 110, LED status indicator light 120, and electric socket 130. The smart device may contain a set of electrical prongs configured to removably connect to a wall outlet or a socket of an extension cord to receive power so therefore the smart device does not operate off of battery power. This is advantageous since elderly people may sometimes forget when it’s time to replace batteries in devices so plugging the smart device into a socket guarantees constant power to the smart device. In some embodiments, the smart device can also be used as a ‘smart plug’ - meaning that a secondary electronic device may be plugged into the smart device via electric socket 130 to be either turned on / off or dimmed if plugged device is light. In one embodiment, the smart device may also measure the power consumption of the secondary device through electric socket 130. While the smart device shown in figure 1 includes electric socket 130, it is to be understood that the smart device can function without the electric socket, so the electric socket is optional.

[0017] LED light 120 of the smart device can be configured to alert the monitored subject, caregiver, or others in the home environment on the air quality or unusual behavior happening near the smart device. Alternatively, LED light 120 can also be configured to alert the monitored subject, caregiver, or others in the home environment that there is a risk score associated with the monitored subject that has been generated and is ready for review. The risk score may be accessed by the monitored subject, caregiver, or other through a website or an application running on a mobile phone, tablet, computer, or other computing device. For example, LED light 120 may be utilized to alert poor air quality. As another example, LED light 120 may be utilized to alert that the stove nearby has been left on, that a nearby window or door has been left open, or that a fall or slip has been detected. The color of this status light can be of any desired color, but the preferred embodiment is green / yellow / red on / off white light as a night light. In some embodiments, alerts represented by the LED light 120 can also be relayed to an application running on a smart phone or smart watch of themonitored subject or persons monitoring the subject. Advantages of these alerts are that they may provide peace of mind to the monitored subject, caregiver, and persons monitoring the subject. For example, a green light status may mean that the smart device has determined that the sensor data collected by the smart device have not detected anything to be concerned about, thus giving the monitored subject or caregiver with visibility to the smart device peace of mind.

[0018] Figure 2 illustrates the sensor array within the smart device according to some embodiments. As shown, the smart device includes CPU 210 and a Wi-Fi / Thread connection 220 which is configured to provide wireless and internet connectivity. In some embodiments, the WIFI / Thread can be a BLE connection. The wireless connectivity may serve to provide sensor data traces captured by the smart device to a local data server or a remote cloud-based storage data center. The Bluetooth (BLE) connectivity allows other health tracker devices to connect to the smart device (i.e. heart rate monitor, blood pressure meters, glucose meter, panic buttons, digital scales, audio or video devices), thus increasing the amount of sensor data available for analysis by the smart device. Today most of these BLE devices connect only to your cell phone and requires various apps to be installed and running. This is tedious and requires the proximity of the phone and that the phone is charged.

[0019] The smart device can include several environmental sensors to detect the ambient environment near the smart device. As shown here, the smart device in figure 2 includes temperature sensor 237, humidity sensor 238, carbon dioxide CO2 gas sensor 230, smell - volatile organic compounds -VOC / VSC 231, barometric pressure sensor 232, thermopile sensor 233, power meter sensor 234, radar presence sensor 235, light sensor photodiode 236, carbon monoxide (CO) sensor 239, infrared IR sensor 240. While all of these sensors are shown here, it is to be understood that not all of these sensors are required in the smart device. In one embodiment, a subset of these sensors that include Radar presence, temperature, humidity, light sensor, CO2, VOC / VSC combo sensor, barometer may be preferred. The VOC sensor may be a variety of VOC sensors, depending on implementation details. In one example, the VOC is a mVOC capable of detecting mold. In another embodiment, the following additional sensors can be added to the 8 sensors above to enhance the sensor robustness of the device: carbon monoxide CO sensor, (passive) infrared, a thermopile, particle counter. The selection of which sensors to include in a particular application may depend on conditions of the home environment that is being monitored.Certainly, having a passive infrared motion (PIR) 240 sensor is cheaper than a radar presence or motion detector 235 but requires the unit to be in the field of view of any events andcannot be obstructed by furniture. PIR also have a reduced sensitivity, field of view and cannot be used to gauge distance. Therefore, if the smart device is likely to be obstructed by furniture in the home environment, then a radar presence of motion detector 235 may be preferred over a passive infrared motion sensor 240. Thermopiles (TP) 233 can be used to gauge if multiple people are in the room and can be somewhat directional. Therefore, a TP sensor may be preferred if the smart sensor can be pointed towards a stove for example, since 80% of all home fires originate from the stove.

[0020] In another embodiment the smart device also has an onboard memory chip 241 to store several days of data in case of network outage or for data analytics performed on the smart device.

[0021] In one embodiment, multiple smart devices may be installed in the home environment and the smart devices may communicate with one another, for example through Wi-Fi, Bluetooth, Thread, Zigbee or LORA mesh. This may be advantageous since multiple smart devices may allow for advanced differential detection capability. For example if a person opens a door, a first barometric pressure sensor on a first smart device plugged into a first room might measure a pressure drop and a second barometric pressure sensor on a second smart device plugged into a neighboring room measures a pressure increase thereby enhancing the detection capability for door or window openings. Furthermore, one can average detected sensor values over multiple rooms (i.e. to detect AC / Heater going on / off or weather patterns).

[0022] Since each smart plus has a CPU processing unit it can also do some data analysis and machine learning on the device itself. This is useful for event classification and to accelerate an alarm / notification to be sent to the caregiver and user. Since each device is connected to each other through Wi-Fi / mesh and / or Bluetooth it leads to an enhanced computing power and improved event classification and a more robust Al smart device.

[0023] All sensor data, including attached Bluetooth device data from health trackers are encrypted and sent to the cloud / enterprise server - where it is stored for long term data analysis. Long term trends that are happening over extended periods of time such as weeks or months can be analyzed and could be used to look at correlation and causation of events. In one embodiment, the smart device utilizes its sensors to detect smells in the home environment. For example, the smart sensor may detect the smell of coffee or fried food and compare that data with the sleep pattern or bathroom activity of the monitored subject the following night. Similarly, CO2 levels or VOC levels may impact sleep or bathroom activityand are logged over years and may indicate behavior changes that could be measured by a recurrent neural network timeseries analysis.

[0024] In one embodiment, the smart device may generate a smell fingerprint for the household, or for each room, or for the ambient environment surrounding the smart device from sensor data received from the VOC sensor. The smell fingerprint can be updated over time and can be based on sensor data collected by the smart device. For example, the smart device may incorporate a VOC sensor that is configured to collect sensor data to detect certain odors and smells. For example, a VOC sensor may be able to detect acidoketosis which is a biomarker for diabetes and or keto diet. VOC can also detect presence of alcohol, gas and / or mold. In one embodiment, the smell fingerprint may detect and differentiate odors in the static ambient environment (e.g., carpets, walls, furniture, paintings, etc.), also known as a reference room smell and odors of the monitored subject, also known as monitored subject smell . The ability of the smart device to generate a smell fingerprint of the monitored subject in the environment may depend on whether the room / environment is vented or unvented. Unvented rooms and environments may collect the odor of the monitored subject over time, thus making it easier to generate a smell fingerprint on the montiored subject. The smart device may be able to detect whether the room is vented or unvented. In other embodiments, the vented / unvented setting may be set by the user of the smart device.

[0025] In one embodiment, the smart device may be configured to generate a reference room smell of the ambient environment or room when the monitored subject is not present and use that reference room smell as a baseline to compare against when the monitored subject is present in the room. The reference room smell, also known as exogenous smell, can be due to odors from the furniture in the environment, the carpets in the environment, the paint in the environment, just to name a few. It may be advantageous to generate a reference room smell to be used as a baseline by smart device because any smells other than the reference room smell can be attributable to changes in the ambient enviroment. For example, the changes in the ambient enviomment may be due to a smell signature that is associated with environmental changes, such as the stove has been left on, there is smoke in the room, there is mold, or the humidity is too high. As another example, the changes in the ambient enviorment may be due odors coming from the monitored subject in the room. The smart device may be configured to analyze the odors coming from the monitored subject and generate a monitored subject smell. The VOC sensor can also differentiate and include smell signatures of deodorants, perfums, soaps, hairgel, beauty products, shower gel, shampoo andconditioner. However, the monitored subject smell may include bio markers for certain diseases such as flu, pneumonia, lung cancer, stroke, or diabetes. The monitored subject smell that includes these bio markers may be combined with other inferences generated from sensor data such as sleeping or moving habits of the monitored subject to generate a risk score for each of these diseases. The risk score can be generated by the smart device or alternatively external to the smart device such as at a data center or other cloud system.

[0026] Commercially available VOC sensors are capable of measuring parts per billion (ppb) of various VOCs. The sensor may receive a gas mixture, preheat the gas mixture, and detect the result of the preheating. The sensor may be able to detect thousands of different odors and smells based on the gas mixture in the air and the exposure time to the sensor. For example, detecting a coffee smell may include preheating the gas mixture for a certain time. In another example, detecting a dirty diaper may include preheating the gas mixture for a different time. As another example, the smart device may be able to detect that a diabetic person is in the ambient environment of the smart device based on the smell fingerprint. In yet other embodiments, the smell fingerprint may be generated external from the smart device. For example, the smell fingerprint can also be generated at a data center or elsewhere in the cloud, on a system executing a machine learning model. Depending on the complexity of the smell fingerprint, generating the smell fingerprint externally may be benefical if additional compute power is required.

[0027] In one embodiment, the smart device may include a mold mVOC sensor. A mVOC sensor is capable of detecting microbial volatile organic compounds (mVOC) that are released during the metabolic processes of decay agents such as fungi, bacteria, and biofilm. The off gassing of mVOCs include a wide range of gasses including ethanol, alcohols, methane gas, ketones, aldehydes, esters, carboxylic acids, lactones, terpenes, aromatic hydrocarbons, sulfurs, and nitrogen compounds. mVOCs produced during primary fungal metabolism include ethanol, l-octen-3-ol, 2-octen-l-ol, and benzyl cyanide. mVOCs produced during secondary metabolism include 2-methyl-isobomeol, geosmin (1-10- dimethyl-trans-9-decalol) and terpenes. These mVOCs may be detectable by a mVOC sensor and used as sensor data in the smart device.

[0028] In one embodiment, a user may specify to the smart device the smell signatures to focus on. If there are predefined smell signatures, the smart device may prioritize detecting the predefined smell signatures. In other embodiments where there are no predefined smell signatures, the smart device may vary the preheat time and attempt to detect a wide range ofsmells. The smart device may take the results of the smell signature and generate inferences such as whether the air is healthy is outside EPA guidelines, whether someone in the room is drinking coffee, whether someone in the room is eating fried foods, whether someone in the room is drinking alcohol, or whether someone is in the room is smoking. In some embodiments may be able to detect diseases and health conditions, such as cancer, COVID, stroke, and diabetes. For normal healthy humans normal endogenous volatile metabolites produces the following ubiquitous VOCs in the breath: isoprene (12-580 ppb), acetone (1.2- 1880 ppb), acetaldehyde (3-7 ppb), ethyl acetate (ND-116 ppb), butanone (6-26 ppb), 1- butene (ND-495 ppb), dimethyl sulfide (ND-46.5 ppb), ethylene (ND-233 ppb), furan (ND- 78.4), hexanal (9-13 ppb) and many more. Exogenous VOC from the surrounding air are mixed in and the machine learning algorithm will determine (based on sleep and room activity) the quality and usability of the smell fingerprint to detect disease.

[0029] In some embodiments, a neural network may analyze the vector array signatures generates by the sensors of the smart device to correlate the sensor data to inferences made from the sensor data. The neural network model may be a Recurrent Neural Network, Deep Neural Network, Convolutional Neural Network, or other neural network model. In one example, the dataset used to train the neural network may come from all households that have the smart device installed. In another example, a subset of all the household data set can be used, wherein the subset is based on region / country / locality or the characteristics of the monitored subjects such as age, gender, health status, or pre-existing conditions. Using a subset specific to the monitored subject may result in a model that is customized for the monitored subject.

[0030] In some embodiments, weather information and global positioning system information can also be inputs to the neural network. The weather information and global positioning information may be useful to group entries in the training dataset so that monitored persons in similar weather conditions are grouped together for training or monitored subjects near one another are grouped together.

[0031] In another embodiment the temperature in the bedroom as measured by the smart device and the relative humidity in the bedroom as measured by the smart device may be correlated with the quality of sleep of the subject. For example, a temperature above 80 degrees Fahrenheit and a relative humidity above 50% may be correlated with poor sleep in the monitored subject. In another embodiment, the relative humidity tracked over time may be able to detect mold. For example, a relative humidity of over 70% for a period of 7 daysmay result in a conclusion by the smart device that there is mold in the ambient environment of the smart device - which can be corroborated by an mVOC sensor.

[0032] There are several use cases for each sensor and combination thereof. The smart device may analyze the sensors independently or in combination and generate predictions, also known as inferences. Each use case can be developed and learned over time in any given environment using a neural network type machine learning model. Alternatively, nearest neighbor (NN) and clustering algorithms are also useful to categorize events. Certainly, simple threshold algorithms can also be applied. For example, if sensors detect that the home temperature is below lOdeg C and or above 30deg C, the smart device can generate an alert which can be transmitted to recipients such as the monitored subject, caregiver, or monitoring party or visually shared through the LED status indicator light. As another example if sensors detect that the relative humidity in the house exceeds a predefined percentage for prolonged period of time, the humidity may create an environment that is prone to the development of mold / fungus. For example, a relative humidity above 70% for a period of 7 days may trigger a risk score of mold in the home. As a result, the smart device may generate an alert which can be transmitted to recipients or visually shared through the LED status indicator light.

[0033] A combinatorial use case example would be if sensors belonging to a smart device plugged into a room detect that CO2 and VOC levels are too high, the smart device could alert the user and caregiver to ventilate the room. Sensors detecting rising CO2 levels could also indicate additional people in the room. A yellow alert can be sent out if CO2 levels exceed 700ppm and a red alert can be sent out if CO2 levels exceed lOOOppm. These thresholds are EPA guidelines but can be adjusted by the user and tailored to the environment. Sensors detecting rising VOC levels could indicate that the fridge was opened or that the stove is on. Or it could indicate spoiled food. It may also indicate pet related odors in certain areas of the house. It may also indicate that the baby diaper is dirty.

[0034] Pressure sensors can be used to estimate door openings, ventilation and A / C units turning on and off. Higher accuracy can be achieved in a differential configuration with various rooms as described above. Pressure sensors can also be used for weather forecasting. In combination with radar presence, temperature, humidity, and light gauges, the smart device can predict whether somebody is entering or leaving a room or opening a window.

[0035] As another example if a smart device is plugged into a bathroom and sensors within the smart device detect high humidity and higher temperature and motion / presence at certaindistance, analysis of the sensor data can predict that the shower is being used. The VOC sensor can also identify if shampoo / shower gel is being used to further enhance event classification. To detect a fall in the shower, the deviation of normal shower time can be taken into account (i.e. typically 1 Imin) then if no motion is detected for prolonged time (i.e. after 20min both inside bathroom and outside bathroom) the analysis can predict that the monitored subject may have slipped and fallen in the bathroom. An alert may be sent to the caregiver / family members in response. In some embodiments, the smart device may go through a decision tree model before sending an alert. For example, if a sensor detects that the monitored person is in the bathroom, the smart device may transition to an event of checking the CO2 levels, light, smell, and humidity. The smart device may then decide whether to generate an alert based on the inferences from this group of sensor data.

[0036] In another embodiment, the smart device may gauge and track habits and report deviations from normal behaviors. For example, let’s assume it usually takes a monitored subject (e.g., elderly person) 3 minutes to get dressed and 30 seconds to walk to the bathroom. As part of aging we observe mental and / or physical decline and if the time it takes to get dressed and walk to the bathroom becomes longer than expected given the subject’s age and health, the smart device may generate an alert to the monitored subject and caregiver to schedule a visit with a doctor. It could also indicate injury.

[0037] In another embodiment, the smart device may generate alerts for intrusion and / or fire. This could particularly be useful in a vacation home or trailer as well as your primary residence. As described above, the pressure from door and window openings can be measured and the LED light on the device can turn red if the smart device determines that the detected sensors data relates to a behavior that is considered ‘not normal.’ For instance, if sensors in the smart device detects certain doors open and motion detected in other areas of the house than the bedroom at 3am at night, then the smart device may determine or predict that an intrusion is taking place. The smart device can in turn generate an alert to the monitored subject, caregiver, or persons monitoring the subject that an intrusion may be taking place.

[0038] In another embodiment, the smart device can be used to track sleep quality. Through the use of sensors on the smart device, the smart device can track how often the monitored subject is getting up at night to use the bathroom. The smart device can also distinguish different sleep phases. For example, the radar presence interface can gauge activity at various distances, which can in turn be analyzed by the smart device to determine whether themonitored subject is tossing and turning in bed or is in a quiet deep sleep phase. Sound sleep is an important indicator of overall wellbeing - and it is related also to mental decline and has shown to correlate with Alzheimer’s. Hence tracking deep sleep times over year may be useful to predict mental decline.

[0039] In another embodiment, the smart device can be used to track infant sleep habits and to monitor breathing and motion issues. Similar techniques described above can be applied to detect infant breathing and motion. Sudden infant death syndrome (SIDS) may be preventable if the caregiver is alerted in a timely fashion.

[0040] Figure 3 illustrates a system diagram according to some embodiments. As shown, smart device 110 is in communication with the data server 310 and Al / machine learning model 300. Phone / tablet or laptop (“PTL”) 320 can communicate with the data server 310 and check raw sensor traces stored in the cloud. PTL can also communicate with the machine learning model 300 to track habits and set up notifications. For example, if the monitored subject goes to bed at 10pm every night and gets up at 7am, the machine learning model 300 can generate a habit for the monitored subject. If that habit is changing over time (i.e. we get up later and later) it may indicate some mental or physical problems and smart device can generate alert that is sent out to the monitored subject, caregivers, or persons monitoring the subject (e.g., family members). Advantages of a such a system as shown in figure 3 include the ability to generate both short term predictions and long-term predictions. The short-term predictions may be generated by the smart device with sensor data measured by the smart device. Long term predictions may be generated by the AI / ML model 300 or data server 310 which can analyze the sensor data over a longer period of time. Longer predictions may include the sleeping habits or deviations of behavior of the monitored subject while short term predictions may include whether the CO2 or VOC levels are too high.

[0041] Figure 4 illustrates a flow diagram for a machine learning model according to some embodiments. Workflow 400 can be programmed in software code and stored in computer readable medium to be read and executed a machine learning model residing on a data server. Alternatively, workflow 400 can also be programmed in software code and stored in computer readable medium to be read and executed by the smart device. Workflow 400 begins by wirelessly collecting and storing sensor data collected from one or more smart devices in the home environment at 401. Workflow 400 then continues by analyzing the data to determine if there are any unusual behaviors at step 402. In one embodiment, the machine learning model may establish baseline norms and compare currently collected sensor datawith prior events. If there is unusual behavior, workflow 400 continues by informing the caregiver or family member at step 403. The alert can be in the form of changing the LED status indicator light on one or more smart devices in the home or transmitting a notification to a device belonging to the caregiver or family member. The machine learning model can be unsupervised or supervised learning. In an unsupervised model, all the data is automatically fed into the computer and ‘normal’ baseline data is created after a few months of monitoring. In a supervised learning model, certain thresholds are already pre-programmed. For example, if temperature and humidity or CO2 levels are out of normal range an alert is sent.

[0042] Figure 5 illustrates another flow diagram for the machine learning model according to some embodiments. Workflow 500 can be programmed in software code and stored in computer readable medium to be read and executed a machine learning model residing on a data server. Workflow 500 begins by wirelessly collecting and storing sensor data collected from one or more smart devices in the home environment at 501. Workflow 500 then continues with data analysis using neural networks or clustering algorithms to extract key features at step 502. The neural network machine learning algorithms may identify events / patterns based on those features at step 503 and then detect anomalies from the recognized patterns at 504. If there is unusual behavior, workflow 500 continues by informing the caregiver or family member at step 505. The alert can be in the form changing the LED status indicator light on one or more smart devices in the home or transmitting a notification to a device belonging to the caregiver or family member. These alerts and use cases and habits can be automatically created by the machine learning model or by the user. Based on long term trends of various behavior patterns, the machine learning model can be unsupervised or supervised learning. In an unsupervised model, all the data is automatically fed into the computer and ‘normal’ baseline data is created after a few months of monitoring. In a supervised learning model, certain thresholds are already pre-programmed. For example, if temperature and humidity or CO2 levels are out of normal range an alert is sent.

[0043] In another embodiment, the smart device can identify early onset of dementia based on activity changes that followed poor sleep patterns. A deep neural network (DNN) can learn and determine the most relevant features from the raw sensor data that are indicative of Alzheimer's and Dementia-related behaviors. For example, it can learn patterns in movement that suggest forgetfulness or detect anomalies in daily routines (i.e. walking to the kitchen repeatedly because of forgetting items). Over time, the DNN can recognize patterns in the sensor data that correlate with certain behaviors associated with cognitive decline, such aschanges in (deep) sleep quality, irregular showering habits, or instances where the stove is left on.

[0044] In one embodiment, the network can be trained to identify deviations from normal behavior patterns. If a person starts taking showers less frequently than usual or leaves the stove on repeatedly, the DNN can flag these as anomalies that may indicate worsening symptoms.

[0045] After notifying the caregiver or family member, workflow 500 continues by updating predictive modeling of heath trajectory at 506. As the DNN is exposed to more data over time, it can start to make predictions about the monitored person’s health trajectory. For instance, if certain behaviors are increasing in frequency or new patterns emerge, the DNN can predict a higher likelihood of an acute event or suggest the need for medical intervention.

[0046] With recurrent neural networks (a type of DNN suited for time-series data), the system can understand the sequence and timing of events, which is crucial for monitoring diseases like Alzheimer's where the progression pattern over time is informative.

[0047] Figure 6 illustrates how sensor data is processed according to some embodiments. On the bottom, level 601 consists of raw sensor data. The raw sensor data can be filtered into filtered data at level 602 where edge computing means Kalman filtering of the raw data can be applied to further filter the data. In one embodiment, edge computing may be performed by the smart device. At level 603, event classification can be applied to the filtered data and the event classification can occur on the smart device or in the cloud. Once the events are classified, the machine learning model searches for anomalies of patterns at 604. The machine learning model may be in the cloud. If any anomalies are found, the caregiver or other person monitoring can be notified. However, a logical hub can be assigned from any of the devices within the household to make a decision to notify the caregiver in a more timely fashion than the cloud for some short-term events. Long term data analytics and seasonal changes occur at level 605 and may require cloud computing.

[0048] Figure 7 illustrates the classification of events that a deep neural network (DNN) can accomplish according to some embodiments. In some embodiment, the DNN 720 can receive one or more inputs from input data 710 to generate one or more output outputs from output data 720. Possible inputs for DNN 720 include sleep quality, moving patterns, VOCs, light sensors, social interaction, bathroom activity, data collected from other devices communicating wirelessly with the smart device, and air quality. These inputs may be generated by one or more smart devices located within the home by analyzing data collectedfrom sensors and data collected from other devices (i.e. BT connected microphone or heart rate monitor). The DNN may receive the input variables and generate the output variables. The output variables can be a risk score for output data 730. For example, an output of the DNN 720 can be a risk score for cognitive decline, a risk score for physical impairments, a risk score for prostate issues, a risk score for heart issues, a risk score for asthma, a risk score for depression, and other risk scores. Although correct health diagnostic should be done by a physician, the DNN can produce a risk score for various diseases based on the home environment and eating / sleep / bathroom patterns which may be useful for the caregiver and family members to keep track of the monitored subject in real time. For example, poor air quality is directly linked with high-risk pregnancies and asthma. If the Air quality index (AQI) is above 150, the smart device or the server may send an alert to the caregiver. Similarly, if the particle counter PM2.5 exceeds 10 - or above 35 pg / m3(in a 24hr window) - an alert may be sent to the caregiver and / or the user and the light on the device may change to red. Frequent bathroom activities are also strongly correlated with urinary tract infections and / or prostate bladder issues. Normal bathroom visits are 0 to 1 time per night. However, as people get older, the frequency of bathroom visits can increase slightly. If the smart device detects that the monitored person uses the restroom more than 3 times per night, the smart device may send an alert to the caregiver and user. However, if the monitored person drank coffee in the evening (which can be detected with VOC smell sensor) the threshold for notification increases - i.e. if ‘normal behavior’ of monitored person is 2 bathroom visits every night and the smart plug detects the monitored person drank coffee this evening and the bathroom visit frequency increases to 3x per night, no alert is sent to the caregiver because drinking coffee in the evening increases the number of bathroom visits. Yet another example for sleep anomaly tracking would be if the total sleep time is less than 6 hours and the total deep sleep time is less than Ihr. Again, typically every person should have about 25% of total sleep time to be deep sleep. If it is less than that, an alert is sent out and if this happens repeatedly, the risk for Alzheimer’s, heart disease, stroke and diabetes go up. After each week of poor deep sleep, the risk score will go up one point. Risk scores can range from 0 (low risk) to 10 (highest risk). Again, risk score depends also on the air quality, food type (VOC can smell fried food), alcohol consumption (VOC may be able to detect this), smoking habits and activity level and may also include data from Bluetooth connected devices, i.e. heart rate (i.e. Fitbit watch), and weight (BT scales). Publicly available information that shows correlation between environmental conditions and health issues can be used as training datafor DNN 720. In some embodiments, the risk score can subsequently be shared with a health care professional, caregiver, or transmitted back to the smart device.

[0049] Figures 8a, 8b, 8c, and 8d illustrate different views of an exemplary smart device according to some embodiments. As shown in the front view of figure 8a, the smart device is circular in shape with a circular electric socket 810. A secondary electronic device may be plugged into electric socket 810. As shown in the side view of figure 8a, the smart device includes a set of electrical prongs 820 to be removably connected to a wall outlet or a socket of an extension cord. The smart device also includes sensor array 830 around the perimeter of the smart device. Sensors of the sensor array may be dispositioned along the outer perimeter of the housing of the smart device. Figures 8c and 8d illustrate a periphery view of the smart device. As shown in figure 8c, smart device also includes vents 840 to cool the smart device.

[0050] Figure 9 depicts a simplified block diagram of an example computer system 900, which can be used to implement some of the techniques described in the foregoing disclosure. As shown in Figure 9, system 900 includes one or more processors 902 that communicate with several devices via one or more bus subsystems 904. These devices may include a storage subsystem 906 (e.g., comprising a memory subsystem 908 and a file storage subsystem 910) and a network interface subsystem 916. Some systems may further include user interface input devices and / or user interface output devices (not shown).

[0051] Bus subsystem 904 can provide a mechanism for letting the various components and subsystems of system 900 communicate with each other as intended. Although bus subsystem 904 is shown schematically as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses.

[0052] Network interface subsystem 916 can serve as an interface for communicating data between system 900 and other computer systems or networks. Embodiments of network interface subsystem 916 can include, e.g., Ethernet, a Wi-Fi and / or cellular adapter, a modem (telephone, satellite, cable, etc.), and / or the like.

[0053] Storage subsystem 906 includes a memory subsystem 908 and a file / disk storage subsystem 910. Subsystems 908 and 910 as well as other memories described herein are examples of non-transitory computer-readable storage media that can store executable program code and / or data that provide the functionality of embodiments of the present disclosure.

[0054] Memory subsystem 908 comprise one or more memories including a main random access memory (RAM) 918 for storage of instructions and data during program execution and a read-only memory (ROM) 920 in which fixed instructions are stored. File storage subsystem 910 can provide persistent (e.g., non-volatile) storage for program and data files, and can include a magnetic or solid-state hard disk drive, an optical drive along with associated removable media (e.g., CD-ROM, DVD, Blu-Ray, etc.), a removable flash memory -based drive or card, and / or other types of storage media known in the art.

[0055] It should be appreciated that system 900 is illustrative and many other configurations having more or fewer components than system 900 are possible.

[0056] The above description illustrates various embodiments of the present invention along with examples of how aspects of the present invention may be implemented. The above examples and embodiments should not be deemed to be the only embodiments and are presented to illustrate the flexibility and advantages of the present invention as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the invention as defined by the claims.FURTHER EXAMPLES

[0057] Each of the following non-limiting features in the following examples may stand on its own or may be combined in various permutations or combinations with one or more of the other features in the examples below. In various embodiments, the present disclosure may be implemented as a processor or method.

[0058] In some embodiments the present disclosure includes a home monitoring device, comprising: a housing having electrical prongs adapted to be plugged into a first electrical outlet in the home; a sensor array housed within the housing, the sensor array including a plurality of sensors configured to generate sensor data related to the ambient environment surrounding the home monitoring device, wherein the plurality of sensors do not include audio or video sensors; a wireless interface housed within the housing configured to communicate with a data server; a LED status indicator light attached to an edge of the housing configured to provide notifications; and a processor housed within the housing and in communication with the sensor array, the wires interface, and the LED status indicator light,the processor configured to execute software containing instructions to: receive the sensor data from the sensor array; transmit the sensor data to the data server via the wireless interface; receive an outcome from the data server via the wireless interface, the outcome based on analysis of the sensor data by the data server; transmitting signals to the LED status indicator light, the signals configured to light up the LED status indicator light a color that corresponds with the outcome.

[0059] In one embodiment, the home monitoring device further comprises a second electrical outlet configured to receive an electronic device, wherein the home monitoring device provides power to the electronic device through the second electrical outlet.

[0060] In one embodiment, the home monitoring device monitors the power consumption of the electronic device.

[0061] In one embodiment, the home monitoring device is powered solely by power received from the first electrical outlet.

[0062] In one embodiment, the plurality of sensors include a temperature sensor, a smell sensor (volatile organic compounds - VOC, Volatile sulfur compounds - VSC), a presence radar sensor, a CO2 gas sensor, and a light sensor.

[0063] In one embodiment, the outcome received from the data server includes a risk score indicating the likelihood of a disease.

[0064] In some embodiments the present disclosure includes a method, comprising: receiving, from at least one sensor in a sensor array, sensor data collected from monitoring an ambient environment of a home monitoring device; storing the sensor data in a local memory, the local memory storing other sensor data; analyzing the sensor data and the other sensor data in local memory; determining that an unusual behavior has occurred in the ambient environment in response to the analysis; and setting the color of an LED status light indicator based on the determined unusual behavior.

[0065] In one embodiment, the method further comprises generating a smell fingerprint from the sensor data.

[0066] In one embodiment, the smell fingerprint includes a reference room smell and a monitored subject smell.

[0067] In one embodiment, the method further comprises transmitting the sensor data and the unusual behavior to a data server.

[0068] In one embodiment, the method further comprises receiving a risk score from the data server, the risk score indicating the likelihood of a disease.

[0069] In one embodiment, the method further comprises transmitting the unusual behavior to a device operated by a caregiver of a monitored subject within the ambient environment.

[0070] In one embodiment, the method further comprises wirelessly receiving data from another health tracker device, wherein the received data is included in the analysis.

[0071] In one embodiment, synergistic analysis is performed on the received data and the sensor data to generate an inference, wherein a notification is transmitted to a caregiver based on the inference.

[0072] In some embodiments, a non-transitory computer-readable medium stores a program executable by one or more processors, the program comprising sets of instructions for receiving, from at least one sensor in a sensor array, sensor data collected from monitoring an ambient environment of a home monitoring device; storing the sensor data in a local memory, the local memory storing other sensor data; analyzing the sensor data and the other sensor data in local memory; determining that an unusual behavior has occurred in the ambient environment in response to the analysis; and setting the color of an LED status light indicator based on the determined unusual behavior.

[0073] In one embodiment, the non-transitory computer-readable medium further comprises generating a smell fingerprint from the sensor data.

[0074] In one embodiment, the smell fingerprint includes a reference room smell and a monitored subject smell.

[0075] In one embodiment, the non-transitory computer-readable medium further comprises transmitting the sensor data and the unusual behavior to a data server.

[0076] In one embodiment, the non-transitory computer-readable medium further comprises receiving a risk score from the data server, the risk score indicating the likelihood of a disease.

[0077] In one embodiment, the non-transitory computer-readable medium further comprises wirelessly receiving data from another health tracker device, wherein the received data is included in the analysis.

Claims

WHAT IS CLAIMED IS:

1. A home monitoring device, comprising: a housing having electrical prongs adapted to be plugged into a first electrical outlet in the home; a sensor array housed within the housing, the sensor array including a plurality of sensors configured to generate sensor data related to the ambient environment surrounding the home monitoring device, wherein the plurality of sensors do not include audio or video sensors; a wireless interface housed within the housing configured to communicate with a data server; a LED status indicator light attached to an edge of the housing configured to provide notifications; and a processor housed within the housing and in communication with the sensor array, the wires interface, and the LED status indicator light, the processor configured to execute software containing instructions to: receive the sensor data from the sensor array; transmit the sensor data to the data server via the wireless interface; receive an outcome from the data server via the wireless interface, the outcome based on analysis of the sensor data by the data server; transmitting signals to the LED status indicator light, the signals configured to light up the LED status indicator light a color that corresponds with the outcome.

2. The home monitoring device of claim 1, further comprising a second electrical outlet configured to receive an electronic device, wherein the home monitoring device provides power to the electronic device through the second electrical outlet.

3. The home monitoring device of claim 2, wherein the home monitoring device monitors the power consumption of the electronic device.

4. The home monitoring device of claim 1, wherein the home monitoring device is powered solely by power received from the first electrical outlet.

5. The home monitoring device of claim 1, wherein the plurality of sensors include a temperature sensor, a smell sensor (volatile organic compounds - VOC, Volatile sulfur compounds - VSC), a presence radar sensor, a CO2 gas sensor, and a light sensor.

6. The home monitoring device of claim 5, wherein the outcome received from the data server includes a risk score indicating the likelihood of a disease.

7. A method comprising: receiving, from at least one sensor in a sensor array, sensor data collected from monitoring an ambient environment of a home monitoring device; storing the sensor data in a local memory, the local memory storing other sensor data; analyzing the sensor data and the other sensor data in local memory; determining that an unusual behavior has occurred in the ambient environment in response to the analysis; and setting the color of an LED status light indicator based on the determined unusual behavior.

8. The method as in claim 7, further comprising generating a smell fingerprint from the sensor data.

9. The method as in claim 8, wherein the smell fingerprint includes a reference room smell and a monitored subject smell.

10. The method as in claim 7, further comprising transmitting the sensor data and the unusual behavior to a data server.

11. The method as in claim 10, further comprising receiving a risk score from the data server, the risk score indicating the likelihood of a disease.

12. The method as in claim 7, further comprising transmitting the unusual behavior to a device operated by a caregiver of a monitored subject within the ambient environment.

13. The method as in claim 7, further comprising wirelessly receiving data from another health tracker device, wherein the received data is included in the analysis.

14. The method as in claim 13, wherein synergistic analysis is performed on the received data and the sensor data to generate an inference, wherein a notification is transmitted to a caregiver based on the inference.

15. A non-transitory computer-readable medium storing a program executable by one or more processors, the program comprising sets of instructions for: receiving, from at least one sensor in a sensor array, sensor data collected from monitoring an ambient environment of a home monitoring device; storing the sensor data in a local memory, the local memory storing other sensor data; analyzing the sensor data and the other sensor data in local memory;determining that an unusual behavior has occurred in the ambient environment in response to the analysis; and setting the color of an LED status light indicator based on the determined unusual behavior.

16. The non-transitory computer-readable medium of claim 15, further comprising generating a smell fingerprint from the sensor data.

17. The non-transitory computer-readable medium of claim 16, wherein the smell fingerprint includes a reference room smell and a monitored subject smell.

18. The non-transitory computer-readable medium of claim 15, further comprising transmitting the sensor data and the unusual behavior to a data server.

19. The non-transitory computer-readable medium of claim 18, further comprising receiving a risk score from the data server, the risk score indicating the likelihood of a disease.

20. The non-transitory computer-readable medium of claim 15, further comprising wirelessly receiving data from another health tracker device, wherein the received data is included in the analysis.

Images