Non-invasive monitoring system

Abstract

Methods, devices, and monitoring systems are described that include a hub in communication with sensors that aggregate activity in an enclosure, such as a building. The hub identifies when those activities occur more frequently or less than expected and classifies the activities to combine together and learn daily activities of the user or individual. Additional satellite devices may be added that increase the range and capacity of the sensors for collecting more information and activities within the building. In this manner, a monitoring system uses a redundant array of sensors to consistently monitor activities occurring in or around a building to create traces and heat maps of events and activities, which are classified by a hub of the monitoring system, validated with other sensors, learned, and then correctly processed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 268,329, filed on February 22, 2022, entitled “NON-INTRUSIVE MONITORING SYSTEM,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to monitoring devices and, more particularly, to non-invasive monitoring systems. Background Art

[0004] Conventional monitoring systems require considerable user intervention and interaction to reliably set up to monitor human activity in a building and provide appropriate responses to the human activity. However, after being set up to monitor human activity, the process of monitoring human activity in conventional monitoring systems is still prone to false positives, and conventional monitoring systems require training and additional information. Therefore, there is a need for improved monitoring systems that can be implemented at a lower cost while providing reliable monitoring capabilities. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An exemplary network environment for implementing an exemplary monitoring system according to an exemplary embodiment of the present disclosure is shown.

[0006] Figure 2 An exemplary hub device of a monitoring system in which some implementations of the present disclosure may be implemented is conceptually illustrated.

[0007] Figure 3 Exemplary satellite devices of a monitoring system in which some implementations of the present disclosure may be implemented are conceptually illustrated.

[0008] Figure 4 An exemplary sensor device of a monitoring system in which some implementations of the present disclosure may be implemented is conceptually illustrated.

[0009] Figure 5 Shows communication with other smart devices or remote computing devices Figure 1-Figure 4 An exemplary embodiment of a monitoring system.

[0010] Figure 6 It is shown Figure 1 A flow chart of an exemplary method of communicating with an exemplary hub within an exemplary network environment.

[0011] Figure 7 is a flow chart illustrating an exemplary method for monitoring a system according to some implementations of the present disclosure.

[0012] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the present disclosure is not intended to be limited to the specific forms disclosed. On the contrary, it is intended to cover all modifications, equivalents, and alternatives that fall within the scope of the present disclosure as defined by the appended claims. DETAILED DESCRIPTION

[0013] In this patent document, the term "exemplary" is used to describe examples or specific embodiments of disclosed devices, assemblies, systems and / or methods. Specific details of several embodiments of the monitoring system and associated systems and methods are described below.

[0014] In the present disclosure, many specific details are discussed to provide a thorough and enabling description of the embodiments of the present disclosure. Those of ordinary skill in the art will recognize that the present disclosure can be practiced without one or more specific details. Known structures and / or operations typically associated with monitoring devices and monitoring systems may not be shown and / or may not be described in detail to avoid obscuring other aspects of the present disclosure. Generally, it should be understood that various other devices, systems and / or methods other than those specific embodiments disclosed herein may be within the scope of the present disclosure.

[0015] The term "monitoring system" may refer to an assembly of one or more electronic devices, smart devices, cameras, sensors, radio transmitters, and receivers. The term "monitoring system" may generally refer to a device coupled to one or more sensors that measure physical quantities and convert the measured quantities into signals. A monitoring system may be a single monitoring device that collects or measures one or more physical quantities. A monitoring system may exclude image sensors to allow non-invasive or non-visual data collection or data collection outside the visible spectrum. A monitoring system may include a low-resolution image sensor or a high-resolution image sensor that is configured to selectively collect visual data in a non-invasive manner. A monitoring device may include, for example, one or more sensors, electronic circuits, and one or more communication modules.

[0016] Traditional home monitoring systems use one or more smart home devices (e.g., cameras, lights, thermostats, locks, etc.) and smart device accessories (e.g., motion sensors, environmental sensors, and trigger sensors) in combination with pre-programmed sounds and detected events to detect and monitor human and pet activities, smoke, fire, air quality and humidity, and listen for specific sounds, such as a baby crying or glass breaking. Although traditional home monitoring systems can be used to monitor all activities in the home and filter specific activities before notifying the user, in order for the home monitoring system to be practical and effective, users are usually required to network multiple cameras with smart home devices and / or sensors to determine what is happening in the home, and then require the user to manually enter the corresponding tasks, collect data, analyze and report the activity data to the user. The process of determining activities and corresponding responses is a complex process and trouble for ordinary users. In addition, user intervention and device training are usually required to prevent false alarms, for example, filtering out break-in or suspicious activity alarms from events such as curtains blowing or pets moving around the house. In order to prevent false alarms, traditional home monitoring systems then need to be trained or set up to monitor specific home settings or floor plans by knowing / monitoring the activities of each family member and pet in the home. For example, monitoring for glass breaking or break-ins requires the home monitoring system to constantly monitor and know (preferably multiple) user locations, phone locations, personal information, and sensed motion information (preferably audio and / or visual recognition) from all potential entry points to ensure the safety of family members at all times.

[0017] Furthermore, more accurate and expensive conventional monitoring systems use complex integrations of equipment and / or expensive hardware that collect, store, retrieve, and compare profile, voice, facial, or image recognition data of users that are vulnerable to security hacking, can result in identity theft, and still require considerable training and setup to work reliably. Thus, properly setting up a monitoring system to account for pets, curtain blowing, and random human activity and behavior can be very expensive, complex, and cumbersome for the average user. Furthermore, in order to be practical and effective in monitoring users and providing peace of mind, monitoring systems need to comprehensively monitor multiple spaces within a building to account for user variability and human activity, which is often invasive or expensive, puts users at risk for security hacking or identity theft, and / or creates privacy issues for users who want peace of mind but do not want their personal information collected, stored, and shared.

[0018] Conventional monitoring systems can also generate measurements, alarms, and more generally, redundant data points that can convey useful information unless they are properly aggregated, analyzed, and / or interpreted. Embodiments of the present disclosure address these and other challenges by providing a monitoring system hub that communicates with simple and redundant sensors that aggregate activities in enclosures such as buildings that are sent to the hub. In some embodiments, the hub then identifies when those activities occur more or less frequently than expected, and classifies the activities to group together and learn the daily activities of users or individuals. Additional satellite devices can be added to the monitoring system hub, which increases the range and capacity of sensors for collecting more information and activities within the building. In this way, the monitoring system of the present disclosure uses a redundant array of sensors to consistently monitor activities occurring in or around a building to create traces and heat maps of events and activities, which the monitoring system hub classifies, verifies with other sensors, learns, and then processes correctly.

[0019] A monitoring system provides simple and redundant sensors to aggregate activities occurring in and around an enclosure such as a building or structure that are used to generate useful and relevant information that can be easily understood by appropriate (multiple) users. For example, in a home care scenario where an elderly person is monitored for multiple activities, the monitoring system can determine when these activities occur more or less frequently than expected, and increase the ease of briefly and concisely communicating with multiple caregivers when these activities occur or do not occur. The monitoring system of the present disclosure includes one or more components, including a hub device, a satellite device, and a sensor device for collecting visual or strictly non-visual information, which strictly non-visual information includes but is not limited to environmental measurements (such as temperature, humidity, airflow, etc.), measurements made by medical instruments (such as blood pressure measurements, body temperature measurements), and other measurements related to the health and mobility of (multiple) building occupants.

[0020] The hub device can collect non-visual information locally in the room where it is installed. To collect information from other rooms with different events or characteristics, the hub device can be paired with one or more satellite devices, which can act as Wi-Fi extenders for one or more hub devices. Satellite devices can transmit information collected from sensor devices installed throughout the building. Satellite devices can also pair with other compatible electronic devices (e.g., bathroom scales or blood pressure monitoring devices) and collect information from other compatible electronic devices. The monitoring system can use "ESPNow", a proprietary Wi-Fi communication protocol from chip manufacturer Espressif. One or more components of the monitoring device may include an activity or event classifier to learn and determine an individual's daily activities (e.g., walking, meal preparation, or toileting), which is used with one or more sensors to confirm what happened and confirm what was learned.

[0021] Figure 1 An exemplary network environment for implementing an exemplary monitoring system according to an exemplary embodiment of the present disclosure is shown. Figure 1 As can be seen, the exemplary network environment 100 for implementing the monitoring system includes various sensors and devices that are communicatively coupled together according to one or more embodiments of the present disclosure. Not all of the components depicted are required. However, one or more implementations may require additional components, fewer components, or different components not shown in the network environment 100. Therefore, any changes in the network environment 100 may be implemented without departing from the scope of the present disclosure.

[0022] The network environment 100 may include one or more networks, such as an IoT network, a WiFi network, a Bluetooth network, a private network, the Internet, any other network, or a combination thereof. The network environment 100 includes one or more satellite devices 160A, 160B, 160C, etc. (hereinafter referred to as 160). One or more of the satellite devices 160 (such as satellite device 160C) may be configured to communicate with one or more satellite devices 160B, one or more sensor devices 170B, and / or one or more hub devices 101 (e.g., via wired or wireless communication). The network environment 100 includes one or more sensor devices 170A, 170B, 170C, 170D, 170E, 170F, 170G, 170H, 170I, etc. (hereinafter referred to as 170). One or more of the sensor devices 170 (such as sensor device 170H) may be configured to communicate with one or more satellite devices 160B and / or one or more hub devices 101 (e.g., via wired or wireless communication). The network environment 100 includes one or more electronic devices 190A, 190B, 190C, 190D, etc. (hereinafter referred to as 190). One or more of the electronic devices 190 (such as the electronic device 190A) can be configured to communicate with one or more satellite devices 160B and / or one or more hub devices 101 (e.g., via wired or wireless communication). The network environment 100 includes one or more hub devices 101 that communicate with one or more satellite devices 160 and one or more electronic devices 190. Each hub device 101 can be communicatively coupled to one or more servers within the cloud infrastructure 135.

[0023] In some embodiments, electronic device 190 may include, may be embedded in, or may be coupled to a portable communication device, such as a mobile phone, a laptop computer, a wearable device, a tablet computer, or any other communication device. Electronic device 190 may be communicatively coupled to one or more of satellite devices 160, and / or one or more other devices coupled to electronic device 190. Figure 1 As shown, examples of electronic device 190 may include a scale, a finger oximeter, a blood pressure monitoring device, a spirometer, a wearable device (e.g., a watch, a band, a belt, etc.), a thermometer, an oxygen machine or mask (e.g., a nasal cannula oxygen machine), a camera, a computer, a desktop, a laptop, a tablet computer, a fax machine, a printer, a light bulb, an appliance, etc.

[0024] In some embodiments, satellite devices 160 facilitate wireless communications between one or more hub devices 101 and one or more electronic devices 190 and sensor devices 170 and one or more servers within cloud infrastructure 135. In some embodiments, satellite devices 160 may be on the same local area network as hub devices 101. In some embodiments, satellite devices 160 may be configured to form a new local area network that is communicatively coupled to hub devices 101. Satellite devices 160 may communicatively couple one or more electronic devices 190 to one or more sensor devices 170 and vice versa. In one or more implementations, one or more of satellite devices 160 may be referred to as an IoT network and / or a machine-to-machine (M2M) network.

[0025] One or more of the sensor devices 170 may be referred to as IoT devices and / or M2M devices, and may include human-machine interface (HMI) applications and machine interface applications. In some embodiments, the sensor device 170 may be implemented as a tag that can be attached to a person or carried by a person, or may be mounted on a wall or another surface. There may be multiple paths between one or more sensor devices 170 and / or one or more satellites 160. One or more of the satellites 160 and / or sensor devices 170 may be configured to communicate with each other or with other systems and with one or more hub devices 101. One or more of the sensor devices 170 may include or may be a sensor that measures a physical quantity from the surrounding environment and converts the physical quantity into a signal. For illustration only and not limitation, examples of sensors include temperature sensors, cameras, audio recorders, motion sensors, humidity sensors, smoke detectors, and other sensors. In some embodiments, the sensor devices 170 may all measure the same physical quantity from the surrounding environment and convert the physical quantity into a signal to provide a redundant / verified physical quantity for measurement, for example, the sensor devices 170 may all measure motion and / or sound to confirm motion or sound occurring in the space.

[0026] One or more of the electronic devices 190 may be referred to as IoT devices and / or M2M devices, and may include human-machine interface (HMI) applications and machine interface applications. Multiple paths may exist between one or more of the electronic devices 190 and / or one or more of the satellites 160. One or more of the satellites 160 and / or the electronic devices 190 may be configured to communicate with each other or with other systems and with one or more hub devices 101. One or more of the electronic devices 190 may include or may be sensors that measure physical quantities from the surrounding environment and convert the physical quantities into signals. By way of illustration only and not limitation, examples of sensors include temperature sensors, cameras, audio recorders, motion sensors, humidity sensors, smoke detectors, and other sensors. In some embodiments, the electronic devices 190 may all measure similar physical quantities from the surrounding environment and convert the physical quantities into signals to provide redundant / verified measured physical quantities, for example, the electronic devices 190 may all measure motion and / or sound to confirm motion or sound occurring in the space.

[0027] In one or more embodiments, the electronic device 190 may include one or more of an active device, a passive device, and / or be implemented in whole or in part as a system-on-chip device. The electronic device 190 may include a transmitter, a receiver, a global positioning system (GPS), a Bluetooth (BT) / BLE transceiver, and / or a WiFi TM Similarly, sensor device 170 may include one or more of an active device, a passive device, and / or be implemented in whole or in part as a system-on-chip device. Sensor device 170 may include a transmitter, a receiver, a global positioning system (GPS), a Bluetooth (BT) / BLE transceiver, and / or a WiFi TM In one or more embodiments, satellite device 160 may include and provide one or more network access points, such as wireless access points (WAPs), that communicatively couple electronic devices 190 and / or sensor devices to one or more hub devices 101 .

[0028] Various communication protocols A, B, C, and D may be used for device provisioning and / or communication within the network environment 100. Provisioning between the hub device 101, satellite devices 160, and sensor devices 170 may use proprietary Wi-Fi communication “A” and out-of-band signaling. Communication “A” may include a proprietary Wi-Fi communication protocol “ESPNow” from the Espressif chip manufacturer. The ESPNow protocol may be used to directly broadcast new devices (e.g., satellite devices 160 or sensor devices 170), providing information directly to the hub device 101. The hub device 101 may then build a whitelist of known sensors and devices in and around the building, and request a wireless user to manually add a device or sensor that is not on the whitelist through a human-machine interface (HMI) (e.g., App), and the MAC address of the gateway used to pair the device may then be sent directly to the new device or sensor.

[0029] Communication "B" (e.g., Bluetooth (BT) / Bluetooth Low Energy (BLE)) can be used as a communication protocol between the electronic device 190 and the satellite device 160. Communication "C" (e.g., IEEE 802 standard) can be used as a communication protocol between the hub device 101 and the cloud infrastructure 135, a communication protocol between the local antenna 131 and the cloud infrastructure 135, and a communication protocol between the IoT / camera device 190D and the cloud infrastructure 135. Communication "D" (e.g., broadband cellular network, such as 2G / 3G / 4G / 5G, or wireless broadband communication, such as Long Term Evolution (LTE)) can be used as a communication protocol between the pendant 180 and the cloud infrastructure 135.

[0030] All detected or user-entered environmental information (e.g., temperature, humidity, sound, pressure, airflow, air quality, pets, door and window positions, ambient light, outside weather, etc.) and individual information (age, posture, mobility, gait, height, weight, vital signs, medication, physical address, medical condition, physical activity requirements, etc.) can be classified in the hub device 101. Aggregated environmental and / or individual information (hereinafter referred to as "activity information") is collected by the hub device 101 from sensor devices 170 and electronic devices 190, either directly from the hub device 101 or through one or more satellite devices 160. The hub device 101 then uses the collected activity information to build a graph of what happened, where it happened, and what sensors were involved. The hub device 101 then classifies each activity information from the various sensor devices 170 and electronic devices 190 to determine the occurrence or non-occurrence of an event. For example, a caregiver may want to monitor and ensure that an individual with a medical condition (e.g., cirrhosis of the liver) has adequate physical activity and urination / defecation movements. The hub device 101 can receive individual information (e.g., individual medical conditions, height, weight, vital signs, etc.) and requests to monitor certain activities of individuals to manage or improve medical conditions from a caregiver / user input / remote computing device. The caregiver can place the electronic device 190 and / or the sensor device 170 near the hub device 101 to monitor the progress of the individual. As an example, the individual can leave the bed to go to the bathroom, and when doing so, the sensor device 170 and the electronic device 190 can detect the movement and trajectory of the individual entering the bathroom, and then only detect the sound of the faucet that has occurred when the individual visits the bathroom. The hub device 101 collects activity information from the sensor device 170 and the electronic device 190, and multiple classifications can be performed, the first classification includes physical activities (e.g., walking events), and then turning the faucet handle (e.g., arm movement / motion). However, additional activity information (subsequent activities) from the sensor device in or near the bathroom confirms that the shower or toilet is not used. Therefore, a second classification is performed, that is, the defecation movement does not occur. The hub device 101 may continue to monitor the second classification of the individual, and if the classification does not occur, the hub device 101 may contact / notify the caregiver and / or remind the individual to take medication to stimulate a bowel movement, perform an additional walking event, or eat or drink to aid digestion, or any combination thereof. The hub device 101 may report activity to the caregiver in a text format as a daily summary or log of individual events that did and did not occur.

[0031] One or more sensor devices 170 and electronic devices 190 may also collect additional information, such as individuals entering or leaving a room, standing, sitting, falling, various sound effects, sounds or movements of pets or animals. The sensor devices 170 and electronic devices 190 may be stored locally in a database, or access remotely stored acoustic and / or video signatures for indicating one or more events / activities that occurred. In some embodiments, the hub device 101 may be stored locally in a database or access remotely stored acoustic and / or video signatures, and compare / classify the activity with the stored acoustic and / or video signatures. Artificial intelligence (AI) or machine learning (ML) may be used with time-of-flight (ToF) sensors, acoustic and / or video signatures to determine when an individual is feeling unwell or in trouble. For example, a time-of-flight (ToF) sensor and sound and / or video signature paired with AI / MI may be used to determine whether the activity information collected from various sensor devices 170 and electronic devices 190 indicates that the individual is prone to falling. In this case, the monitoring device may be used to provide fall detection to a caregiver based on, for example, posture or movement information of an individual entering / leaving / toward a chair or bed. The caregiver can add additional sensor devices 170 and electronic devices 190 to monitor activities in other rooms or the vital signs of individuals. The hub device 101 can then use the activity information collected from the individual sensors and devices to build the individual's habits, preferences, and progress in treating or managing medical conditions. The hub device 101 can be configured to collect only non-visual information from the activity information to protect the privacy of the individual. The collected activity information can then be sent to the caregiver via text message.

[0032] Figure 2 An exemplary hub device of a monitoring system that can implement some implementations of the present disclosure is conceptually illustrated. Figure 2As can be seen, the exemplary hub device 201 may include one or more of the following: a processor 202, a memory 203 having instructions 204 and / or data 205, a network module 211, a network interface 213, a wireless controller 215, an additional / external storage device 220, an antenna 230, a power supply 240, a power regulator / converter 241, a speaker 251, a microphone 252 (e.g., a PDM microphone), a codec (CODEC) 253, and a controller 254. The hub device 201 may also include one or more sensor components 255, which include at least one environmental sensor 256, a motion sensor 257, and an image sensor 258 (e.g., a CMOS sensor), other sensors 259 (e.g., a passive infrared (PIR) sensor, a white light PIR sensor, etc.) and at least one light source 259a (e.g., an LED, an LED array, a night light, etc.) and one or more press features 259b. In some embodiments, one or more sensor components 255 may not be implemented as part of the hub device 201, but rather as separate components or sensors that can communicate with the hub device 201. The hub device 201 can be any electronic device that transmits signals over a network, such as an electronic device embedded in smart appliances and other smart systems. The hub device 201 may include various types of computer-readable media (e.g., non-transitory computer-readable media) and interfaces for various other types of computer-readable media. The satellite device 360 ​​and the sensor device 470 may include one, none, some, or all of the components of the hub device 201 as described below and in the present disclosure.

[0033] The hub device 201 may use a custom, public or private graphical user interface for IP voice and telephony (e.g., FreePBX) or Session Initiation Protocol (SIP) with a dual tone multi-frequency (DTMF) attachment to facilitate virtual conference calls. The hub device 201 may include a custom, public or private DSP to facilitate duplex audio. The hub device 201 may include a message queue Telemetry Transport Technology Committee (MQTT) for device-to-device messaging.

[0034] The processor 202 may retrieve and execute instructions 204 and / or data 205 from the memory / storage device 203 to perform the processes of the present disclosure. In different implementations, the processor 202 may be a single processor, a multi-core processor, or multiple processors. Figures 2 to 6, instructions and data for operating the hub device 201 may be stored on, sent from, or received by any computer-readable storage medium (e.g., memory 203, 403 or attached / external storage 320, 420 of a satellite device 360 ​​or sensor device 470, or memory / storage 512 of one or more servers 511) that stores data (e.g., data 205) accessible to a processor (e.g., a processor of a server 511) during an operating mode of the hub device 201. The hub device 201 may access and execute instructions and / or data stored on any remote computing device 531 (e.g., a mobile device 532, a laptop computer 533, or a tablet computer 534). The data 205 may be, for example, Figure 6 The method instructions depicted in . The method instructions may be executed by the processor 202. In some embodiments, one or more other hub devices 501, satellite devices 560, sensor devices 570, servers 511, electronic devices 590, remote computing devices 531, or any combination thereof may collect and measure physical quantities and / or provide instructions for configuring and operating the hub device 201 and / or communicating between a user and other remote, local, and / or wireless electronic devices.

[0035] The memory / storage 203 may include dynamic random access memory (DRAM) and / or read-only memory (ROM). The memory / storage 203 may provide a temporary location to store data 205 and instructions 204 retrieved and processed by the processor 202. The memory / storage 203 may include non-volatile read-write memory that stores data 205 and instructions 204 even when Wi-Fi / Internet is turned off, which data and instructions can be retrieved and processed by the processor 202. For example, the memory / storage 203 may include magnetic, solid-state and / or optical media, and the memory / storage 203 may be a single or multiple memory units as desired. The memory / storage 203 stores all collected visual, audio, text, voice, motion, heat, proximity, etc. information provided directly from one or more satellite devices 560, sensor devices 570, electronic devices 590, or servers 511, or all collected visual, audio, text, voice, motion, heat, proximity, etc. information provided indirectly to another one or more electronic devices, sensor devices (e.g., wearable devices) through a wireless connection.

[0036] Instructions may be stored in the memory 203 and / or the additional / external storage device 220 for machine learning, including pose estimation for identifying a person or pet and their body position (e.g., sitting or lying down). The memory 203 and / or the storage device 220 may also include instructions for time-of-flight measurement, such as determining pose estimation for identifying a person or pet and their body position by measuring the trajectory and duration of an object occluding the image sensor 258.

[0037] The hub device 201 is coupled to a network via a network interface 213. In some embodiments, the network interface 213 is a machine interface. In this manner, the hub device 201 may be part of a computer network, a local area network (LAN), a wide area network (WAN), or an intranet or a network of networks (e.g., the Internet). A wireless controller 215 may be coupled to the processor 202. The wireless controller 215 may be further coupled to an antenna 230. The network module 211 may be integrated as a system-in-package or system-on-chip device and / or collectively defined as having a network interface 213 and a wireless controller 215. The network interface 213 and the wireless controller 215 are integrated into the network module 211 and coupled to the antenna 230. Any or all components of the hub device 201 may be used in conjunction with the present disclosure. The network interface 213 may include a cellular interface, a WiFi TM interface, infrared interface, RFID interface, ZigBee interface, Bluetooth interface, Ethernet interface, coaxial interface, optical interface or any communication interface that can generally be used for device communication.

[0038] The hub device 201, the satellite device 360 ​​and / or the sensor device 470 may use narrowband IoT (NB-IoT), mobile IoT (MIoT), 3rd Generation Partnership Project (3GPP), enhanced machine type communication (eMTC), extended coverage GSM Internet of Things (EC-GSM-IoT) or other similar low power wide area network (LPWAN) radio technologies to enable connection of a wide variety of devices and services using cellular telecommunication bands.

[0039] The hub device 201 is powered by a power source 240. The power source 240 may include disposable and / or rechargeable batteries (e.g., a 2800 mAh rechargeable lithium polymer battery), existing electrical wiring, a power adapter, or any combination thereof. A disposable battery or a rechargeable battery, such as a nickel cadmium (NiCd), lithium (Li), AA, AAA, and / or a rechargeable capacitor, such as a supercapacitor (SC) or a supercapacitor. The power source 240 may power the hub device 201, for example, via a power adapter for connection to an outlet, a solar panel / battery, or any other renewable / alternative power source. The hub device 201 may use a variety of battery types, multiple power sources, and the like, for example, using a button cell battery to operate some sensor components 255 or to provide auxiliary power to power and operate one or more emergency electronic devices 590 during a brown out, black out, or other power outage. In addition, the converter / regulator 241; a transformer or voltage regulator, AC to DC or DC to DC power converter or frequency converter can be used alone (electrically coupled to the hub device 201) or integrated within the hub device 201 to provide sufficient input power (e.g., 12 VDC) to the hub device 201.

[0040] The hub device 201 may include a speaker 251 and a microphone 252 for communicating with an individual or receiving control commands from an individual located near the hub device 201. The speaker 251 and the microphone 252 may be coupled to a CODEC 253. The coder / decoder (CODEC) 253 may also be coupled to the processor 202 via a controller 254. The processor 202 may provide audio information captured from the microphone 252 to any electronic device (e.g., the server 511 or the wireless user device 532), which may facilitate communication with an individual located near the hub device 201 via the speaker 251.

[0041] The hub device 201 can provide external audio feedback, such as playing a greeting, an audio message, or a doorbell ring through the speaker 251. In addition, the hub device 201 can provide internal audio feedback, such as a ringing number or a mechanical chime or a greeting or message. The hub device 201 can communicate with one or more local electronic devices 590, remote computing devices 531, and servers 511 to provide remote audio and / or visual feedback to one or more users.

[0042] The hub device 201 may include a controller 254 for controlling sensors and processing data collected by sensors, satellite devices 360, sensor devices 470, electronic devices 190, or remote computing devices 531. The controller 254 may include a processor, a memory / storage device (storing sensor instructions, settings, etc.), and a network module wireless chip for communicating with the hub device 201. The controller 254 may send measured / detected environmental conditions and characteristics and individual activities to the processor 202 for further processing. In some exemplary embodiments, the hub device 201 may not include a controller 254 and function as a sensor-only device or satellite device 360 ​​that transmits environmental and individual activities collected in and around a building to another hub device 201 for processing.

[0043] In some exemplary embodiments, the hub device 201 includes a controller 254 to share or divide processing tasks or priorities of data, video, audio, or environmental sensor data with other hub devices 201. For example, the controller 254 can process certain motions (e.g., individuals, homeowners, pets or animals, etc.) or sounds (e.g., a window or door closing or opening, a window breaking) and sound an alarm, request verbal input from a user, or trigger an action instead of sending to another hub device 201 for further processing (or trigger an action before sending to another hub device 201). Similarly, the hub device 201 can process individual and environmental activities before sending to the server 511 for further processing if necessary.

[0044] The hub device 201 may include and control various sensor components 255 for sensing environmental activities near the building (e.g., temperature, sound, motion, and location of individuals, and their respective changes over time). The sensor components 255 may monitor environmental conditions (e.g., humidity, temperature, pressure, etc.) by using one or more environmental sensors 256, and monitor individual activities by using one or more motion sensors 257, other sensors 259, and image sensors 258 and microphones 252. Thus, a combination of sensor components 255 may be implemented to provide comprehensive monitoring or improved accuracy in monitoring individual and environmental activities.

[0045] The hub device 201 can store information collected from the sensor assembly 255, the speaker 251, the microphone 252, the satellite device 360, the sensor device 470, the electronic device 190, the remote computing device 431, and the server 511 in a database. The database can be stored on the additional / external storage device 220 of the hub device 201, the memory 203, the storage device 512 of the server 511, or on an application stored on the remote computing device 531. The spatial and individual information (e.g., individual activity information) in the database is updated and / or confirmed with the individual and spatial information obtained by the collected information of the surrounding environment. The user or individual can be prompted to update or approve the update of the database with the additional spatial and individual information obtained by one or more sensors. The user or individual can also store user preferences and / or classifiers in the database, which can include specific instructions or actions based on the collected spatial or individual information, schedule, time of day, temperature, humidity, etc.

[0046] Spatial and individual information obtained by the hub device 201 from various sources (such as but not limited to sensor components 255, satellite devices 360, sensor devices 470, electronic devices 190, or remote computing devices 531) is compared to classifiers and / or user activities stored in a database, and the hub device 201 can then use the classifiers and user preferences to determine whether to notify other users, call emergency services, or connect, power, or operate various electronic devices, such as controlling existing light switches, ceiling fan controls, ceiling light fixtures, light fixture controls, dimmers, sound or motion sensor units, and traditional light switch sockets, IoT devices, smart home devices, thermostats, cameras, speakers, intercoms, virtual assistants (e.g., voice-operable AI devices), etc.

[0047] As previously described, one or more of the sensor components 255 can be implemented as components separate from the hub device 201. For example, the environmental sensor 256 can detect and collect information about environmental conditions in and around one or more buildings. The environmental sensor 256 can include, for example, a temperature sensor, an ambient light sensor, a humidity sensor, a barometric pressure sensor, an air quality sensor (e.g., for detecting allergens, gases, pollution, pollen, etc.), an infrared sensor, a CO2 sensor, a CO sensor, a piezoelectric sensor, an airflow or airspeed sensor, etc. The environmental conditions collected by the environmental sensor 256 can be used by the processor 202 of the hub device 201 to determine whether to notify the user (e.g., via the wireless user device 532). The environmental sensor 256 can include, for example, a motion sensor, an image sensor, a camera, and other sensors (e.g., a proximity sensor, an occupancy sensor, an ambient light sensor). The microphone 252 can also be used to detect features or verify the opening or closing of an entry door, movement, body position, or the presence of an individual or any type of environmental activity in and around a building.

[0048] In an exemplary embodiment, the hub device 201 includes one or more motion sensors 257 for detecting motion information. For example, the motion sensor 257 can detect moving objects and / or pedestrians. In some exemplary embodiments, one or more sensors (e.g., motion sensor 257, image sensor 258, etc.) can be positioned along one or more edges of the hub device 201. The motion sensor 257 can be a passive infrared motion detector. Infrared motion sensors are also referred to as PIR (passive infrared) motion sensors or simply PIR sensors. This detector has an arc of approximately 120° and a detection area of ​​approximately 50 feet. In situations where an increased motion detection field of view or more accurate motion detection is required, two or more motion detectors can be used.

[0049] Suitable alternative motion detectors may also be used, such as ultrasonic, optical, microwave, or video motion detectors. Still other alternative types of motion detectors may also be used to sense intrusion, including laser scanning or frequency sensitive detectors, commonly referred to as "glass break detectors." The motion sensor 257 may include an image sensor having any type of low light level imaging sensor for monitoring during daytime and unattended to complete darkness, such as a low light complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor.

[0050] The motion sensor 257 may also be supplemented with other devices to help detect motion, such as, for example, photocell sensors, cadmium sulfide (CdS) cells, light dependent resistors (LDRs), and photoresistors. In addition to motion sensors, photocell sensors may be used to determine if there is something blocking the light in front of a sensor or a series of sensors. The sensitivity of the motion sensor and photocell may be adjusted, for example, by an application on an electronic device (e.g., smart device 534 or laptop computer 531). In addition, a server or application may decide whether a situation or application warrants motion detection for nighttime use or twenty-four hour operation through an alternative device such as a photocell sensor. If nighttime operation is selected, the server or application will process the detected photocell information to determine if motion is detected.

[0051] The hub device 201 may include an image sensor 258 for capturing visual information, such as video and still images of the surrounding environment. The image sensor 258 may be coupled to the controller 254 for controlling the camera to capture visual information that may be sent to the processor 202. The controller 254 may be coupled to the processor 202 for processing the visual information. The processor 202 may provide the visual information captured from the image sensor 258 to any electronic device (e.g., a server 511 or a remote computing device 531), which may facilitate interaction or communication with an individual or target located near the hub device 201. The image sensor 258 may be any optical instrument for recording or capturing images, which may be stored locally, transmitted to another location, or both. The image may be a still photo or a sequence of images forming a video or movie. The image sensor 258 may be any type of camera, for example, a high-end professional camera type, a digital camera, a panoramic camera, a fisheye lens type camera, a multi-lens type camera, a VR camera, etc.

[0052] The hub device 201 may include or communicate with any number of other or additional detectors or sensors, such as other sensors 259. By way of illustration only and not limitation, examples of other sensors 259 that may be used include time of flight (ToF) sensors, temperature sensors, cameras, audio recorders, motion sensors, ambient light sensors, light sensors, humidity sensors, smoke detectors, and other sensors such as, for example, electric field proximity sensing (EFPS) sensors to determine whether a person or object behind a wall is nearby.

[0053] The hub device 201 may include a light emitting source 259a, which may include an LED array, such as an LED ring. The LED array may include a plurality of RGB lights that respond to input or feedback received from the sensor assembly 255, touch input from a user, or remote or wireless input from a human-machine interface (HMI), such as instructions from one or more remote computers (e.g., one or more servers, mobile devices, etc.), or any combination thereof. In some embodiments, the light emitting source 259a may be wirelessly operated or activated by, for example, another hub device 301, another visitor or secondary user via an HMI, one or more satellites 160, a tag 170, a smart home device, an electronic device, or any combination thereof. The light emitting source 259a may include one or more multi-color LEDs, daylight LEDs, soft light LEDs, animated multi-color LED arrays, or any combination thereof to aid in peaceful sleep. The light emitting source 259a may be embedded within an outer surface of the hub device 301, or formed along one or more outer surfaces or curves of the hub device 301. The light source 259a may include one or more light sources (e.g., LED) arrays to display animations, cartoons, visual emoticons, images, text, brands. In some embodiments, the hub device 301 may allow the user to customize and / or include preset pre-programmed settings for the light source 259a. For example, the light source 259a may be configured by the hub device 301 or the user to display visual feedback (e.g., themes, colors, images, animations, videos, text, or information) for a predetermined event, reminder, appointment, task, or seasonal, cultural, historical, economic, socioeconomic, geographic event, or to allow individuals to celebrate or commemorate events or traditions of cultural or religious significance. In addition, the user may train the hub device 301 in the following ways: communicating through one or more remote computing devices (e.g., one or more servers, mobile devices, etc.), or communicating through audio (e.g., voice commands) or motion (e.g., gestures, body positions, or body movements) to configure certain themes, colors, and / or interactions with certain times, dates, reminders, etc. The light source 259a may also include a display, such as, but not limited to, a resistive touch display or a capacitive touch display, a projector display, or other touch or pressure sensitive surface for receiving user input, etc. In some exemplary embodiments, other forms of interaction with the hub device 201 may be user input commands through the microphone 252, the wireless user device 532, one or more electronic devices 590, the remote computing device 531, the server 511, or any combination thereof.

[0054] The push feature 259b can be a toggle mechanical button or switch and / or a touch-sensitive button or touchpad. In some embodiments, the push feature 259b can include a light source to communicate with the user by, for example, providing visual feedback of a button press, visual feedback for a scheduled event, reminder, appointment, task, or visual feedback for other programming or scheduled events or information. The visual feedback can be the same, different, preset by the hub device 301, or configured by the user. The user can confirm the visual feedback by pressing the push feature 259b, communicating through one or more remote computing devices (e.g., one or more servers, mobile devices, etc.), or communicating through audio (e.g., voice commands) or motion (e.g., gestures, body positions, or body movements) to confirm the feedback.

[0055] Figure 3 An exemplary satellite device of a monitoring system that can implement some embodiments of the present disclosure is conceptually illustrated. Figure 3 It can be seen that the exemplary satellite device 360 ​​may include one or more of the following: a processor 302, a memory 303 with instructions 304 and / or data 305, a network module 311, a network interface 313, a wireless controller 315, a storage device 320, an antenna 330, a power supply 340, a power regulator / converter 341, and a controller 354. The satellite device 360 ​​may also include one or more sensor components 355, which include at least one motion sensor 357, at least one light source 359a (e.g., an LED, an LED array, a night light, etc.), and one or more press features 359b. The one or more sensor components 355 may be implemented as components separate from the satellite device 360. The satellite device 360 ​​may be any electronic device that transmits signals over a network, such as an electronic device embedded in smart appliances and other smart systems. The satellite device 360 ​​may include various types of computer-readable media (e.g., non-transitory computer-readable media) and interfaces for various other types of computer-readable media. Satellite device 360 ​​may include one, none, some, or all of the components of hub device 201 as described above and in this disclosure.

[0056] Processor 302 may retrieve and execute instructions 304 and / or data 305 from memory / storage 303 to perform the processes of the present disclosure. In different implementations, processor 302 may be a single processor, a multi-core processor, or multiple processors. Figure 2-6, instructions and data for operating the satellite device 360 ​​may be stored on, sent from, or received by any computer-readable storage medium (e.g., memory 203, 403 or attached / external storage 220, 420 of the hub device 201 or sensor device 470, or memory / storage 512 of one or more servers 511) that stores data (e.g., data 305) accessible to a processor (e.g., a processor of a server 511) during an operating mode of the satellite device 360. The satellite device 360 ​​may access and execute instructions and / or data stored on any remote computing device 531 (e.g., a mobile device 532, a laptop computer 533, or a tablet computer 534). The data 305 may be method instructions for configuring the network settings of the satellite device 360 ​​and / or configuring data collection or device settings for various sensor devices 470 and / or electronic devices 590. The method instructions may be executed by the processor 302. In some embodiments, one or more other satellite devices 560, hub device 501, sensor device 570, server 511, electronic device 590, remote computing device 531, or any combination thereof can collect and measure physical quantities and / or provide instructions for configuring and operating satellite device 360 ​​and / or communicating between a user and other remote, local, and / or wireless electronic devices.

[0057] The memory / storage 303 may include dynamic random access memory (DRAM) and / or read-only memory (ROM). The memory / storage 303 may provide a temporary location to store data 305 and instructions 304 retrieved and processed by the processor 302. The memory / storage 303 may include non-volatile read-write memory that stores data 305 and instructions 304 even when Wi-Fi / Internet is turned off, which can be retrieved and processed by the processor 302. For example, the memory / storage 303 may include magnetic, solid-state and / or optical media, and the memory / storage 303 may be a single or multiple memory units as desired. The memory / storage 303 stores all collected visual, audio, text, voice, motion, heat, proximity, etc. information provided directly from one or more satellite devices 560, sensor devices 570, electronic devices 590, or servers 511, or all collected visual, audio, text, voice, motion, heat, proximity, etc. information provided indirectly to another one or more electronic devices, sensor devices (e.g., wearable devices) through a wireless connection.

[0058] Satellite device 360 ​​is coupled to a network via network interface 313. In some embodiments, network interface 313 is a machine interface. In this manner, satellite device 360 ​​may be part of a computer network, a local area network (LAN), a wide area network (WAN), or an intranet or a network of networks (e.g., the Internet). Wireless controller 315 may be coupled to processor 302. Wireless controller 315 may also be coupled to antenna 380. Network module 311 may be integrated as a system-in-package or system-on-chip device and / or collectively defined as having a network interface 313 and a wireless controller 315. Network interface 313 and wireless controller 315 are integrated into network module 311 and coupled to antenna 380. Any or all components of satellite device 360 ​​may be used in conjunction with the present subject disclosure. Network interface 313 may include a cellular interface, Wi-Fi TM interface, infrared interface, RFID interface, ZigBee interface, Bluetooth interface, Ethernet interface, coaxial interface, optical interface or any communication interface that can be generally used for device communication. Satellite device 360 ​​can be configured as Wi-Fi TM Bridge or Wi-Fi TM An extender to extend the range of the hub device 201 and thereby connect and collect additional individual and environmental information collected from sensor devices 570, electronic devices 590, server 511, or remote computing devices outside the range of the hub device 201, as well as to indirectly collect such information through a wireless connection to another one or more electronic devices, sensor devices (e.g., a wearable device).

[0059] Satellite device 360 ​​is powered by power supply 340. Power supply 340 may include disposable and / or rechargeable batteries (e.g., 3800mAh rechargeable lithium polymer battery), existing electrical wiring, power adapter, or any combination thereof. Disposable or rechargeable batteries, such as nickel cadmium (NiCd), lithium (Li), AA, AAA, and / or rechargeable capacitors, such as supercapacitors (SC) or supercapacitors. Power supply 340 may power satellite device 360, for example, through a power adapter for connecting to an outlet, a solar panel / battery, or any other renewable / alternative power source. Satellite device 360 ​​may use a variety of battery types, multiple power sources, etc., for example, using a button cell battery to operate some sensor components 355. In addition, converter / regulator 341; transformer or voltage regulator, AC to DC or DC to DC power converter, or frequency converter may be used alone (electrically coupled to satellite device 360) or integrated within satellite device 360 ​​to provide sufficient input power (e.g., 12VDC) to satellite device 360.

[0060] The satellite device 360 ​​may include a controller 354 for controlling the sensors and processing data collected by the sensors, other satellite devices 360, sensor devices 470, electronic devices 190, or remote computing devices 531. The controller 354 may include a processor, memory / storage (storing sensor instructions, settings, etc.), and a network module wireless chip for communicating with the satellite device 360. The controller 354 may send the measured / detected environmental conditions and characteristics and individual activities to the processor 302 for further processing. In some exemplary embodiments, the satellite device 360 ​​may not include a controller 354 and function as a sensor-only device or a satellite device 360 ​​that transmits the collected environmental and individual activities in and around the building to another satellite device 360 ​​for processing.

[0061] In some exemplary embodiments, satellite devices 360 include controllers 354 to share or divide processing tasks or priorities of data, video, audio, or environmental sensor data with other satellite devices 360, sensor devices 470, or hub device 401. For example, controller 354 can process certain motions (e.g., individuals, homeowners, pets or animals, etc.) or sounds (e.g., windows or doors closing or opening, windows breaking) and sound an alarm, request verbal input from a user, or trigger an action instead of sending to another satellite device 360 ​​for further processing (or before sending to another satellite device 360). Similarly, satellite devices 360 can process individual and environmental activities before sending to hub device 201 and / or server 511 for further processing, if desired.

[0062] The satellite device 360 ​​may transmit information collected from the sensor assembly 355, other satellite devices 360, sensor device 470, electronic device 190, remote computing device 431, and server 511 to one or more hub devices 201 and / or one or more satellite devices 360. In some embodiments, the satellite device 360 ​​may store part or all of the information collected from the sensor assembly 355, other satellite devices 360, sensor device 470, electronic device 190, remote computing device 431, and server 511 in a database before or after transmitting the information to one or more hub devices 201 and / or one or more satellite devices 360. The database may be stored on the additional / external storage device 320 of the satellite device 360, the memory 303, the storage device 512 of the server 511, or on an application stored on the remote computing device 531. The spatial and individual information (e.g., individual activity information) in the database is updated and / or confirmed with the individual and spatial information obtained by the collected information of the surrounding environment. The user or individual may be prompted to update or approve an update to the database with additional spatial and individual information acquired by one or more sensors. The user or individual may also store user preferences and / or classifiers in the database, which may include specific instructions or actions based on collected spatial or individual information, scheduling, time of day, temperature, humidity, etc.

[0063] Spatial and individual information acquired by the satellite device 360 ​​from various sources (e.g., but not limited to, sensor assembly 355, other satellites 360, sensor device 470, electronic device 190, or remote computing device 531) can be compared to classifiers and / or user activities stored in a database, and the satellite device 360 ​​can then use the classifiers and user preferences to determine whether to transmit the information to one or more hub devices 201 to notify other users, call emergency services, or connect, power, or operate various electronic devices, such as controlling existing light switches, ceiling fan controls, ceiling light fixtures, light fixture controls, dimmers, sound or motion sensor units, and traditional light switch sockets, IoT devices, smart home devices, thermostats, cameras, speakers, intercoms, virtual assistants (e.g., voice-operable AI devices), etc.

[0064] In an exemplary embodiment, satellite device 360 ​​includes one or more motion sensors 357 for detecting motion information. For example, motion sensor 357 can detect moving objects and / or pedestrians. In some exemplary embodiments, one or more sensors (e.g., motion sensor 357, image sensor 358, etc.) can be positioned along one or more edges of satellite device 360. Motion sensor 357 can be a passive infrared motion detector. Infrared motion sensors are also referred to as PIR (passive infrared) motion sensors or simply PIR sensors. This detector has an arc of approximately 120° and a detection area of ​​approximately 50 feet. In the case of a need for an increased motion detection field of view or more accurate motion detection, two or more motion detectors can be used.

[0065] Suitable alternative motion detectors may also be used, such as ultrasonic, optical, microwave, or video motion detectors. Still other alternative types of motion detectors may also be used to sense intrusion, including laser scanning or frequency sensitive detectors, commonly referred to as "glass break detectors." Motion sensor 357 may include an image sensor having any type of low light level imaging sensor for monitoring during daytime and unattended to complete darkness, such as a low light complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor.

[0066] The motion sensor 357 may also be supplemented with other devices to help detect motion, such as, for example, photocell sensors, cadmium sulfide (CdS) cells, light dependent resistors (LDRs), and photoresistors. In addition to motion sensors, photocell sensors may be used to determine if there is something blocking the light in front of a sensor or a series of sensors. The sensitivity of the motion sensor and photocell may be adjusted, for example, by an application on an electronic device (e.g., smart device 534 or laptop computer 531). In addition, a server or application may decide whether a situation or application warrants motion detection for nighttime use or twenty-four hour operation through an alternative device such as a photocell sensor. If nighttime operation is selected, the server or application will process the detected photocell information to determine if motion is detected.

[0067] Satellite device 360 ​​may include a light source 359a, which may include an LED array, such as an LED ring. The LED array may include a plurality of RGB lights that respond to input or feedback received from sensor assembly 355, touch input from a user, or remote or wireless input from a human-machine interface (HMI), such as instructions from one or more remote computers (e.g., one or more servers, mobile devices, etc.), or any combination thereof. In some embodiments, light source 359a may be wirelessly operated or activated by, for example, hub device 201, another visitor or secondary user through HMI, one or more satellites 160, tags 170, smart home devices, electronic devices, or any combination thereof. Light source 359a may include one or more multi-color LEDs, daylight LEDs, soft light LEDs, animated multi-color LED arrays, or any combination thereof to aid in peaceful sleep. Light source 359a may be embedded within an outer surface of satellite device 360, or formed along one or more outer surfaces or curves of satellite device 360. Light source 359a may include one or more light source (e.g., LED) arrays to display animations, cartoons, visual emoticons, images, text, brands. In some embodiments, the satellite device 360 ​​may allow the user to customize and / or include preset pre-programmed settings for the light source 359a. For example, the light source 359a may be configured by the satellite device 360 ​​or the user to display visual feedback (e.g., themes, colors, images, animations, videos, text, or information) for a scheduled event, reminder, appointment, task, or seasonal, cultural, historical, economic, socioeconomic, or geographic event, or to allow individuals to celebrate or commemorate events or traditions of cultural or religious significance. The light source 359a may also include a display, such as, but not limited to, a resistive touch display or a capacitive touch display, a projector display, or other touch or pressure-sensitive surfaces for receiving user input, etc. In some exemplary embodiments, other forms of interaction with the satellite device 360 ​​may be user input commands through the microphone 252 to the hub device 201, the wireless user device 532, one or more electronic devices 590, the remote computing device 531, the server 511, or any combination thereof.

[0068] The push feature 359b can be a toggle mechanical button or switch and / or a touch-sensitive button or touch pad. In some embodiments, the push feature 359b can include a light source to communicate with the user by, for example, providing visual feedback of a button press, visual feedback for a scheduled event, reminder, appointment, task, or visual feedback for other programmed or scheduled events or information. The visual feedback can be the same, different, preset by the satellite device 360, or configured by the user. The user can confirm the visual feedback by pressing the push feature 359b, communicating through one or more remote computing devices 531 (e.g., one or more servers, mobile devices, etc.), or communicating through audio (e.g., voice commands) or motion (e.g., gestures, body positions, or body movements) to confirm the feedback.

[0069] Figure 4 Conceptually illustrates an exemplary sensor device of a monitoring system that can implement some implementations of the present disclosure. Figure 4 As can be seen, the exemplary sensor device 470 may include one or more of the following: a processor 402, a memory 403 with instructions 404 and / or data 405, a network module 411, a network interface 413, a wireless controller 415, a storage device 420, an antenna 430, a power supply 440, a power regulator / converter 441, and a controller 454. The sensor device 470 may also include one or more sensor components 455, which include at least one environmental sensor 456, at least one motion sensor 457, at least one other sensor 459, at least one light source 459a (e.g., an LED, an LED array, a night light, etc.), and one or more press features 459b. The sensor device 470 may be any electronic device that transmits signals over a network, such as an electronic device embedded in a smart home appliance and other smart systems. The sensor device 470 may include various types of computer-readable media (e.g., non-transitory computer-readable media) and interfaces for various other types of computer-readable media. The sensor device 470 may include one, none, some, or all of the components of the hub device 201 as described above and in this disclosure.

[0070] The processor 402 may retrieve and execute instructions 404 and / or data 405 from the memory / storage device 403 to perform the processes of the present disclosure. In different implementations, the processor 402 may be a single processor, a multi-core processor, or multiple processors. Figures 2 to 6, instructions and data for operating the sensor device 470 may be stored on, sent from, or received by any computer-readable storage medium (e.g., memory 203, 303 or attached / external storage 220, 320 of the hub device 201 or satellite device 360, or memory / storage 512 of one or more servers 511) that stores data (e.g., data 405) accessible to a processor (e.g., a processor of a server 511) during an operating mode of the sensor device 470. The sensor device 470 may access and execute instructions and / or data stored on any remote computing device 531 (e.g., a mobile device 532, a laptop computer 533, or a tablet computer 534). The data 405 may be method instructions for configuring network settings for the sensor device 470 and / or configuring data collection or device settings for various sensor devices 470 and / or electronic devices 590. The method instructions may be executed by the processor 402. In some embodiments, one or more satellite devices 560, hub device 501, other sensor devices 570, server 511, electronic device 590, remote computing device 531, or any combination thereof can collect and measure physical quantities and provide instructions for configuring and operating sensor device 470 and / or communicating between a user and other remote, local, and / or wireless electronic devices.

[0071] Memory / storage 403 may include dynamic random access memory (DRAM) and / or read-only memory (ROM). Memory / storage 403 may provide a temporary location to store data 405 and instructions 404 that are retrieved and processed by processor 402. Memory / storage 403 may include non-volatile read-write memory that stores data 405 and instructions 404 that may be retrieved and processed by processor 402 even when Wi-Fi / Internet is turned off. For example, memory / storage 403 may include magnetic, solid-state, and / or optical media, and memory / storage 403 may be a single or multiple memory cells as desired. Memory / storage 403 stores all collected visual, audio, text, voice, motion, thermal, proximity, etc. information provided directly from one or more onboard sensors.

[0072] The sensor device 470 is coupled to the network via the network interface 413. In some embodiments, the network interface 413 is a machine interface. As such, the sensor device 470 may be part of a computer network, a local area network (LAN), a wide area network (WAN), or an intranet or a network of networks (e.g., the Internet). The wireless controller 415 may be coupled to the processor 402. The wireless controller 415 may also be coupled to the antenna 430. The network module 411 may be integrated as a system-in-package or system-on-chip device and / or collectively defined as having a network interface 413 and a wireless controller 415. The network interface 413 and the wireless controller 415 are integrated into the network module 411 and coupled to the antenna 430. Any or all components of the sensor device 470 may be used in conjunction with the present subject disclosure. The network interface 413 may include a cellular interface, Wi-Fi TM interface, infrared interface, RFID interface, ZigBee interface, Bluetooth interface, Ethernet interface, coaxial interface, optical interface or any communication interface that can generally be used for device communication.

[0073] The sensor device 470 may be powered by a power source 440. The power source 440 may include disposable and / or rechargeable batteries (e.g., a 4800 mAh rechargeable lithium polymer battery), existing electrical wiring, a power adapter, or any combination thereof. A disposable battery or a rechargeable battery, such as nickel cadmium (NiCd), lithium (Li), AA, AAA, and / or a rechargeable capacitor, such as a supercapacitor (SC) or a supercapacitor. The power source 440 may power the sensor device 470, for example, through a power adapter for connecting to an outlet, a solar panel / battery, or any other renewable / alternative power source. The sensor device 470 may use a variety of battery types, multiple power sources, etc., for example, using a button cell battery to operate some sensor components 455. In addition, a converter / regulator 441; a transformer or voltage regulator, an AC to DC or DC to DC power converter, or a frequency converter may be used alone (electrically coupled to the sensor device 470) or integrated within the sensor device 470 to provide sufficient input power (e.g., 12 VDC) to the sensor device 470. In some embodiments, sensor device 470 may be a passive device that is remotely powered upon receiving electromagnetic waves, which, for example, charges a capacitor to power sensor device 470 .

[0074] The sensor device 470 may include a controller 454 for controlling the sensor and processing data collected by the sensor, other satellite devices 460, the sensor device 470, the electronic device 190, or the remote computing device 531. The controller 454 may include a processor, a memory / storage device (storing sensor instructions, settings, etc.), and a network module wireless chip for communicating with the sensor device 470. The controller 454 may send the measured / detected environmental conditions and characteristics and individual activities to the processor 402 for further processing. In some exemplary embodiments, the sensor device 470 may not include a controller 454 and function as a sensor-only device or a sensor device 470 that transmits the collected environmental and individual activities in and around the building to the hub device 201, the satellite device, and / or relays to another sensor device 470 for processing.

[0075] In some exemplary embodiments, the sensor device 470 includes a controller 454 to share or divide processing tasks or priorities of data, video, audio, or environmental sensor data with other sensor devices 470, satellite devices 360, or hub device 401. For example, the controller 454 can process certain motions (e.g., individuals, homeowners, pets or animals, etc.) or sounds (e.g., windows or doors closing or opening, windows breaking) and sound an alarm, request verbal input from a user, or trigger an action instead of sending to another sensor device 470 for further processing (or before sending to another sensor device 470). Similarly, if desired, the sensor device 470 can process individual and environmental activities before sending to the hub device 201 and / or server 511 for further processing.

[0076] The sensor device 470 may include and control various sensor components 455 for sensing environmental activities near the building (e.g., temperature, sound, motion, and position of individuals, and their respective changes over time). The sensor components 455 may monitor environmental conditions (e.g., humidity, temperature, pressure, etc.) by using one or more environmental sensors 456, and monitor individual activities by using one or more motion sensors 457 and other sensors 459. Thus, a combination of sensor components 455 may be implemented to provide comprehensive monitoring or improved accuracy when monitoring individual and environmental activities. In some embodiments, the sensor device 470 includes only a single sensor component (e.g., a motion sensor, a temperature sensor, etc.) that is operable to produce or more specific measurements.

[0077] The sensor device 470 may transmit information collected from the sensor assembly 455, the satellite device 460, the other sensor devices 470, the electronic device 190, the remote computing device 431, and the server 511 to the one or more hub devices 201 and / or the one or more satellite devices 460. In some embodiments, the sensor device 470 may store the information collected from the sensor assembly 455, the other satellite devices 460, the sensor device 470, the electronic device 190, the remote computing device 431, and the server 511 in a database, in part or in whole, before or after transmitting the information to the one or more hub devices 201 and / or the one or more satellite devices 460. The database may be stored on the additional / external storage device 420 of the sensor device 470, the memory 403, the storage device 512 of the server 511, or on an application on the remote computing device 531.

[0078] Environmental sensors 456 can detect and collect information about environmental conditions in and around one or more buildings. Environmental sensors 456 can include, for example, temperature sensors, ambient light sensors, humidity sensors, air pressure sensors, air quality sensors (e.g., for detecting allergens, gases, pollution, pollen, etc.), infrared sensors, CO2 sensors, CO sensors, piezoelectric sensors, airflow or airspeed sensors, etc. The environmental conditions collected by environmental sensors 456 can be used by processor 202 of hub device 201 to determine whether to notify a user (e.g., via wireless user device 532). Environmental sensors 456 can include, for example, motion sensors, image sensors, cameras, and other sensors (e.g., proximity sensors, occupancy sensors, ambient light sensors).

[0079] In an exemplary embodiment, the sensor device 470 includes one or more motion sensors 457 for detecting motion information. For example, the motion sensor 457 can detect moving objects and / or pedestrians. In some exemplary embodiments, one or more sensors (e.g., motion sensor 457, image sensor 458, etc.) can be positioned along one or more edges of the sensor device 470. The motion sensor 457 can be a passive infrared motion detector. Infrared motion sensors are also referred to as PIR (passive infrared) motion sensors or simply PIR sensors. This detector has an arc of approximately 120° and a detection area of ​​approximately 50 feet. In the case of a need for an increased motion detection field of view or more accurate motion detection, two or more motion detectors can be used.

[0080] Suitable alternative motion detectors may also be used, such as ultrasonic, optical, microwave, or video motion detectors. Still other alternative types of motion detectors may also be used to sense intrusion, including laser scanning or frequency sensitive detectors, commonly referred to as "glass break detectors." Motion sensor 457 may include an image sensor having any type of low light level imaging sensor for monitoring during daylight hours and unattended monitoring to complete darkness, such as a low light complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor.

[0081] The motion sensor 457 may also be supplemented with other devices to help detect motion, such as, for example, a photocell sensor, a cadmium sulfide (CdS) cell, a light dependent resistor (LDR), and a photoresistor. In addition to the motion sensor, the photocell sensor may be used to determine if there is something blocking the light in front of the sensor or a series of sensors. The sensitivity of the motion sensor and the photocell may be adjusted, for example, by an application on an electronic device (e.g., smart device 534 or laptop computer 531). In addition, the server or application may decide whether the situation or application warrants motion detection for nighttime use or twenty-four hour operation through an alternative device such as a photocell sensor. If nighttime operation is selected, the server or application will process the detected photocell information to determine if motion is detected.

[0082] Sensor device 470 may include any number of other or additional detectors or sensors, such as other sensors 459. Examples of other sensors 459 that may be used include, by way of illustration and not limitation, time of flight (ToF) sensors, temperature sensors, cameras, audio recorders, motion sensors, ambient light sensors, light sensors, humidity sensors, smoke detectors, and other sensors such as, for example, electric field proximity sensing (EFPS) sensors to determine if a person or object is nearby behind a wall.

[0083] The sensor device 470 may include a light source 459a, which may include an LED array, such as an LED ring. The LED array may include a plurality of RGB lights that respond to input or feedback received from the sensor assembly 455, touch input from a user, or remote or wireless input from a human-machine interface (HMI), such as instructions from one or more remote computers (e.g., one or more servers, mobile devices, etc.), or any combination thereof. In some embodiments, the light source 459a may be wirelessly operated or activated by, for example, the hub device 201, another visitor or secondary user through the HMI, one or more satellites 160, tags 170, smart home devices, electronic devices, or any combination thereof. The light source 459a may include one or more multi-color LEDs, daylight LEDs, soft light LEDs, animated multi-color LED arrays, or any combination thereof to help with peaceful sleep. The light source 459a may be embedded within the outer surface of the sensor device 470, or formed along one or more outer surfaces or curves of the sensor device 470. The light source 459a may include one or more light source (e.g., LED) arrays to display animations, cartoons, visual emoticons, images, text, brands. In some embodiments, the sensor device 470 may allow the user to customize and / or include preset pre-programmed settings for the light source 459a. For example, the light source 459a may be configured by the sensor device 470 or the user to display visual feedback (e.g., themes, colors, images, animations, videos, text, or information) for a predetermined event, reminder, appointment, task, or seasonal, cultural, historical, economic, socioeconomic, or geographic event, or to allow individuals to celebrate or commemorate events or traditions of cultural or religious significance. The light source 459a may also include a display, such as, but not limited to, a resistive touch display or a capacitive touch display, a projector display, or other touch or pressure-sensitive surfaces for receiving user input, etc. In some exemplary embodiments, other forms of interaction with the sensor device 470 may be user input commands via the microphone 252 to the hub device 201, the wireless user device 532, one or more electronic devices 590, the remote computing device 531, the server 511, or any combination thereof.

[0084] The push feature 459b can be a toggle mechanical button or switch and / or a touch-sensitive button or touchpad. In some embodiments, the push feature 459b can include a light source to communicate with the user by, for example, providing visual feedback of a button press, visual feedback for a scheduled event, reminder, appointment, task, or visual feedback for other programmed or scheduled events or information. The visual feedback can be the same, different, preset by the sensor device 470, or configured by the user. The user can confirm the visual feedback by pressing the push feature 459b, communicating through one or more remote computing devices 531 (e.g., one or more servers, mobile devices, etc.), or communicating through audio (e.g., voice commands) or motion (e.g., gestures, body positions, or body movements) to confirm the feedback.

[0085] To minimize power consumption, the hub device 201, satellite device 360, sensor device 470, and cellular pendant 180 can intermittently use GPS without constant polling to minimize power usage. In addition, an additional battery (e.g., a button battery) can be used as a dedicated power source for one or more sensors and GPS. In addition, low-power extended discontinuous reception (EDRX) can be used for out-of-band signaling.

[0086] Figure 5 Shows communication with other smart devices or remote computing devices Figure 1-4 An exemplary embodiment of a monitoring system. Figure 5 As can be seen, an exemplary embodiment of a monitoring system 500 (hub device 501, one or more satellite devices 560, and one or more sensor devices 570) (hereinafter referred to as "monitoring system 500"). In the following exemplary embodiments, the description of the monitoring system 500 may refer to one of the devices, for example, the hub device 501 that notifies the user of individual and / or environmental activities or the satellite device 560 that notifies the user (e.g., wireless device 532) and the hub device 501 of individual and / or environmental activities. Alternatively, the monitoring system 500 may refer to a group of devices working together, for example, the satellite device 560 works together with the hub device 501 to notify the user of individual and / or environmental activities.

[0087] The sensor device 570 as described in the present disclosure can be configured to measure only one physical quantity, for example, the sensor device 570 can include only a motion sensor. In some embodiments, the sensor device 570 can be configured to measure multiple physical quantities, for example, the sensor device 570 can include an ambient light sensor, a temperature sensor, an occupancy sensor, and a motion sensor. In some embodiments, several sensor devices 570 can be placed throughout the building to provide an array of measurements that a caregiver may wish to monitor throughout the building. Then, the array of sensor devices 570 can consistently collect and measure the same physical quantity, which can be relayed to one or more hub devices 501 (or remote computing devices) for further processing. The redundancy of the measurements collected by the sensor device 570 can promote redundancy of data collection (e.g., measuring and confirming human movement in a building) and a heat map of activities that often or rarely occur in a building. This information can provide valuable information to caregivers about where to place the sensor device 570 in and around the building and what activities to collect (i.e., what type of sensor device 570 is placed in the space).

[0088] In addition, the hub device 501 or any other component of the monitoring system 500 (e.g., one or more satellite devices 560, one or more sensor devices 570, etc.) can process the collected activity information without sharing or distributing the information to the server 511 or remote computing device 531 to further maintain privacy. In some embodiments, the processing of the measured physical quantities can be completed within the building through edge computing (i.e., not going to the server 511) to remove "noise" and produce results. The caregiver or user selectively selects the sensor device 570 to collect relevant information about the individual / patient processed by the hub device 501 and / or one or more components of the monitoring device 500.

[0089] In addition, the sensor device 570 can be configured to shut down or enter a low power mode in the absence of information or activity, in the collection of activity, or in the transmission of activity, so as to save battery life or reduce power consumption. For example, audio and video broadcasts can be paused, stopped, compressed or downsampled, or broadcasted from one or more sensor devices 570 to one or more satellites 560 and / or hub devices 501 at a lower resolution or less frequently. Then, the sensor device 570 can become active when needed or when one or more specific activities occur. As another example, the sensor device 570 may include one or more accelerometers configured to run at a low speed (e.g., 25Hz) and generate an interrupt, and when there is a sudden movement, the accelerometer can turn the sensor device 570 to full power, and when there is a lack of movement from the accelerometer, the sensor device 570 can be powered off to, for example, only run an inertial measurement unit (IMU). The interrupt line can be bound to a button on any component of the IMU and the monitoring device 500, and if the IMU or button sends an interrupt signal, any such component of the monitoring device 500 can be powered on.

[0090] In some exemplary embodiments, the monitoring system 500 can be linked to a transmitter (e.g., a wireless user device 532 or a remote computing device 531) via Wi-Fi, LAN, WAN, Bluetooth, a two-way pager, a cellular connection, etc. The monitoring system 500 can learn user habits, patterns, and behaviors by communicating with one or more electronic devices 590, remote computing devices 531, and servers 511 via, for example, a wireless router 521.

[0091] The monitoring system 500 may include wireless communications with one or more local electronic devices 590, remote computing devices 531, and servers 511, for example, via a wireless router 521. The local electronic devices 590 may include, for example, healthcare or nursing equipment (e.g., scales, blood pressure monitoring devices, oximeters, spirometers, electrogram monitoring devices, suction devices, oxygen machines or masks, etc.), IP cameras, smart sockets, smart switches, smart light bulbs, smart locks, smart thermostats, video game consoles and smart TVs, smart blinds, garage door monitoring and control devices, smart refrigerators, smart washers / dryers, solar-powered smart devices, etc. The monitoring system 500 may also be connected to a laptop computer 533, a portable device 534, a wireless user device 532, and a server 511 and / or server storage 512.

[0092] The monitoring system 500 can collect, classify, store, and process user habits, patterns, and behaviors to predict and / or learn appropriate actions based on the user's interactions with the monitoring system 500, the electronic device 590, the remote computing device 531, and the server 511. For example, the monitoring system 500 can collect and process the user's interactions with, for example, one or more sensor modules 570, the electronic device 590, the server 511, the state and location of a transmitter (e.g., a wireless user device 280), or the user's interactions with the sensor module 570, the electronic device 590, or any combination of the foregoing.

[0093] The monitoring system 500 can transmit user interactions, habits, patterns, and behaviors to the server 511, electronic device 590, remote computing device 531, etc. for further processing. For example, the hub device 501 can activate or operate the satellite device 560 or sensor device 570 at certain times based on scheduling or user interaction to collect and process user interactions, habits, patterns, and behaviors.

[0094] In addition, user interactions can be classified, cataloged and / or stored in one or more databases (e.g., hub device storage 502 or server storage 512, etc.) for mapping user habits, patterns and behaviors to predict and / or learn appropriate actions and responses that can be taken by the monitoring system 500, server 511 and / or transmitted by the monitoring system 500 or server 511 to one or more local electronic devices 590 or remote computing devices 531 to take one or more appropriate actions.

[0095] For example, when detected user activity conflicts with the state or location of the transmitter or with a user pattern or habit, the monitoring system 500 can notify the user of the location of the transmitter. The user activity can be collected by the monitoring system 500 and / or one or more local electronic devices 590 or remote computing devices 531. For example, the monitoring system 500 can notify the user by playing an audio message when the user needs to take their medication, prepare a meal, or perform daily activities or exercise.

[0096] In some exemplary embodiments, components of the monitoring system 500 may include one or more communication modules for wirelessly communicating (e.g., ESPNow, Bluetooth, Wi-Fi, etc.) between one or more sensor devices 570, satellite devices 560, and hub devices 501 and / or with one or more remote computing devices 531, servers 511, local electronic devices 590, or any other electronic devices described above to further improve the efficiency of the monitoring system 500.

[0097] The one or more communication modules may include, for example, basic low-power communication modules for communicating with and between components of the monitoring system 500, as well as more robust or higher-power communication modules for communicating with other electronic devices 590, connecting to the Internet, or streaming or distributing audio, visual, or motion information to other electronic devices 590 via P2P or direct connections. The data / audio / video sent by the satellite device 560 to the hub device 501 may be sent as an uncompressed data / audio / video file, which the hub device 501 may then compress and send to the server 511.

[0098] The monitoring system 500 may include an anti-tampering component that can activate the monitoring system 500 image sensor 258 and / or one or more cameras 190D to record video and stream it to one or more remote computing devices 531, server 511, or local electronic device 590 when the housing or portion of the housing of any component of the monitoring system 500 (e.g., a battery cover) is tampered with or damaged, and / or when an entry door or window is broken (e.g., an entry door is opened or glass breaking sound is detected).

[0099] Additionally, the monitoring system 500 may include nighttime LEDs that may operate based on the time in the installation time zone to provide visitors or owners with better lighting conditions for collecting video at night and / or to provide a convenient nighttime lighting feature in entryways to buildings.

[0100] In some exemplary embodiments, when an access point (e.g., a router) loses Internet connectivity, or when one or more components of the monitoring system 500 lose network connectivity, the hub device 501 or the satellite device 560 may temporarily store data / video / audio in the additional / external storage module 220, 320.

[0101] In addition, in some exemplary embodiments, the components of monitoring system 500 can be in a normally dormant state (e.g., economy mode, sleep mode, etc.). For example, sensor device 570 and / or satellite device 560 can be turned off or substantially turned off (e.g., low power mode) until motion, sound, or finger press triggers the device to go to full power mode. In addition, in some exemplary embodiments, the housing of any of the components of the monitoring system can include a resistive or capacitive touch sensor and a fingerprint sensor formed on one or more components or externally as a manual button and fingerprint reader.

[0102] Once activated by a voice command, body gesture, or posture, the monitoring system 500 may attempt to initiate an audio or video call and / or an audio or video walkie-talkie session. The monitoring system 500 may collect individual conversations or activities and send the communications as live audio or video streams or recorded video clips or audio clips to one or more servers 511, remote computing devices 531, or local electronic devices 590, or any combination thereof. The communication will initiate a video or audio conference call with the user using the speaker 251, microphone 252, and image sensor 258. The video or audio conference call may be terminated when the individual at the entry door leaves, or when the user terminates the video or audio conference call by, for example, interacting with the wireless user device 532 (e.g., finger presses, eye movements, or other control commands).

[0103] The monitoring system 500 can be configured to wirelessly communicate and collaborate with the local electronic device 590 in real time based on the collected individual and / or environmental activities or visual, motion, audio, individual and / or environmental information stored in the hub device storage 202 or server storage 512. The processor 202, controller 254 and / or server 511 can operate the hub device 501 to play a digital or analog ringtone, greeting or collect environmental activities (e.g., video, audio, temperature, etc.) to send to a computing device (e.g., local electronic device 590, remote computing device 531, server 511, etc.) based on the triggered individual and / or environmental activities collected by the monitoring system 500. The user can further define activity areas for collecting, sorting and storing information or triggering notifications for the user, for example, the user can select or define areas or regions within the space of the building by the image or live video of the environment or the proximity of the activities of the devices from the monitoring system 500.

[0104] Other local electronic devices 590 (e.g., security cameras, thermostats, smoke detectors, smart locks, smart TVs, etc.) can collaborate with or supplement sensor devices 570 to provide comprehensive information about environmental activity around the building or one or more areas around the building. In some exemplary embodiments, security cameras 190D can add additional monitoring (data, audio or video) information to allow one or more hub devices 501 to collect, filter or learn tenant activity around the building.

[0105] The monitoring system 500 can be configured to communicate between the above-mentioned local electronic devices 590 (e.g., security devices, smart thermostats, smart devices, or smart appliances) by sending and retrieving proximity information, scheduling information, text (e.g., email, SMS, MMS, text, etc.), visual, motion, or audio information, and user access information shared between electronic devices. For example, the monitoring system 500 can be configured to be notified by these smart devices of external weather conditions, user health or life conditions, vehicle or user location, air quality, allergens / pollen, rush hours, etc. Notification can be performed by text, email, visual, or audio information provided by the hub device 501 to the remote computing device 531, the server 511, and / or the local electronic device 590 or any other electronic device mentioned above.

[0106] In this way, the monitoring system 500 acts as a hub for collecting and processing environmental activities from other electronic devices, classifying the frequency, pattern and / or time of occurrence, and then prompting the server 511 or remote computing device 531 to perform control instructions to play audio files, messages, videos, or collect environmental activities (e.g., data, video, audio, temperature, etc.) to send to the computing device (e.g., base module 301, local electronic device 590, remote computing device 531, server 511, etc.). The monitoring system 500 can also operate the local electronic device 590 based on the classifier, user conditions or preferences. For example, if the user must perform a daily activity (e.g., exercise or walking), the monitoring system 500 can set the electronic device 590 to relay information by playing a message or displaying a sequence or color to indicate that a specific daily activity needs to be performed, and the sensor device 570 and / or electronic device 590 will listen and collect motion and / or audio information within the building to confirm that the daily activity has been performed.

[0107] The monitoring system 500 may be communicatively coupled to and controlled, programmed, or reprogrammed by local electronic devices 590 in a building, a remote computing device 531, or one or more servers 511 to collect such data or to collect additional data.

[0108] The monitoring system 500 may also include a cellular hanger 180 with a panic button that the user can carry to operate a local electronic device 590 (e.g., a smart lock for an entry door) or contact emergency services. In some exemplary embodiments, the hanger 180 may be, for example, but not limited to, an RFID card or RFID device with an accelerometer that can be attached to a remote computing device 531. In some exemplary embodiments, the monitoring system 500 may be programmed by the user to respond to the hanger 180 based on a schedule, the user's geographic location, user preferences, user mobility, user falls, etc. The response may include any combination of operating one or more hub devices 501, one or more entry point devices 260, operating a local electronic device 590, calling a caregiver and / or emergency services, etc.

[0109] The monitoring system 500 may use a shared IP or a dedicated IP. A monitoring system 500 with a fixed or static IP may benefit from many advantages, such as, but not limited to, less downtime or power consumption from IP address refreshes, private SSL certificates, anonymous FTP, remote access, and access when a domain name is inaccessible.

[0110] The monitoring system 500 may also be communicatively coupled to one or more door sensors and window sensors. The door sensors and window sensors may notify the monitoring system 500 if a window or door is open, and the monitoring system 500 may then turn on and begin capturing audio and video of the event, and simultaneously or subsequently notify one or more local electronic devices 590, remote computing devices 531, servers 511, etc.

[0111] Figure 6 is a flow chart illustrating an exemplary method of classifying individual activities and corresponding environmental activities. The exemplary method is provided by way of example, as there are various ways of performing the method. Figure 6 Each block shown in the drawings represents one or more processes, methods, or subroutines performed in the exemplary method. Figure 1-5 Shows the execution Figure 6 The exemplary method may begin at block 600. Further for purposes of explanation, the blocks of the exemplary process 600 are described herein as occurring serially or linearly. However, multiple blocks of the exemplary process 600 may occur in parallel. Additionally, the blocks of the exemplary process 600 may be performed in an order different from that shown, and / or one or more blocks of the exemplary process 600 may not be performed.

[0112] Figure 6 The exemplary method of includes collecting individual and / or environmental activities in and / or around a building (block 601). The collected activities may be new activities, matching stored or previously occurring activities, or similar to new or previously stored activities.

[0113] The method also includes classifying the activity (block 603), which may include comparing the collected activity (sensor information) to a database of various measured physical quantities that have been used to define a previous activity or set of activities.

[0114] The method also includes creating or updating a count of occurrences of the collected activity, and if the collected activity matches the classified activity, the detected activity is added to the database as an additional occurrence of the classified activity (block 605).

[0115] The method also includes determining whether the classified activity occurs consistently (block 607). For example, the collected sensor information can be used to determine whether the user engages in meal preparation every day. One or more sensor devices 170 can be placed in the kitchen to measure movement in the kitchen, and the hub device 101 determines whether the movement occurs every day, at specific times (e.g., morning, afternoon, and evening), with specific sound signatures (e.g., faucet running, refrigerator opening, noise from pots or pans, etc.) to classify the activity / event as meal preparation.

[0116] The method also includes comparing the classified activity to other occurrences of the activity or similar activities (block 609). For example, if the sound of pots and pans is not detected, but the sound of microwaves is detected, the hub device 101 can continue to classify the event as meal preparation.

[0117] The method also includes determining whether the detected activity is a normal occurrence of activity (box 611). If the detected activity is a normal occurrence, the caregiver and / or other users of the activity are notified (box 613). The activity is then evaluated to determine whether the activity needs to be classified as a new activity, or otherwise stored in a database as an occurrence of a classified activity (box 615).

[0118] However, if the detected activity does not normally occur (box 611), the patient's activity is evaluated to determine if they have recently moved, eaten, and / or had a bowel movement (box 614). If it is determined that the patient has not moved for a predetermined amount of time, the caregiver and / or specialist is notified (box 616). If it is determined that the patient has moved, the caregiver and / or other users of the activity are notified (box 613). The activity is then evaluated to determine if the activity needs to be classified as a new activity, or otherwise stored in a database as an occurrence of a classified activity (box 615).

[0119] Figure 7 is a flow chart illustrating an exemplary method for monitoring a system. The exemplary method is provided by way of example, as there are various ways of performing the method. Figure 7Each frame shown in the example process 700 represents one or more processes, methods or subroutines performed in the exemplary method. For the purpose of explanation, the frames of the example process 700 are described as occurring serially or linearly in this article. However, multiple frames of the example process 700 can occur in parallel. In addition, the frames of the example process 700 can be performed in an order different from the order shown, and / or one or more frames of the example process 700 can not be performed.

[0120] Figure 7 An exemplary method of the invention includes collecting activity information of individuals in an area via a plurality of sensor devices distributed across an area of ​​a monitoring system (block 701). In some embodiments, collecting the activity information includes capturing sensor data including the activity information of individuals in the area by the plurality of sensor devices; receiving sensor data including the activity information from the plurality of sensor devices by a satellite device of the monitoring system; and providing the activity information by the satellite device.

[0121] The method also includes classifying the activity information based on comparing the activity information with historical activity information stored in a database (block 703). In some embodiments, the activity information includes at least one of temperature, humidity, sound, or airflow of the area and individual information including a medical condition or daily activity amount of the individual.

[0122] The method also includes generating an activity heat map based on updating the number of activity occurrences according to the classified activity information (block 705). In some embodiments, the activity heat map is generated in a memory storage device that is divided into rows and columns. Each row of the memory storage device corresponds to a different sensor device from which the activity information originated, and each column of the memory storage device corresponds to a different time period associated with the activity information.

[0123] The method also includes providing an alert notification to a user device associated with the individual in response to detecting a non-occurrence of the normally occurring activity identified by the activity heat map (block 707). In some embodiments, detecting the non-occurrence of the normally occurring activity is performed using a deterministic analysis based on the activity heat map and detections obtained from one or more of the plurality of sensor devices.

[0124] In some embodiments, the method further comprises providing placement information indicating where to place the plurality of sensor devices within the area based on the activity heat map. In some embodiments, the method further comprises: dynamically including a new sensor device in the plurality of sensor devices based on a response to the new sensor device broadcasting a signal on the monitored open receiving channel; and collecting activity information of the individual from the new sensor device.

[0125] The following technical solutions can be implemented through some preferred embodiments.

[0126] 1) A monitoring system, comprising: a plurality of sensor devices, each of which is configured to collect sensor data from a corresponding environment, wherein the plurality of sensor devices are distributed at a plurality of locations within an area; one or more satellite devices, each of which is configured to facilitate wireless communication between the plurality of sensor devices and a hub device of the monitoring system; and the hub device communicates with the one or more satellite devices. The hub device is configured to: dynamically add new sensor devices to the plurality of sensor devices; receive sensor data originating from the plurality of sensor devices from the one or more satellite devices; generate an activity record for the area based on classifying activity information captured by the sensor data; and send an alert notification to a user device in response to determining that the occurrence or non-occurrence of a particular activity deviates from the activity record in the particular sensor data.

[0127] 2) The monitoring system according to solution 1, wherein the one or more satellite devices are each configured to communicate with the plurality of sensor devices via a first communication protocol and to communicate with the hub device via a second communication protocol.

[0128] 3) The monitoring system according to any one of Solution 1 or Solution 2, wherein dynamically adding a new sensor device to the plurality of sensor devices comprises building a whitelist identifying each of the plurality of sensor devices.

[0129] 4) A monitoring system according to any one of solutions 1 to 3, wherein the activity record includes a heat map identifying the location and frequency of activity.

[0130] 5) A monitoring system according to any one of solutions 1-4, wherein the multiple sensor devices include a pendant device carried by an individual and configured for cellular network communication.

[0131] 6) A monitoring system according to any one of solutions 1 to 5, wherein the activity record is specific to the individual monitored by the monitoring system, and wherein the user device to which the alarm notification is sent is associated with a caregiver who is responsible for the individual and is away from the area.

[0132] 7) A monitoring system according to any one of solutions 1 to 6, wherein the plurality of sensor devices include a specific sensor device, the specific sensor device including: a wireless controller, the specific sensor device communicates with the one or more satellite devices via the wireless controller; a microcontroller unit, the microcontroller unit is configured to execute instructions to operate the wireless controller; and a micro-electromechanical inertial measurement unit (IMU) coupled to the microcontroller unit. The micro-electromechanical IMU is configured to provide a digital input to the microcontroller unit to change the state of the microcontroller unit to a powered-on state in response to movement achieved on the micro-electromechanical IMU.

[0133] 8) A monitoring system according to solution 7, wherein the micro-electromechanical IMU is coupled to a chip enable pin of the microcontroller unit, and the chip enable pin is configured to keep the microcontroller unit in a power-off state until the micro-electromechanical IMU provides the digital input to change the state of the microcontroller unit.

[0134] 9) A monitoring system according to any one of solutions 1 to 8, wherein the hub device is configured to dynamically add the new sensor device in response to the new sensor device broadcasting a signal on an open receiving channel monitored by the hub device and further in response to receiving a confirmation of a query sent by the hub device to the user device, wherein the query identifies the new sensor device.

[0135] 10) A monitoring system according to any one of solutions 1-9, wherein the hub device is configured to generate the activity record in a memory storage device divided into rows and columns, wherein each row of the memory storage device corresponds to a different sensor device from which the hub device receives sensor data, and wherein each column of the memory storage device corresponds to a different time period for which the hub device receives sensor data from the multiple sensor devices.

[0136] 11) A monitoring system according to any one of solutions 1 to 10, wherein the multiple sensor devices include a time-of-flight distance sensor paired with an imaging sensor, wherein the time-of-flight distance sensor is suitable for contributing target size information to image information captured by the imaging sensor.

[0137] 12) A monitoring system according to any one of solutions 1 to 11, wherein the multiple sensor devices include at least three of the following types of sensor devices: temperature sensor, ambient light sensor, humidity sensor, barometer sensor, air quality sensor, infrared sensor, carbon dioxide sensor, carbon monoxide sensor, piezoelectric sensor or airflow sensor.

[0138] 13) A monitoring system according to any one of solutions 1 to 12, wherein the activity information includes both environmental information of the area and information about one or more individuals present in the area.

[0139] 14) A monitoring system according to solution 13, wherein the environmental information includes one of temperature, humidity, sound or airflow, and the information about the one or more individuals includes medical conditions or daily activity levels.

[0140] 15) A monitoring system comprising: one or more sensor devices, each sensor device configured to collect sensor data from a corresponding environment, wherein the one or more sensor devices are located within a structure; and a hub device located within the structure. The hub device is configured to dynamically include a new sensor device with the one or more sensor devices; obtain sensor data originating from the one or more sensor devices; generate a heat map of one or more areas within the structure based on classifying activity information captured by the sensor data, wherein the heat map identifies the type of activity, the location of the activity, and the sensor devices involved in detecting the activity; use the heat map as a reference to determine whether the occurrence or non-occurrence of a particular activity deviates from the heat map activity; and in response to determining that a deviation has occurred, generate a notification indicating the deviation.

[0141] 16) A monitoring system according to solution 15, wherein the one or more sensor devices are included in the hub device.

[0142] 17) A hub device of a monitoring system, the monitoring system comprising a plurality of sensor devices distributed across an area, the hub device comprising a processor and a memory having instructions stored therein, wherein the instructions, when executed by the processor, configure the processor to: dynamically maintain a network whitelist identifying the plurality of sensor devices; receive sensor data originating from the plurality of sensor devices from one or more satellite devices; generate an activity record for the area based on classifying activity information captured by the sensor data; and send an alarm notification to a user device in response to determining that the occurrence or non-occurrence of a particular activity deviates from the activity record in the particular sensor data.

[0143] 18) A hub device according to Solution 17, wherein dynamically maintaining the network whitelist includes: detecting a broadcast signal on an open receiving channel, wherein the broadcast signal originates from a new sensor device that is not identified by the network whitelist; performing a series of network performance tests using the new sensor device; and updating the network whitelist to additionally identify the new sensor device.

[0144] 19) A hub device according to any one of Solution 17 or Solution 18, wherein the plurality of sensor devices comprises a pendant device carried by an individual and configured for cellular network communication.

[0145] 20) A hub device according to any one of solutions 17 to 19, wherein the activity record is specific to an individual and wherein the user device to which the alarm notification is sent is associated with a caregiver who is responsible for the individual and is away from the area.

[0146] 21) A hub device according to any one of Solutions 17 to 20, wherein the hub device comprises: a wireless controller, wherein the hub device communicates with the one or more satellite devices via the wireless controller; a microcontroller unit, wherein the microcontroller unit comprises the processor; and a microelectromechanical inertial measurement unit (IMU) coupled to the microcontroller unit, wherein the microelectromechanical IMU is configured to change the state of the microcontroller unit to a powered-on state in response to movement implemented on the microelectromechanical IMU.

[0147] 22) A hub device according to any one of solutions 17-21, wherein the activity record is generated in a memory of the hub device, and the memory for the activity record is divided into rows and columns, wherein each row of the memory for the activity record corresponds to a different sensor device from which the sensor data originates, and wherein each column of the memory for the activity record corresponds to a different time period in which the hub device receives sensor data originating from the multiple sensor devices.

[0148] The remote computing device can be a smart device, smart phone, vehicle, tablet, laptop, TV, or any electronic device that can wirelessly connect to a network or join a wireless network. The remote computing device can access the network or server (e.g., through a user account on the server or WiFi) TM Login information) or through visual information collected by the hub device wirelessly and communicatively associated with the individual. The terms remote computing device, individual, and user may be used interchangeably throughout this disclosure.

[0149] A server can be a computer that provides data to other computers. It can provide data to systems on a local area network (LAN) or a wide area network (WAN) over the Internet. A server can include one or more types of servers (e.g., a web server or a file server), each running its own software specific to the purpose of the server sharing services, data, or files over a network. A server can be any computer that is configured to act as a server (e.g., a desktop computer or a single or multiple rack-mountable server) and can be accessed remotely using remote access software.

[0150] Proximity determination can be made using a combination of visual, motion, and audio information. A sensor assembly or sensors, a server, a remote computing device, and / or a monitoring system (hub device, satellite device, and / or sensor device) can define a virtual perimeter of a real-world geographic area. The monitoring system can also respond to geofencing triggers. Geofencing can be accomplished using location-aware devices through, for example, GPS, RFID technology, wireless network connection information, cellular network connection information, and the like. The monitoring system or server can collect visual, motion, and audio information to confirm the physical location of (multiple) individuals / remote computing devices.

[0151] The network may be a computer network, a local area network (LAN), a wide area network (WAN) or an intranet or a network of networks, such as the Internet. In addition, various interfaces may be used to connect to the network, such as a cellular interface, WiFi, or a wireless network. TM Interface. Infrared interface, RFID interface, ZigBee interface, Bluetooth interface, Ethernet interface, coaxial interface, optical interface or any communication interface that can be used for device communication in general. The purpose of the network is to enable sharing of files and information between multiple systems.

[0152] Low Power Wi-Fi Device Examples

[0153] A popular trend in microcontroller design today is to incorporate more analog circuits into the microcontroller along with digital transistor circuits, thereby providing a subcategory of microcontroller units (MCUs) called system-on-chip (SOCs). A capability found in many common SOC integrated ICs is an integrated 2.4GHz radio that can be programmed to send or receive data. Unfortunately, when these ICs are designed to use Wi-Fi protocols, they are very difficult to use in battery-powered devices due to their high current requirements. Even in their lowest power operating modes of "sleep mode" or "idle mode", these MCUs consume too much power and do not allow battery-powered devices to operate for a long time. Another trend in the development of ICs today is microelectromechanical systems (MEMS) ICs. Today, several MEMS inertial measurement devices (IMUs) integrate digital transistor logic circuits that can be used to analyze analog inertial measurement values ​​without the help of an MCU. Certain aspects of the disclosed technology can be implemented in an embodiment that uses an extremely low power operating mode of the IMU's internal logic circuits to act as a power-on watchdog that guides the MCU when necessary. The MCU can be completely powered off; for example, the ESP8266 from Espressif can be completely shut down by, for example, pulling the chip enable pin down to ground to completely shut down the chip. The static power supply to the IMU can remain on, with its processing clock now set to the lowest rate (~25Hz). When there is a jerk (acceleration of acceleration), an interrupt can be generated from the IMU, and the chip enable pin on the MCU can be set high to allow it to power on. After running its instructions, the MCU powers itself off again until the next time the IMU wakes it up. In addition to other methods disclosed in the present disclosure, this method can be used to greatly extend the battery life of the device.

[0154] Example latch for determining state

[0155] The challenge of using hardware to monitor the environment without actively powered logic control is to deal with the lack of continuous state information. In some embodiments of the disclosed technology, a component of the monitoring system contains a button (capacitive, touch screen, mechanical, etc.) that allows the user to press to wake up any component device of the monitoring system. Any or all components of the monitoring system may include an IMU to detect environmental motion to wake up the component device. However, this presents a problem to be solved when there is no active MCU. When the user presses a button on the device, this suddenly causes motion. The challenge is to distinguish the action that triggers the MCU to power on: is the motion simply detected? Or is it the motion caused by pressing the button? According to one embodiment, by implementing a resettable latch that is set by a pressed button, the resettable latch can be identified by the MCU once powered on, and then can be reset or reset by the MCU before powering off. For example, using ESP8285 as our MCU, and using an IMU similar to Bosch BMA400 or ST LIS2DW12, a latch using a simple JK flip-flop can be set, that is, set by pressing a button, read by the MCU at power-on, and then reset before powering off. This process, in addition to other processes disclosed in this disclosure, can be used to significantly reduce the cost of materials required to achieve a stateful response.

[0156] Predefined Conference Bridge Example

[0157] Modern commercial or home monitoring or commercial and home security devices are typically designed to allow individual users to see and respond to events that occur in a building or home. Some of these devices are designed to allow multiple users to see events simultaneously, but in all cases, there is a single user in control during the response period. When dealing with a group of peers that are monitoring and responding simultaneously, handling voice communications between peers is often problematic. One of the features of the disclosed technology includes using full-duplex VOIP phone capabilities incorporated into at least a hub device of the monitoring system. The hub device is typically installed in the facility being monitored. During the user setup process, a virtual conference bridge on a cloud-based PBX is created that is unique to the hub device and shared within the peer group. According to an example embodiment, when a user registers to become part of a peer group that will monitor a home or building where the hub device has been installed, the connection address and encryption keys and certificates are shared with the new user joining the group. All peer users running applications on their mobile phones and all devices containing our VOIP technology installed in the home are able to send and receive command signals sent via standard IP protocol signaling when joining a pre-established conference bridge. In one example embodiment, an ESP32 MCU from Espressif and a ZLS38063 DSP from Micro Semi, among others, can be incorporated into a hub device to implement a hands-free full-duplex phone call using a hub device. The ESP32 connects to a public network using a Wi-Fi connection and makes a secure and encrypted connection with a message broker running on a remote server. This message broker connects all peers that remotely monitor the hub device. When any remote peer wishes to establish a voice call to the hub device or with other users in the peer group, they send a signal via the message broker to signal the recipient to connect to the already established conference bridge. MQTT can be used as a message transport, for example, on the hub device, and AWS IoT MQTT can be used as our message broker. The connection can be securely implemented as, for example, a Transport Layer Security (TLS) encrypted IP connection.

[0158] This solves many problems, some of which include:

[0159] 1. None of the participants in the call need to dial into a dynamically changing number.

[0160] 2. All participants can choose to join or opt out of the call.

[0161] 3. Participants can connect automatically (hands-free).

[0162] 4. By eliminating dynamic circuit switching requirements and processing conference calls in an atomic and discreet manner that allows conference groups to be distributed across multiple independent servers, costs are greatly reduced and scaling, balancing, privacy capabilities, and security capabilities are simplified.

[0163] Localized device configuration example

[0164] Devices sold today that are designed to automate tasks in the home suffer from the requirement that they be individually provisioned by the user to connect to a local wireless network in the home. Devices made using common 2.4GHz Wi-Fi radio technology must have their security credentials, including the ID and access password of their local wireless access point, individually shared one at a time with each new device added to the home. Additionally, these devices typically join the wireless network in a random and unbalanced manner, thereby generating network bottlenecks and often resulting in poor network performance for all devices connected to that local wireless network.

[0165] Component devices in a monitoring system according to the disclosed technology can solve this and other problems by using out-of-band radio signaling between new devices to securely share credentials with minimal user intervention and provide deterministic load balancing of devices connected to a wireless network. For example, several modern SOC ICs containing 2.4GHz radios can support a connectionless transmission mode that allows UDP formatted packets to be sent to devices without an established Wi-Fi connection over a short distance. In this way, a central controller (e.g., a hub device or a satellite device) can be established similar to a domain name server (DNS) on a first device installed by a customer and connected to their home wireless network. In an exemplary method, during the installation, the customer enters network security credentials through an encrypted secure wireless connection with an application running on his mobile phone. Then, when a new device in the house is powered on, a signal is broadcast from the new device on a universal address that is always available as an open receive channel. The central controller monitors the open receive channel and, when a new device transmitting wirelessly on the channel is detected, a request is sent to the mobile phone over a secure connection, requesting confirmation and permission to connect the new device. If permission is granted, the central controller performs a series of tests on the new device by sending wireless signals back and forth to determine wireless network quality characteristics and uses intelligent algorithms that compare packet loss, signal strength, and other parameters to determine the best solution to balance the wireless network load with all other devices in the network. The central controller then sends credentials to the new device so that the new device can establish and maintain a secure connection to the network.

[0166] Examples of unconnected and connected devices

[0167] A common way to operate a radio device for data transmission at low power is to reduce the number of transmissions required to establish a connection "channel" between devices. Typically, before a device can transmit data, a series of small signals are sent between the device and the host device on multiple different sub-frequencies, and signal testing is performed to establish the best sub-frequency for sending data. This ideal sub-frequency for transmission is called a "channel." Common current wireless technologies in homes are 2.4GHz or 5GHz Wi-Fi. Radio transmissions using 2.4GHz frequencies require much lower power than radio transmissions that communicate using higher 5GHz frequencies. However, radio signals sent on 2.4GHz frequencies have far fewer available sub-channels than radio signals sent on 5GHz frequencies. One way to reduce the power of devices using wireless radio technology is to cache channel connection information to reduce the number of transmissions required to establish an ideal channel to be used, and also reduce the number of devices that require two-way transceiver capabilities for devices that may only require one-way transmission or reception capabilities.

[0168] According to some embodiments of the disclosed technology, component devices in the monitoring system can solve this problem by using two types of devices: those with radio transceivers that have a larger power supply; and those radio transmitters that have only less powered radio transmitters. The component devices so classified in the network topology can be further optimized to form a balanced network topology, in which lower power devices are connected to nodes in the network in an extended star topology. The component devices of the monitoring system of the present disclosure can dynamically assign low-power only transmission devices to be connected to devices equipped with local transceivers that act as host nodes and have much greater available power. The connection between the low-power device and the high-power device is cached, and when data needs to be transmitted, the transmitter is freed from the unnecessary step of establishing a connection channel. Data is sent with the minimum required amount of power. The component devices in the monitoring system of the present disclosure can be distributed throughout the area as multiple nodes, each of which has access to high power and transceiver radio capabilities, and each of which can establish a low-power dynamic connection to communicate with these nodes in a balanced network configuration. In some embodiments, some component devices in the monitoring system of the present disclosure may be low-power transmit-only devices utilizing SOCs such as ESP8266. Component devices with higher-power transceiver-equipped devices in the monitoring system of the present disclosure may incorporate both an ESP32 connected to a local Wi-Fi network and an ESP8266 in a receive-only state to communicate with low-power devices. This allows high-power nodes to use the capabilities of the Wi-Fi protocol to dynamically adjust their network connections to maximize throughput and maintain robust connections, while also allowing us to use cached channel RF protocols to maintain low-power connections.

[0169] PIR CPLD Example

[0170] Passive infrared motion sensors (PIR sensors) typically operate in a binary manner - setting an electrical signal high to indicate that a warm body in motion is detected nearby, or low to indicate that no motion is detected. In order to suppress the generation of excessive signals sent from these devices, there is usually a timed latch capability so that the circuit can clamp the high signal for a fixed amount of time before releasing the clamp and resetting the circuit to detect motion again. Some PIR sensors contain logic ICs to allow the time value used for the clamp circuit to be adjusted by control logic running on the host microcontroller, or to adjust the set point temperature used to determine whether a warm body in motion is detected. In both PIR circuits, the amount of information available to the host microcontroller is too limited to detect a wide range of parameters that may be useful for the application. In some embodiments, the component devices in the monitoring system of the present disclosure can integrate low-cost PIR sensors with complex programmable logic devices (CPLDs) and time-of-flight distance sensors to allow deterministic detection of a wide range of parameters while maintaining very low cost and a very small number of components. The PIR sensor connected to the CPLD IC can be programmed to allow recording of complex waveforms broken down into fixed time intervals. Incorporating a time-of-flight distance sensor provides the additional fidelity needed to detect complex motion activity using inexpensive components. The component devices in the disclosed monitoring system can detect with high fidelity the relative size of objects in motion, direction and rate of travel, and detect when the sensor is blocked by an inanimate object. The parameters extracted from the circuit are robust enough to be input into a random table generated by a machine learning process (probabilistic signal determination) or run through a rule-based process (deterministic signal determination).

[0171] Example array for determining the PID function of ADL

[0172] A challenge in the area of ​​monitoring activities of elderly people living alone is to determine their activities of daily living (ADL). For example: bathing, eating, sleeping, exercising. Current solutions involve incorporating multiple cameras that watch the elderly and stream the video to a remote server that performs visual analysis, or by integrating complex and expensive sensors in the home environment to measure and record activities. In some embodiments of the disclosed technology, the monitoring system can use a mechanism to deterministically identify activities using common low-cost sensors, where the fidelity of the signal can be easily increased simply by adding more low-cost sensors. The component devices in such a monitoring system can provide a deterministic solution rather than just a probabilistic solution.

[0173] In an exemplary embodiment, the monitoring system may include a microcontroller operating as a master control device. The component devices may be wirelessly connected to any number of smaller wireless devices, each of which contains a minimum of one sensor. The output of each sensor is reported from the smaller device to the master control device for analysis. The master control device (e.g., a hub device) contains a memory storage device that is divided into rows and columns. Each sensor output is recorded into a cell in the memory based on the time it occurred. Arbitrarily, we can choose each row to represent a different sensor, and each column to represent a different hour within a 30-day period. Each sensor in the platform is configured with a threshold. When the sensor input exceeds the threshold, a signal with the amplitude of the detected signal is sent from the device to the master control device. The master control device runs a filtering algorithm on the received sensor data (PID function works well, but it can be a variety of different algorithms) and then stores the filtered data in its partitioned memory based on the sensor id and the current time.

[0174] The partitioned memory then represents a heat map of the categories, which can show the times and sensor readings expected to be generated at any given moment. Thus, ADL can be measured deterministically, and deviations from regular activities can be detected - activities that occur when they do not normally occur or activities that do not occur when they normally do occur.

[0175] Example hierarchical organization of cheap, simple sensors for adjudicating triggers on complex behaviors

[0176] In some embodiments of the present technology, by layering sensors based on their location information, a master control device (e.g., a hub device) can arbitrate the generation of triggers based on complex behaviors that are not discernible by any individual sensor. In addition, the fidelity of the arbitrating of these triggers can be increased at any time by adding additional inexpensive sensors.

[0177] An image sensor combined with a time-of-flight range sensor to approximate Z for object classification.

[0178] Passive infrared motion detection is very popular due to its low cost. More sophisticated motion detection is possible using imaging sensors; however, their increased cost is generally not suitable for low-cost passive infrared unless the application requires video to be recorded or streamed remotely. The capabilities of low-cost PIRs are generally good enough for local processing of motion detection, and the added capabilities of image sensors are not utilized unless the device is recording video.

[0179] In some embodiments of the present technology, by incorporating a time-of-flight distance sensor, a much more robust motion detection circuit can be created that can be used for local processing even when video recording is not used. A standard image sensor captures light that reaches the sensor through a viewing cone. The data captured from the sensor can be processed by a microcontroller to determine objects in motion and determine the range of pixels that have changed; this method produces results that are not too dissimilar to what a PIR can detect, although more expensive. By incorporating a time-of-flight distance sensor along with an imaging sensor, we can get a good estimate of object size. Combined with other data, we can detect size, volume, travel speed, orientation, and count the number of independent targets moving within the viewing cone. Image sensors (e.g., hub devices) and passive infrared sensors (e.g., satellite devices) can be used interchangeably, for example, to obtain the X and Y components of motion within a space. The extracted X and Y component information can then be combined with a time-of-flight sensor to obtain the Z component, thereby obtaining a good estimate of object size and motion in space, a process similar to light detection and ranging (LIDAR).

[0180] Additionally, in some embodiments, various camera types (e.g., near infrared light sources and single photoelectric sensors) may be used to detect the progression of microvascular damage prior to the development of a decubitus ulcer. In some embodiments, a macro camera may be implemented in one of the sensor assemblies (e.g., a sensor device or cellular pendant) to enable a patient or caregiver to immediately photograph a skin or other visually apparent developing condition and communicate it to a healthcare provider (e.g., via an HMI interface of a mobile device) to determine appropriate medication or activities for treatment or care. Additionally, the hub device and sensor assembly may be configured to monitor certain related activities for treatment or care, or to notify a caregiver (user) of appropriate over-the-counter medications or purchase of comfort care items.

[0181] Example classifier on video and / or audio (edge ​​computing)

[0182] One of the challenges of running an audio or video ML classifier on an embedded microcontroller rather than processing the information on a remote server where they receive audio and video data via a stream is that the computing power on the microcontroller is insufficient compared to the computing power on the remote server. One improvement to this problem is to allow the ML classifier to perform earlier trivial data rejection and reduce the processing required to run the audio and video classifier.

[0183] The classifiers of the present disclosure may be binary state machines that are compiled on a computer with large processing power and access to large amounts of memory, which analyzes large and diverse data sets to reduce the state machine to a minimum size that is likely to detect one of the data sets. These binary classifiers are then installed on a microcontroller with limited processing power and access to much less memory.

[0184] A problem arises when the initial dataset is very diverse. The more diverse the original dataset, the larger the compiled classifier will be. One possible optimization of this process is to dynamically swap the binary classifier running on the device to match only the dataset represented in that environment.

[0185] In some embodiments, component devices in the monitoring system of the present disclosure can solve this and other problems by installing classifiers on devices that are generic and large when they are first attached to a wireless network. Using other audio and video sources captured using a client mobile device, an over-the-air (OTA) update to the microcontroller firmware can be used to select and install an optimized classifier on the wireless device.

[0186] In various embodiments of the present disclosure, the component devices may include a number of inexpensive commodity hardware that can be used to facilitate comprehensive spatial and environmental information collection. Instead of providing possible outcomes or using learning algorithms, artificial intelligence, and / or machine learning to analyze behavior, this inexpensive commodity hardware can be distributed throughout a building to consistently monitor one or more aspects of individual or patient activity, thereby providing deterministic analysis. Thus, the care provider (user) experience is improved because the care provider (user) has peace of mind knowing that events and activities that support the patient's health have occurred. In addition, the care provider (user) experience is improved because the care provider (user) has the information necessary to plan daily changes in patient activity to improve the health of the monitored patient (e.g., the amount of walking during the day as determined by the component sensors).

[0187] Listed below are some examples of the embodiments disclosed herein.

[0188] 1. A monitoring device comprising: at least one memory; at least one button; one or more light sources; and a processor, the processor being coupled to the at least one memory and communicatively coupled to at least one inertial measurement unit; wherein the processor is configured to adjust an operating mode of the monitoring device based on a state of the at least one inertial measurement unit; wherein the processor is communicatively coupled to one or more sensors and configured to classify one or more activities transmitted from the one or more sensors.

[0189] 2) A monitoring device according to Example 1, wherein one of the one or more sensors includes a time-of-flight sensor.

[0190] 3) A monitoring device according to any one of Examples 1 or 2, wherein when the at least one inertial measurement unit receives a first input, the state of the at least one inertial measurement unit enters a first state, and the operating mode of the monitoring device is set to a low-power operating mode.

[0191] 4) A monitoring device according to Example 3, wherein pressing the at least one button transmits the first input to the inertial measurement unit.

[0192] 5) A monitoring device according to any one of Examples 1 to 4, wherein the processor is configured to adjust an operating mode of the monitoring device based on one or more activities transmitted from the one or more sensors.

[0193] 6) A monitoring device according to any of Examples 1 to 5, wherein one of the one or more sensors includes a microphone communicatively coupled to the processor, and wherein the processor is further configured to initiate a Voice over Internet Protocol (VOIP) session when an audio signature from the microphone is detected.

[0194] 7) A monitoring device according to any of Examples 1 to 6, wherein the processor is configured to classify the activity based on the activity being transmitted as occurring in multiple sensors among the one or more sensors.

[0195] 8) A monitoring system implemented as described in this patent document.

[0196] 9) A method of monitoring a building or fence according to any of the methods described in this patent document.

[0197] Those skilled in the art will appreciate that some of the aforementioned disclosed systems and functionalities may be designed and configured as computer files (e.g., RTL, GDSII, GERBER, etc.) stored on computer readable media. Some or all of such files may be provided to manufacturing processing personnel who manufacture equipment based on such files. The resulting product includes semiconductor wafers separated into semiconductor dies and packaged into semiconductor chips. The semiconductor chips are then employed in devices such as IoT devices, monitoring systems, or any combination thereof.

[0198] Those skilled in the art will further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0199] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0200] In addition, specific details are given in the description to provide a thorough understanding of the embodiments. However, embodiments can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures and techniques are shown without unnecessary details to avoid blurring the embodiments. This description only provides example embodiments and is not intended to limit the scope, applicability or configuration of the present invention. On the contrary, the foregoing description of the embodiments will provide enabling descriptions for implementing embodiments of the present invention for those skilled in the art. Various changes may be made to the functions and arrangements of elements without departing from the spirit and scope of the present invention.

[0201] Where applicable, the various embodiments provided by the present disclosure may be implemented using hardware, software, or a combination of hardware and software. In addition, where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware, or both. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0202] Software or applications (such as program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems, networked, and / or otherwise. Where applicable, the order of the various steps described herein may be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0203] It should be noted that the above methods describe possible implementations, and that operations and steps may be rearranged or otherwise modified, and other implementations are possible. Furthermore, embodiments from two or more methods may be combined.

[0204] In light of the foregoing, it should be understood that specific embodiments of the present disclosure have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of the present disclosure. On the contrary, in the foregoing description, many specific details have been discussed to provide a thorough and enabling description of the embodiments of the present disclosure. However, those skilled in the relevant art will recognize that the present disclosure may be practiced without one or more specific details. In other examples, well-known structures or operations typically associated with memory systems and devices are not shown or described in detail to avoid confusing other aspects of the present invention. Generally, it should be understood that various other devices, systems, and methods other than those specific embodiments disclosed herein may be within the scope of the present disclosure.

Claims

1. A monitoring system comprising: a plurality of sensor devices, each sensor device configured to collect sensor data from a respective environment, wherein the plurality of sensor devices are dispersed across a plurality of locations within a region; one or more satellite devices, each satellite device configured to facilitate wireless communications between the plurality of sensor devices and a hub device of the monitoring system; as well as The hub device is in communication with the one or more satellite devices, wherein the hub device is configured to: dynamically adding a new sensor device to the plurality of sensor devices; receiving sensor data originating from the plurality of sensor devices from the one or more satellite devices; generating an activity record for the area based on classifying the activity information captured by the sensor data; as well as In response to determining that the occurrence or non-occurrence of a particular activity deviates from the activity record in the particular sensor data, an alert notification is sent to the user device.

2. The monitoring system of claim 1, wherein the one or more satellite devices are each configured to communicate with the plurality of sensor devices via a first communication protocol and to communicate with the hub device via a second communication protocol. 3 . The monitoring system of claim 1 , wherein dynamically adding a new sensor device to the plurality of sensor devices comprises building a whitelist identifying each of the plurality of sensor devices.

4. The monitoring system of claim 1, wherein the activity record includes a heat map identifying the location and frequency of activity occurrence.

5. The monitoring system according to claim 1, wherein: The plurality of sensor devices includes a pendant device carried by an individual and configured for cellular network communications.

6. The monitoring system according to claim 1, wherein: The activity record is specific to an individual monitored by the monitoring system, and wherein the user device to which the alarm notification is sent is associated with a caregiver who is in charge of the individual and is away from the area.

7. The monitoring system according to claim 1, wherein: The plurality of sensor devices include a specific sensor device, and the specific sensor device includes: a wireless controller, the particular sensor device communicating with the one or more satellite devices via the wireless controller; a microcontroller unit configured to execute instructions to operate the wireless controller; and A micro-electromechanical inertial measurement unit (IMU) is coupled to the micro-controller unit, wherein the micro-electromechanical IMU is configured to provide a digital input to the micro-controller unit in response to movement achieved on the micro-electromechanical IMU to change the state of the micro-controller unit to a powered-on state.

8. The monitoring system according to claim 7, wherein: The micro-electromechanical IMU is coupled to a chip enable pin of the micro-controller unit, the chip enable pin being configured to keep the micro-controller unit in a powered-off state until the micro-electromechanical IMU provides the digital input to change the state of the micro-controller unit.

9. The monitoring system according to claim 1, wherein: The hub device is configured to dynamically add the new sensor device in response to the new sensor device broadcasting a signal on an open receive channel monitored by the hub device and further in response to receiving an acknowledgement of a query sent by the hub device to the user device, the query identifying the new sensor device.

10. A monitoring system according to claim 1, wherein the hub device is configured to generate the activity record in a memory storage device divided into rows and columns, wherein each row of the memory storage device corresponds to a different sensor device from which the hub device receives sensor data, and wherein each column of the memory storage device corresponds to a different time period during which the hub device receives sensor data from the multiple sensor devices.

11. The monitoring system according to claim 1, wherein: The plurality of sensor devices includes a time-of-flight distance sensor paired with an imaging sensor, wherein the time-of-flight distance sensor is adapted to contribute object size information to image information captured by the imaging sensor.

12. The monitoring system according to claim 1, wherein: The plurality of sensor devices include at least three of the following types of sensor devices: a temperature sensor, an ambient light sensor, a humidity sensor, a barometer sensor, an air quality sensor, an infrared sensor, a carbon dioxide sensor, a carbon monoxide sensor, a piezoelectric sensor, or an airflow sensor.

13. The monitoring system according to claim 1, wherein: The activity information includes both environmental information of the area and information about one or more individuals present in the area.

14. The monitoring system according to claim 13, wherein: The environmental information includes one of temperature, humidity, sound, or air flow, and the information about the one or more individuals includes a medical condition or an amount of daily activity.

15. A monitoring system comprising: one or more sensor devices, each sensor device configured to collect sensor data from a corresponding environment, wherein the one or more sensor devices are located within the structure; as well as a hub device located within the structure and configured to: dynamically including a new sensor device among the one or more sensor devices; obtaining sensor data from the one or more sensor devices; generating a heat map of one or more areas within the structure based on classifying the activity information captured by the sensor data, wherein the heat map identifies a type of activity, a location of the activity, and the sensor devices involved in detecting the activity; using the heat map as a reference to determine whether the occurrence or non-occurrence of a particular activity deviates from the heat map activity; as well as In response to determining that a deviation has occurred, a notification is generated indicating the deviation.

16. The monitoring system according to claim 15, wherein: The one or more sensor devices are included within the hub device.

17. A hub device of a monitoring system, the monitoring system comprising a plurality of sensor devices distributed across an area, the hub device comprising a processor and a memory, the memory having instructions stored therein, wherein the instructions, when executed by the processor, configure the processor to: dynamically maintaining a network whitelist identifying the plurality of sensor devices; receiving sensor data originating from the plurality of sensor devices from one or more satellite devices; generating an activity record for the area based on classifying the activity information captured by the sensor data; as well as In response to determining that the occurrence or non-occurrence of a particular activity deviates from the activity record in the particular sensor data, an alert notification is sent to the user device.

18. The hub device of claim 17, wherein dynamically maintaining the network whitelist comprises: detecting a broadcast signal on an open receiving channel, the broadcast signal originating from a new sensor device not identified by the network whitelist; performing a series of network performance tests using the new sensor device; as well as The network whitelist is updated to additionally identify the new sensor device.

19. The hub device according to claim 17, wherein: The plurality of sensor devices includes a pendant device carried by an individual and configured for cellular network communications.

20. The hub device according to claim 17, wherein: The activity record is specific to an individual, and wherein the user device to which the alert notification is sent is associated with a caregiver who is in charge of the individual and is away from the area.

21. The hub device of claim 17, wherein the hub device comprises: a wireless controller, the hub device communicating with the one or more satellite devices via the wireless controller; a microcontroller unit, the microcontroller unit comprising the processor; as well as A micro-electromechanical inertial measurement unit (IMU) is coupled to the micro-controller unit, wherein the micro-electromechanical IMU is configured to change a state of the micro-controller unit to a powered-on state in response to movement effected on the micro-electromechanical IMU.

22. A hub device according to claim 17, wherein the activity record is generated in the memory of the hub device, and the memory for the activity record is divided into rows and columns, wherein each row of the memory for the activity record corresponds to a different sensor device from which the sensor data originates, and wherein each column of the memory for the activity record corresponds to a different time period in which the hub device receives sensor data originating from the multiple sensor devices.

23. A method for a monitoring system for an area, the method comprising: collecting, by a processor, activity information of individuals in the area via a plurality of sensor devices of the monitoring system distributed across the area; classifying, by the processor, the activity information based on comparing the activity information with historical activity information stored in a database; generating, by the processor, an activity heat map based on updating the number of activity occurrences according to the classified activity information; as well as In response to detecting a non-occurrence of a normally occurring activity identified by the activity heat map, an alert notification is provided, by the processor, to a user device associated with the individual.

24. The method of claim 23, wherein collecting the activity information comprises: capturing, by the plurality of sensor devices, sensor data including the activity information of the individuals within the area; receiving, by a satellite device of the monitoring system, sensor data including the activity information from the plurality of sensor devices; as well as The activity information is provided to the processor by the satellite device.

25. A method according to claim 23, wherein the activity heat map is generated in a memory storage device, the memory storage device is divided into rows and columns, wherein each row of the memory storage device corresponds to a different sensor device from which the activity information originates, and wherein each column of the memory storage device corresponds to a different time period associated with the activity information.

26. The method of claim 23, further comprising: Placement information indicating where to place the plurality of sensor devices within the area is provided to a user by the processor based on the activity heat map.

27. The method of claim 23, further comprising: dynamically including a new sensor device in the plurality of sensor devices based on a response to the new sensor device broadcasting a signal on the monitored open receive channel; as well as Activity information of the individual is collected from the new sensor device.

28. The method of claim 23, wherein: The activity information includes at least one of temperature, humidity, sound, or air flow of the area, and individual information including a medical condition or a daily activity amount of the individual.

29. The method of claim 23, wherein detecting the non-occurrence of the normally occurring activity is performed using a deterministic analysis based on the activity heat map and detections obtained from one or more of the plurality of sensor devices.

Images