System and method for analyzing sleep behavior

Abstract

A sleep application receives initial sensor data from one or more sensors of a sensor set within a physical environment during a certain time period, and the sensor set includes at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor. A sleep application behavior pattern of a set of sleep events of a target subject based on the initial sensor data. The sleep application generates recommendations based on the behavior pattern to achieve the target result of the target subject within the physical environment. The sleep application provides the recommendations.

Description

Technical Field

[0001] Cross - Reference to Related Patent Applications This application claims the benefit of priority of U.S. Patent Application No. 17 / 750,142, filed on May 20, 2022, entitled SYSTEM AND METHOD FOR ANALYZING SLEEPING BEHAVIOR, the entire content of which is incorporated herein by reference in its entirety.

Background Art

[0002] Commercial baby monitors capture images and sounds of babies and provide video and / or audio feeds to parents.

[0003] Some sensors are incorporated into wearables and attempt to capture more data, including a baby's movements, heart rate, oxygen levels, etc. These wearables have limitations and challenges because they need to be worn either on the baby or on the baby's clothing. Wearables can cause discomfort, be easily removed, and often collect data that is unrelated to the baby's behavior.

[0004] The description of the background art provided herein is for the purpose of presenting the context of the present disclosure broadly. The research of the inventors whose names are listed in the context of this background art section, and aspects of this specification that would not normally be considered prior art at the time of filing, are not admitted as prior art to the present disclosure, either expressly or implicitly.

Summary of the Invention

Means for Solving the Problems

[0005] A method implemented by a computer includes receiving, during a time period, initial sensor data from one or more sensors of a set of sensors in a physical environment, the set of sensors including at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, a camera, and a motion sensor. The method further includes determining a behavioral pattern of a set of sleep events of a target subject based on the initial sensor data. The method further includes generating recommendations based on the behavioral pattern to achieve a target outcome of the target subject in the physical environment. The method further includes providing the recommendations.

[0006] In some embodiments, the method further includes a step of determining whether the recommendations are followed, and a step of determining whether the target action has been achieved in response to the determination that the recommendations are followed. In some embodiments, the method further includes a step of updating the action pattern based on subsequent sensor data and a step of updating the target result based on the updated action pattern in response to the determination that the target action has been achieved. In some embodiments, the method further includes a step of determining whether the recommendations are followed, and a step of offering a reward if the recommendations are followed thereafter in response to the determination that the recommendations are not followed. In some embodiments, the sensor set includes a thermal imaging sensor, and the method further includes a step of detecting a target subject, one or more persons near the target subject, and one or more objects in the physical environment based on the initial sensor data, a step of determining the distance between the target subject and one or more persons based on the initial sensor data, a step of determining the movement pattern of the target subject in the physical environment, a step of determining one or more movement patterns corresponding to one or more persons, and a step of determining one or more movement patterns of one or more objects, and the step of determining the action pattern is based on the movement pattern of the target subject, one or more movement patterns corresponding to one or more persons, and one or more movement patterns of one or more objects. In some embodiments, the sensor set includes a sound sensor, and the method further includes a step of detecting the sound level in the physical environment based on the initial sensor data, a step of detecting the sound of the target subject, the sound of one or more persons near the target subject, and other sounds in the physical environment based on the initial sensor data, a step of removing specific sounds with a filter, and a step of determining the sound pattern, and the step of determining the action pattern is based on the sound pattern.In some embodiments, the sensor set includes a motion sensor, and the method further includes detecting, based on initial sensor data, the movement of a target subject, one or more persons near the target subject, and one or more objects within a physical environment; and determining a target subject movement pattern, a person movement pattern, and one or more object movement patterns, wherein the step of determining an action pattern is based on the target subject movement pattern, the person movement pattern, and one or more object movement patterns. In some embodiments, the method further includes providing a user interface that requests user preferences regarding a target outcome, wherein the target outcome is defined based on user preferences. In some embodiments, the method further includes receiving subsequent sensor data during a subsequent time period; determining, based on a comparison of the subsequent sensor data and the action pattern, that an action is likely to cause a nighttime awakening or a nap awakening; and providing a warning that an action is likely to cause a nighttime awakening or a nap awakening. In some embodiments, the recommendations are provided to a user different from the target subject, and the method further includes determining an action pattern for a set of the user's sleep events based on initial sensor data. In some embodiments, the method further includes receiving initial sensor data associated with a user that identifies a length of time the user is asleep; and providing the user with a user interface that includes the length of time the user is asleep compared to when the target subject is asleep. In some embodiments, the method further includes providing the initial sensor data as an input to a trained machine learning model; and using the trained machine learning model to output recommendations for achieving a target outcome.

[0007] The embodiment may further include a computing device, the computing device including one or more processors and a memory coupled to the one or more processors, the memory having instructions stored thereon, and the instructions, when executed by the processors, cause the processors to receive initial sensor data from one or more sensors of a set of sensors in a physical environment during a certain time period, where the set of sensors includes at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor, and to determine a baseline for one or more of a set of sleep events of a target subject based on the initial sensor data.

[0008] In some embodiments, the operations further include determining a behavior pattern of a set of sleep events of a target subject based on a baseline and the initial sensor data for one or more of the set of sleep events, generating recommendations based on the behavior pattern to achieve a target outcome of the target subject in the physical environment, providing the recommendations, determining whether the recommendations are followed, and in response to determining that the recommendations are followed, determining whether a target action has been achieved. In some embodiments, the operations further include, in response to determining that the target action has been achieved, updating the behavior pattern based on subsequent sensor data and updating the target outcome based on updating the behavior pattern. In some embodiments, the operations further include determining whether the recommendations are followed and, in response to determining that the recommendations are not followed, subsequently offering a reward if the recommendations are followed.

[0009] An embodiment may further include a non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more computers, cause the one or more computers to receive initial sensor data from one or more sensors of a set of sensors in a physical environment during a time period, wherein the set of sensors includes at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor; determine a behavior pattern of a set of sleep events of a target subject based on the initial sensor data; generate recommendations based on the behavior pattern to achieve a target result of the target subject in the physical environment; and provide the recommendations.

[0010] In some embodiments, the operations further include determining whether the recommendations are followed and, in response to determining that the recommendations are followed, determining whether a target action has been achieved. In some embodiments, the operations further include, in response to determining that the target action has been achieved, updating the behavior pattern based on subsequent sensor data and updating the target result based on the updated behavior pattern.

[0011] This specification advantageously describes a non-image-based system that monitors the behavior of a target subject over a continuous time period based on initial sensor data. The system determines the behavior patterns of a set of sleep events of the target subject based on the initial sensor data. For example, the system can identify a baseline for a set of sleep events, such as the duration of bedtime preparation. The system generates recommendations for achieving the target outcomes of the target subject within the physical environment. For example, the system pays attention to the noise level of activities outside the bedroom and determines the causal relationship with the arousal of the target subject. The system then provides a recommendation to reduce the noise level of activities outside the bedroom. Other sleep systems only use images of sensor data and do not provide detailed recommendations, so they are not sufficiently personalized and do not achieve the target outcomes. Furthermore, due to privacy concerns including medical concerns, and because image-based sensor data requires sufficient light in the room, which may interfere with sleep, image-based sensor data may not be desirable.

[0012] In some embodiments, the behavior pattern is determined using a machine learning model trained with training data including labels for each type of sleep event. This advantageously enables the system to provide insights that would not otherwise be obtainable. In some embodiments, it is determined whether the recommendations were followed and, if so, whether following the recommendations was successful in achieving the target outcomes. The system can receive subsequent sensor data and generate updated behavior patterns as well as updated target outcomes. As a result of receiving this feedback and modifying the behavior pattern accordingly, the system provides support over a continuous period to achieve the current age and target outcomes for all ages. Finally, in addition to being useful for target subjects of all ages, this specification describes a system that is useful for target subjects with various health conditions, including those who require special assistance, those who suffer from mental health problems, athletes, and those with medical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 11A

Figure 11B

Figure 12

DETAILED DESCRIPTION OF THE INVENTION

[0014] Network Environment 100 FIG. 1 shows a block diagram of an exemplary network environment 100. In some embodiments, network environment 100 includes a cloud server 101, a user device 115, a sleep hub 120, an activity tracker 127, a night light 130, and a network 105. User 125 may be associated with user device 115. In some embodiments, environment 100 may include other servers or devices not shown in FIG. 1, or may not include other entities shown in FIG. 1 such as activity tracker.

[0015] Cloud server 101 may include a processor, a memory, and network communication hardware. In some embodiments, cloud server 101 is a hardware server. Cloud server 101 is communicatively coupled to the network via a wired connection such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection such as Wi-Fi®, Bluetooth®, or other wireless technology. In some embodiments, cloud server 101 transmits and receives data to and from one or more of user device 115, sleep hub 120, and activity tracker 127 via network 105.

[0016] Cloud server 101 may include a sleep application 103a and a database 199. In FIG. 1 and the remaining figures, the characters following the reference number, e.g., “103a”, represent a reference to the element having that particular reference number. A reference number in text without following characters, e.g., “103”, represents a general reference to embodiments of the element having that reference number.

[0017] The sleep application 103a may receive initial sensor data from the sleep hub 120, the activity tracker 127, and / or the user device 115, determine a behavioral pattern of a set of sleep events of a target subject based on the initial sensor data, generate recommendations based on the behavioral pattern to achieve the target result of the target subject in the physical environment, and include code and routines operable to provide the recommendations. The target result may be provided by the user 125 of the user device 115 or determined based on comparing the behavioral pattern with the expected behavior of a target subject having similar attributes. For example, the user may be a parent, and the parent may specify that if the child wakes up in the middle of the night, the child can return to sleep on their own, and as a result, the sleep application defines the target result as a child who returns to sleep on their own. In another example, the sleep application 103a may determine that the target subject is getting 2 hours less sleep than the expected amount of sleep of a 53-year-old male by comparing the behavioral pattern with the expected behavior of a 53-year-old male, and as a result, define the target result as getting 2 more hours of sleep every night.

[0018] In some embodiments, the sleep application 103a may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any other type of processor, or a combination thereof. In some embodiments, the sleep application 103a may be implemented using a combination of hardware and software.

[0019] The database 199 can store information associated with people. For example, the database 199 can store both the initial sensor data and subsequent sensor data, behavioral patterns, user preferences, and the like.

[0020] The sleep hub 120 may be a computing device including a memory, a hardware processor, a sensor set, a speaker, and an input / output (I / O) interface. The components of the sleep hub 120 will be described in more detail below with reference to FIG. 2. The sensors generate sensor data and transmit the sensor activities to the sleep application 103 stored on the cloud server 101 and / or the user device 115. For example, the sensors can detect activities inside the room where a person is sleeping, changes in light in the room, changes in humidity in the room, and the like. In some embodiments, multiple rooms each include their own sleep hub 120 and the sensor data is coordinated. For example, generally speaking, if a person cannot sleep, the bed should only be associated with sleep, so the person should not be in the bed. As a result, the hub in the bedroom may detect the actions of the person trying to sleep, and the hub in the living room may detect a person trying to read when they cannot sleep.

[0021] In some embodiments, the speaker generates white noise or voices associated with a sleep assistant to assist a person's sleep-related activities. The white noise may use sound attenuation techniques to determine the volume based on factors such as the size of the physical environment, the proximity of the product from the bed, the amount of sound-absorbing fabric in the room, and the like. In some embodiments, the I / O interface is operable to give commands to the speaker. For example, the user may use the sleep application 103b on the user device 115 to activate a white noise machine or another sleep assistant function. The sleep application 103b on the user device 115 can transmit commands to the sleep hub 120 to activate the speaker and execute commands to generate white noise or sleep assistant prompts. In some embodiments, the sleep application 103b includes recorded voices provided by the user, such as a lullaby sung by a grandparent or a story read by a parent.

[0022] The Night Light 130 may be a computing device that includes a memory, a hardware processor, and light. In some embodiments, the Night Light 130 receives a notification from the Sleep Hub 120 that someone has been detected in the room by the Sleep Hub 120. In response, the Night Light 130 can switch from dark mode to emitting light so that a person entering the room can find the Night Light 130. In some embodiments, the Night Light 130 can switch from dark mode to emitting light when a person picks up the Night Light 130 and / or removes the Night Light 130 from the crib.

[0023] The light may be designed to be bright enough for a parent to use the light to change a diaper, but not so bright as to interfere with the baby's ability to sleep. In some embodiments, the Night Light 130 is portable. In some embodiments, when the child grows older, the Night Light 130 may be configured to emit a soft, non-stimulating light. The soft, non-stimulating light may be used, for example, to assist with the child's nighttime toilet training.

[0024] The Activity Tracker 127 may be a computing device that includes a memory, a hardware processor, a display screen, and sensors. For example, the Activity Tracker 127 may have a built-in Global Positioning System (GPS), altimeter, heart rate monitor, etc. that are operable to track activities such as running, cardiovascular activity, hiking, cycling, etc. In some embodiments, the Activity Tracker 127 uses motion sensors to detect when the user is asleep. The Activity Tracker 127 is connected to the Network 105 and transmits sensor data to the Sleep Application 103.

[0025] The user device 115 may be a computing device including a memory and a hardware processor. For example, the user device 115 may include a desktop computer, a mobile device, a tablet computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing the network 105.

[0026] In the illustrated implementation, the user device 115a is coupled to the network 105 by a wired connection such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection such as Wi-Fi (registered trademark), Bluetooth (registered trademark). The sleep application 103 may be stored as the sleep application 103b on the user device 115. Only one user device 115 is shown, but in some embodiments, the user devices 115a and 115n are part of the network environment 100. For example, the sleep hub 120 may be used to analyze the baby's behavior pattern, the user device 115a is used by the first parent (user 125a), and the user device 115b is used by the second parent (user 125b).

[0027] In some embodiments, the sleep application 103b receives sensor data from other applications on the user device 115. For example, the user may use a meditation application to try to achieve a more peaceful sleep and get it, or to fall back asleep after waking up. The sleep application 103b integrates sensor data from other applications on the user device 115.

[0028] Sleep hub 120 FIG. 2 is a block diagram of an exemplary flow of sensor data from the physical environment 205 that is detected by the sleep hub 120 and transmitted to the computing device 300 for analysis. In some embodiments, the physical environment 205 includes the target subject 210 and a person 215 who is near the target subject. For example, the physical environment 205 may be a home, the target subject 210 may be a baby (infant, toddler, child, teenager, etc.), and the person 215 who is near the target subject may be a parent who is in the room with the baby, or a person outside the room who is noisy enough that their actions are detected as sensor data. In another embodiment, the physical environment 205 may be an apartment building, the target subject 210 may be an adult with insomnia, and the person 215 who is near the target subject may be a person in the apartment next to the adult's apartment.

[0029] The sleep hub 120 may include an I / O interface 224 and a sensor set having one or more of a temperature sensor 225, a pressure sensor 230, a humidity sensor 235, a light sensor 240, a sound sensor 245, a motion sensor 250, a thermal imaging sensor 255, a camera 260, and a speaker 265.

[0030] The I / O interface 224 can provide a function that enables interfacing between the sleep hub 120 and the computing device 300 and also interfacing with components within the sleep hub 120. For example, the I / O interface 224 receives sensor data from any of the sensors in the sensor set and transmits the sensor data to the computing device 300. In another example, the I / O interface 224 receives an instruction from the computing device 300, such as to activate the speaker 265 to play white noise, and the I / O interface 224 executes that instruction.

[0031] The temperature sensor 225 includes hardware for detecting the temperature within the physical environment 205. The pressure sensor 230 includes hardware for detecting the pressure within the physical environment 205 or a specific pressure within a specific location. For example, the pressure sensor 230 may be under a baby's crib and is used to detect a change in pressure that would indicate the baby has left the crib. The humidity sensor 235 includes hardware for detecting the humidity within the physical environment 205. The light sensor 240 includes hardware for detecting the light level within the physical environment 205. The sound sensor 245 includes hardware for detecting the sound level within the physical environment 205 and converting the sound level to decibels. In some embodiments, the sound sensor 245 includes a microphone. In some embodiments, sound is recorded and stored as an audio file (user consent is required to store the audio file).

[0032] The motion sensor 250 includes hardware for detecting motion within the physical environment 205. For example, the motion sensor 250 may include one or more of a passive infrared sensor that detects the body's head by looking for temperature changes, a microwave sensor that sends out microwave pulses and measures the reflection from a moving object, an area reflection sensor that emits infrared light from a light-emitting diode (LED) and uses the reflection of that light beam to measure the distance to a target person or object, an ultrasonic motion sensor that measures the reflection from a moving object via ultrasonic pulses, and a vibration motion sensor that detects the small vibrations caused when a person moves through a room.

[0033] The thermal imaging sensor 255 includes hardware for detecting people and objects within the physical environment using infrared (IR) radiation. For example, the thermal imaging sensor 255 may include an IR camera that uses IR radiation or thermal imaging to track the location of people and objects in a room.

[0034] Camera 260 includes hardware for capturing images and / or video of the physical environment 205. In some embodiments, camera 260 receives, via I / O interface 224, a detection of motion from motion sensor 250 that a person has been detected in the room, and in response to the detected motion, captures images and / or video of the physical environment 205. In some embodiments, camera 260 captures images and / or video of the physical environment 250 when the light level in the room exceeds a threshold amount.

[0035] Example of computing device 300 FIG. 3 is a block diagram of an exemplary computing device 300 that may be used to implement one or more features described herein. Computing device 300 can be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 300 is user device 115 used to implement sleep application 103. In another example, computing device 300 is cloud server 101. In yet another example, sleep application 103 is partially on user device 115 and partially on cloud server 101.

[0036] In some embodiments, computing device 300 includes a processor 335, a memory 337, an I / O interface 339, a display 341, and a storage device 345, all coupled via a bus 318. The processor 335 may be coupled to the bus 318 via signal line 322, the memory 337 may be coupled to the bus 318 via signal line 324, the I / O interface 339 may be coupled to the bus 318 via signal line 326, the display 341 may be coupled to the bus 318 via signal line 328, and the storage device 345 may be coupled to the bus 318 via signal line 330. Depending on the type of computing device 300 to which it is applicable, some components may be added or removed. For example, if the computing device 300 is a cloud server 101, the computing device 300 may not include a display 341.

[0037] The processor 335 includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or some other processor array for performing calculations and providing instructions to a display device. The processor 335 may include various computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture implementing a combination of instruction sets. Although FIG. 2 shows a single processor 335, multiple processors 335 may be included. Other processors, operating systems, sensors, displays, and physical configurations may be part of the computing device 200.

[0038] Memory 337 is typically provided in computing device 300 for access by processor 335, and can be any suitable processor-readable storage medium such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by a processor or a set of processors, and is located remotely from and / or integrated with processor 335. Memory 337 can store software that operates on computing device 300 by processor 335, including sleep application 103.

[0039] Memory 337 stores instructions and / or data that can be executed by processor 335. The instructions may include code for implementing the techniques described herein. Memory 337 may be a dynamic random access memory (DRAM) device, static RAM, or some other memory device. In some implementations, memory 337 also includes non-volatile memory such as a (SRAM) device or flash memory, or a similar permanent storage device and medium including a hard disk drive, floppy disk drive, compact disc read-only memory (CD-ROM) device, DVD-ROM device, DVD-RAM device, DVD-RW device, flash memory device, or some other mass storage device for storing information more permanently. Memory 337 includes code and routines operable to execute sleep application 103, which is described in more detail below.

[0040] The I / O interface 339 can provide functions that enable the computing device 300 to interface with other systems and devices. The interfaced devices may be included as part of the computing device 300 or may be separate and communicate with the computing device 300. For example, a network communication device, a storage device (e.g., memory 337 and / or database 199), and an input / output device can communicate via the I / O interface 339. In some embodiments, the I / O interface 339 can be connected to interface devices such as input devices (e.g., keyboard, pointing device, touch screen, microphone, camera, scanner, sensor, etc.) and / or output devices (e.g., display device, speaker device, printer, monitor, etc.). For example, when a user provides a touch input, the I / O interface 339 transmits data to the sleep application 103.

[0041] Some examples of interfaced devices that can be connected to the I / O interface 339 can include a display 341 that can be used to display content, such as a user interface generated by the sleep application 103 as described herein, and to receive touch (or gesture) input from the user. For example, the display 341 may be utilized to display a user interface that receives user input from the user.

[0042] The display 341 can include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display devices. For example, the display 341 can be a flat display screen provided on a mobile device, a plurality of display screens embedded in a glass form factor or a headset device, or a monitor screen for a computer device.

[0043] The memory device 345 stores data related to the sleep application 103. For example, the memory device 345 can store both initial sensor data and subsequent sensor data, behavioral patterns, user preferences, etc. In an embodiment where the sleep application 103 is part of the cloud server 101, the memory device 345 is the same as the database 199 of FIG. 1.

[0044] Sleep application 103 FIG. 3 shows an exemplary sleep application 103 that includes one or more of a processing module 302, a behavior module 304, a machine learning module 306, a user interface module 308, and a sleep assistant 310.

[0045] The processing module 302 receives sensor data from the sleep hub 120 and aggregates the sensor data. In some embodiments, the processing module 302 includes a set of instructions executable by the processor 335 to process the sensor data. In some embodiments, the processing module 302 can be stored in the memory 337 of the computing device 300 and be accessible and executable by the processor 335.

[0046] In some embodiments, the processing module 302 aggregates sensor data into understandable actions or metrics. For example, the processing module 302 may receive sensor data from a thermal imaging sensor, determine a person's identity, and track the person's movement. In another example, the processing module 302 receives sensor data from the light sensor 240 and determines the brightness in the room.

[0047] In some embodiments, the processing module 302 applies a threshold to the aggregated sensor data to determine whether the aggregated sensor data is within an acceptable level. For example, the processing module 302 may retrieve from the memory device 345 a temperature threshold indicating that the temperature should be between 68 degrees and 72 degrees. If the temperature is below or above the temperature threshold, the processing module 302 may instruct the user interface module 308 to send an alert to the user warning about the temperature. In another example, the processing module 302 may apply a brightness threshold to the room, and the brightness threshold may vary depending on whether the target person is taking a nap or sleeping (for example, the brightness threshold may be much lower for a baby sleeping for a nap, but the brightness threshold may be different for a person working the night shift and sleeping at another time of the day).

[0048] FIG. 4 shows a block diagram 400 of an exemplary flow of sensor data from the physical environment 215 for information determination to a thermal sensor. The thermal imaging sensor 255 generates a thermal image of the target person 210 and the people 215 near the target person existing within the physical environment 205. The processing module 302 uses the thermal image to detect 405 the target person and several people near the target person, determine 410 the distance between the target person and the people, determine 415 the target person movement pattern and the people movement pattern, and detect 420 the object pattern movement. For example, if the target person 210 is an adult, the action module 304 may determine an action pattern based on the target person movement pattern, where the target person movement pattern indicates that when another adult enters the room, especially when another adult approaches that adult, the adult becomes restless in bed and starts to turn over. In another example where the target person 210 is an adult, the target person may always experience arousal when sleeping next to another adult, but sleep more soundly when sleeping alone.

[0049] FIG. 5 is a block diagram 500 of an exemplary flow of sensor data from a physical environment 205 to a sound sensor 245 for pattern determination. The sound sensor 245 generates sensor data based on sounds from a target person 210, sounds from a person 215 near the target person, and sounds spreading from other locations into the physical environment 205, etc. The processing module 302 detects the sound level within the physical environment at 505. The processing module 302 identifies who is making the sound (e.g., parent, child, dog, etc.) and the purpose of the noise (e.g., reading a story, singing, calming sounds, a crying baby and a babbling baby, sounds from electronic devices including a TV or music from another room, sounds from daily household activities, etc.). For example, the processing module 302 receives initial sensor data from a sound sensor 245 including a microphone, applies sound processing techniques to determine the sound level, and determines that the decibel level in the room was 25 dB at 8:02 PM, the decibel level increased to 60 dB because the baby cried quietly at 8:03 PM, the decibel level decreased to 40 dB because the baby started to calm down at 8:04 PM, etc.

[0050] The processing module 302 detects sounds of the target person, sounds of a person near the target person, and other sounds within the physical environment at 510. In some embodiments, the sound sensor 245 includes a microphone capable of distinguishing different types of sounds. For example, the processing module 302 can determine that an increase in sound within the physical environment 205 was due to noise generated outside the room. In some embodiments, the processing module 302 detects sounds associated with the target person, sounds associated with a person near the target, and other sounds by capturing the sound and performing speech processing and machine learning algorithms on the captured speech. In some embodiments, the processing module 302 determines the distance between the target person and other people in the room by applying speech processing techniques.

[0051] The processing module 302 removes a specific sound using a filter 515. For example, the processing module 302 applies a filter to a sound called a boom generated by an electronic device. The processing module 302 detects a sound pattern 520. For example, the processing module 302 can identify fluctuations in different noises and the sound levels of each noise as a function of time.

[0052] FIG. 6 is a block diagram 600 of another exemplary flow of how sound sensor data is used to determine a pattern according to some embodiments described herein. In this example, the processing module 302 uses the sound sensor data 605 to map the sound of the target person 610 and determine the actions of the target person from the sound over a certain time period 615. For example, the target person generated sounds at different decibel levels over a certain time period. The processing module 302 uses the sound sensor data 605 to map the sound of a person near the subject 620 and determine the actions of the person over a certain time period 625. The processing module 302 uses the sound sensor data 605 to map the sound of an object in the physical environment 630 and determine the actions of the object over a certain time period 635.

[0053] FIG. 7 is a block diagram 700 of an exemplary flow of sensor data from a physical environment 205 to a motion sensor for pattern determination according to some embodiments described herein. The motion sensor 250 generates sensor data based on the motion from the target person 210, the motion from a person 215 near the target person, the motion of an object within the physical environment 205, and the motion from the motion sensor 250. The processing module 302 detects the motion of the target person, the person near the target person, and the object within the physical environment 705. The processing module 302 determines the target person movement pattern, the person movement pattern, and the object movement pattern 710. The processing module 302 detects the motion of the motion sensor itself when the motion sensor is moved 715.

[0054] Figure 8 is a block diagram 800 of another exemplary flow of how motion sensor data is used to determine a pattern, according to some embodiments described herein. In this example, the processing module 302 uses the motion sensor data 805 to map 810 the motions of the target person and other people and to determine 815 the actions of the target person and other people from the movements. For example, the processing module 302 can determine that the target person woke up while sleeping and that another person entered the bed with the target person. The processing module 302 uses the motion sensor data 805 to map 820 the movements of other objects and to determine 825 the actions of the objects. For example, a toy in the room vibrates and rotates, or a pacifier falls from the baby bed to the floor. The processing module 302 maps 830 the movement of the motion sensor and determines 835 whether the motion sensor has been moved.

[0055] The behavior module 304 determines the behavior patterns of a set of sleep events. In some embodiments, the behavior module 304 includes a set of instructions executable by the processor 335 to determine the behavior patterns. In some embodiments, the behavior module 304 is stored in the memory 337 of the computing device 300 and can be accessed and executed by the processor 335.

[0056] In some embodiments, the behavior module 304 differentiates between the initial sensor data received during a time period and the subsequent sensor data received from the processing module 302. For example, the behavior module 304 can collect the initial sensor data for a period of time, such as 24 hours, 2 days, 1 week, 1 month, etc., to determine a baseline of the behavior pattern. The subsequent sensor data is received after the behavior pattern is determined to evaluate whether the target result has been achieved.

[0057] The action module 304 determines a baseline of the behavioral patterns of a set of sleep events of a target person based on the initial sensor data. The set of sleep events may include preparations for bedtime, nocturnal awakenings due to changes in the sleep cycle, nocturnal awakenings by a parent, the target person leaving the physical environment, the end of the night, preparations for a nap, nap awakenings due to changes in the sleep cycle, nap awakenings by a parent, the end of the nap, and sleep. The action module 304 can determine attributes associated with the sleep events from the set of sleep events, such as the times when the target person started and stopped each of the sleep events. The action module 304 also determines actions related to the set of sleep events as part of the baseline of the behavioral patterns. For example, the action module 304 determines that it would correspond to a nocturnal awakening by a parent when the parent enters the physical environment and attempts to calm the target person and get them to sleep again.

[0058] The attributes of the preparations for bedtime may be the first fidgety movements and vocalizations within the physical environment, the lights being on within the physical environment, or two or more people being present within the physical environment at the start of the bedtime. For example, after a 45-minute bedtime preparation period, a particular baby may then spend a total duration of 10.2 hours in their room at night based on the average determined from the initial sensor data. In some embodiments, the action module 304 applies an offset to one or more of the sleep events before defining them as being in progress. For example, the end of the preparations for bedtime cannot occur unless there are 15 minutes where the light is dim, the sound machine is on (if used), and movement is minimal (if the target person is alone in the room). For example, if movement occurs after 15 minutes, the action module 304 classifies the movement as part of a nocturnal awakening. In some embodiments, the action module 304 also tracks whether the parent remains in the physical environment after the target person has fallen asleep. If the parent remains in the room, the action module 304 can determine that the end of the preparations for bedtime has not occurred unless there are 15 minutes where the light is dim and the target person is not interacting with the parent.

[0059] The attributes of nocturnal awakenings can be short time periods during which the target subject wakes up in the middle of the night and can include movement and vocalizations within the bed (e.g., a baby crying, a person talking in their sleep, coughing, clearing their throat, etc.). In some embodiments, nocturnal awakenings may include changes in the sleep cycle such as the target subject opening their eyes, tossing and turning to get comfortable, getting up to use the toilet and then returning to bed.

[0060] Nocturnal awakenings by a parent include periods of interaction between the parent and the target subject (e.g., a baby, a child, etc.). Nocturnal awakenings by a parent can include vocalizations from both the parent and the person, movement from either the parent or the person (e.g., the parent enters the physical environment or, if the parent sleeps in the room with the target subject, exits the bed), an increase in light saturation within the physical environment, breastfeeding, diaper changing, etc. Nocturnal awakenings by a parent end when the parent leaves the room and the target subject becomes quiet and stops moving.

[0061] The attributes of the target subject exiting the physical environment can include any time during which the target subject exits the physical environment and then returns to a complete state of sleep thereafter.

[0062] The attributes of the end of the night may include an increase in light saturation, the parent returning the target subject to the bed, placing the target subject to sleep in the bed within the physical environment, or movement and vocalizations (e.g., talking, diaper changing, breastfeeding, etc.) that do not end by removing the target subject from the physical environment and not returning them to the physical environment for a period of time.

[0063] The attributes of naptime preparation can include the time between the first fidgety movements and vocalizations within the physical environment at the start of the desired naptime. In some embodiments, the action module 304 differentiates between naptime and night based on time. In some embodiments, the offset is defined as a predetermined amount of time (e.g., 15 minutes) during which the light is dim, the sound machine is on (if used), and movement is minimal.

[0064] The attributes of a nap awakening may be a short time period during which the target person wakes up in the middle of the night and can include movement and vocalization within the bed. In some embodiments, a nap awakening includes a change in the sleep cycle.

[0065] The attributes of a nap awakening by a parent may include vocalization from both the parent and a person, movement from either the parent or the person, an increase in light saturation within the physical environment, breastfeeding, diaper changing, etc. A nap awakening by a parent ends when the parent leaves the room and the target person becomes quiet and stops moving.

[0066] The attributes of the end of a nap may include an increase in light saturation, an increase in movement and vocalization, and the target person leaves the room and does not return for a predetermined amount of time (e.g., 15 minutes).

[0067] The attributes of sleep may include a time period during which the room is quiet, minimal movement from the target person and anyone else within the physical environment, and minimal vocalization defined by a threshold range of decibel levels. For example, if a child sleeps from 10 PM to 2 AM and from 3 AM to 7 AM, the sleep is recorded as 8 hours because it does not include a nighttime awakening from 2 AM to 3 AM.

[0068] One advantage of defining sleep events based on a fixed set of attributes is that the behavior module 304 creates a consistent method for measuring sleep events and determining whether there is a correlation or causal relationship between activity and different types of sleep events. For example, the behavior module 304 defines sleep as occurring when the room is quiet, there is minimal movement from anyone in the room, and minimal vocalization occurs. Conversely, a parent may define waking up from sleep as occurring even when the baby starts to talk, because the parent considers that talk to be minimal. As a result, this parent may decide that the baby only woke up twice at night when the behavior module 304 identified four awakenings. Thus, the behavior module 304 recognizes a problem that is more likely to disrupt sleep than the parent realizes. In a second example, a second parent may decide that the exact same behavior shown to the first parent causes six awakenings at night, because the second parent may interpret a faint murmur as an awakening. As a result, the behavior module 304 identifies that the problem is not as severe as the parent thinks, because the behavior module 304 does not recognize that the parent may still be sleeping while the baby is talking. Further, because the behavior module 304 measures the impact of the intervention, the parent can easily determine whether the sleep metric is improving.

[0069] The action module 304 can determine an action pattern based on initial sensor data from one of the sensors or by combining initial sensor data from multiple sensors. For example, based on the result of the noise from the object detected by the sound sensor 245, the action module 304 can determine that a woman sleeping in bed began to move as detected by the motion sensor 250, and that the motion was sufficient to determine that a nighttime awakening event occurred. In another example, when an insomniac adult plays music set by a timer and falls asleep while the music is playing, the action module 304 can determine that the stop of the music results in a consistent pattern of nightly awakenings where the adult has difficulty falling back asleep for three hours after waking up. In yet another example, the action module 304 can determine that an adult taking medicine at night due to a medical problem (e.g., cancer) may have difficulty falling back asleep after waking up.

[0070] In some embodiments, the initial sensor data is received from a combination of sensors that are part of the sleep hub 120, the activity tracker 127, and the user device 115. The action module 304 can determine an action pattern based on initial sensor data from different sources. For example, the thermal imaging sensor 255 detects that an adult has entered the room, the motion sensor 250 includes a vibration sensor that accurately identifies the vibration as belonging to the woman as accurately as the processing module 302 specified, and the initial sensor data from the activity tracker 127 confirms that the mother was walking when she entered the physical environment with the baby. Therefore, the action module 304 can determine that the mother entered the physical environment while the baby was making crying sounds at night.

[0071] FIG. 9 is a block diagram 900 of an exemplary flow of how initial sensor data is used to determine an action pattern. In this example, the initial sensor data 905 is received from the light sensor 240, the motion sensor 250, and the sound sensor 245, although other combinations and additional types of initial sensor data may be used. The action module 34 receives the initial sensor data 905 aggregated by the processing module 302. The action module 304 determines 910 the action pattern of a person interacting with the target person and the brightness of the physical environment, determines 915 the action pattern of the movement of the target person and the person detected within the physical environment, and determines 920 the action pattern of the sound level and the type of sound within the physical environment. For example, the action module 304 can determine patterns such as a parent sitting in a rocking chair or couch, the target person lying down, a parent changing a diaper, a parent putting a baby to sleep in a baby bed, a parent picking up a baby from a baby bed, etc. The action module 304 compares the action patterns over time and sets a baseline action 925. For example, the action module 304 can identify a baseline for the delay in sleep onset, which is defined as the amount of time elapsed from when the parent leaves the room until the child falls asleep. The action module 304 classifies the action patterns by factor 930 to determine changes in actions over a period of time (e.g., over the course of a week). For example, the start of bedtime preparations may be later on weekends, naps may be skipped on busy days, sleep may be longer on weekdays, etc. Setting a baseline is useful for determining the expected amount of activity. For example, the target person may always get up in the middle of the night to go to the toilet, which may not be something that changes depending on the results of the actions.

[0072] When the action module 304 sets an action pattern, the action module 304 generates recommendations for achieving a target result based on the action pattern. In some embodiments, the target result is set by the user. For example, the user may state that the target result is to reduce the delay in sleep onset to less than 30 minutes, eliminate actions that interfere with sleep, and achieve 10.5 hours of uninterrupted sleep as indicated by the sleep attributes described above (e.g., minimal movement). In another example, the target result may be to reduce the delay in sleep onset to less than 15 minutes, eliminate interfering actions, reduce night awakenings and early awakenings, achieve 12 hours of independent sleep, eliminate parental night-time attendance, eliminate the use of medications, and eliminate the use of supplements.

[0073] In another example, the action module 304 analyzes the action pattern of a target person with autism spectrum disorder (ASD), determines that the delay in sleep onset is between 30 minutes and 1 hour, and proposes reducing the delay in sleep onset as a target result. The action module 304 also identifies an action pattern where bedtime preparation includes the target person changing into pajamas and watching TV for 40 minutes, browsing the Internet on a tablet, or playing noisy games on a tablet. The action module 304 generates recommendations that include playing noisy games on a tablet before changing into pajamas to create a distance between stimulating activities and bedtime preparation.

[0074] In some embodiments, the user may specify one or more target results to be achieved, as described in more detail below with reference to the user interface module 308. For example, the target results may propose target results typical for a target person of a particular age. In some embodiments, the action module 304 may instruct the user interface module 308 to ask a series of questions designed to identify the target results. For example, the user interface may include a section where the user can identify that the sleep problem is related to an inability to fall asleep, an inability to stay asleep, an inability to sleep for a length corresponding to the age, etc. In that case, the user interface may include options for further narrowing down the sleep problem, such as whether the inability to stay asleep is proven by noise coming from the room, a child waking up the parent, the child needing the parent to accompany them to fall asleep, the child acting listless the next morning, etc.

[0075] In some embodiments, the target results include target results for children and target results for parents. For example, if a child starts falling asleep earlier, a parent may need a reminder to go to bed on their own using the extra sleep time. The recommendations may include a list of items the user should continue to do and a list of items that prevent the achievement of the target results. For example, the list of items the user should continue to do may include recommendations such as the ideal temperature of the room, the ideal light level in the room, and since devices with screens that emit blue light interfere with the production of melatonin, stop using such devices. In another example, the list of items the user should continue to do may include not looking at the baby after hearing the first signs of vocalization, not doing a bedtime reading aloud in a lively voice, and not sleeping in the room with the baby after a night awakening.

[0076] The action module 304 sets the actions of the target person and provides recommendations to the user 935. In some embodiments, the recommendations include reports of action patterns. For example, a target person who has insomnia due to post-traumatic stress disorder may receive a report identifying the correlation between a particular type of sound and nocturnal awakenings. In another example, the target person may be an elderly person, and the caregiver may receive recommendations along with a report of the elderly person's activities when the elderly person is trying to sleep.

[0077] The action module 304 can generate recommendations based on the action pattern by identifying factors that contributed to actions that hindered the target outcome. For example, the action pattern may include a baseline of sleep where an adult slept through the night and events when the adult woke up. The action module 304 can identify that the event when the adult woke up was caused by factors different from the baseline information, such as a temperature outside the temperature threshold, a noisy sound following the adult starting to move, or other factors such as subsequent sensor data from the activity tracker 127 when the adult reported experiencing an abnormal amount of stress.

[0078] In some embodiments, the action module 304 identifies exacerbating factors for sleep problems and generates recommendations to minimize the exacerbating factors. For example, the action module 304 can identify an action pattern where a child wakes up an average of four times at night and plays with toys in bed. The action module 304 helps guide the parent to put away the toys and books from the bed and then recommends a series of steps to measure nocturnal disruptions to determine whether the intervention is working as expected, thereby generating recommendations to minimize the exacerbating factors for the "playing with toys" action.

[0079] In some embodiments, the action module 304 assigns a recommendation score to each recommendation based on the likelihood of the recommendation achieving the target result. For example, each recommendation may be scored on a scale of 1 to 100, or other scales such as 0 to 1, percentage, etc. are also possible. In some embodiments, the action module 304 generates a recommendation if the recommendation has a likelihood exceeding a threshold likelihood of achieving the targeted action. For example, if a baseline sleep time is set in a set of conditions and a temperature change results in a 75% probability of modifying the sleep time, since 75% exceeds the threshold of 70%, the action module 304 can assign a recommendation score to the recommendation to maintain the temperature that existed during the baseline sleep time within the threshold range.

[0080] In some embodiments, the action module 304 receives subsequent sensor data aggregated by the processing module 302 and, based on a comparison of the subsequent sensor data with the action pattern, determines that a certain action is likely to cause a nighttime awakening or a nap awakening, and can provide a warning that a certain action is likely to cause a nighttime awakening or a nap awakening. For example, the action module 304 can identify that because the parent is watching TV at too loud a volume and that sound is spreading into the physical environment where the baby is sleeping, and the baby woke up when the same level of noise spread into the physical environment during the setting of the action pattern, this may cause a nighttime awakening.

[0081] If the user follows the recommendations and the target result is achieved, the action module 304 may use subsequent sensor data to update the action pattern. For example, the action module 304 may set a new baseline for each of the set of sleep events. In some embodiments, the action module 304 updates the target result based on updating the action pattern. For example, can the action module 304 automatically determine a new target result and propose it to the user, or can the user interface module 308 ask the user if the user wants to set a new target result based on the success of achieving the previous target result. For example, if the target result for a teenager is to shorten the start delay when the baseline behavior is a 1-hour start delay, the action module 304 may first recommend advancing the teenager's bedtime by 1 hour. Thereafter, if the teenager experiences a start delay of 30 minutes or less, the action module 304 may recommend delaying the bedtime by 30 minutes.

[0082] If the recommendations are not followed, the action module 304 may instruct the user interface module 308 to provide the user with some reward for following the recommendations. In some embodiments, the reward is determined during the preference assessment and modified based on the age of the target subject. For example, a parent can provide information on how a target subject is motivated by a certain type of candy, sticker, praise, etc. In another example, some children are motivated by charts, access to preferred activities (e.g., going to the park with mom), attention, or tangible rewards. For example, the user interface module 308 may provide the user with a $5 gift card as a tangible reward if the user subsequently follows the recommendations.

[0083] In some embodiments, when the action module 304 determines that it has followed the recommendations, the action module 304 determines whether the target result has been achieved. If it has followed the recommendations but the target result has not been achieved, the action module 304 may generate new recommendations. In some embodiments, if the user follows the recommendations multiple times and this does not help achieve the target result, the action module 304 may investigate the completeness of the procedure and may propose a new intervention or approach based on the feedback. The new intervention or approach is part of a larger plan to address the target result.

[0084] In some embodiments, the new recommendations may be selected based on the scores associated with each of the recommendations. For example, if the action module 304 scores different recommendations and provides only one of the recommendations, the action module 304 may provide the second best recommendation as a result of the first recommendation not cooperating with the modifications to the overall plan to achieve the target result. For example, the first recommendation may be that an adult does not watch TV within two hours of bedtime, and the second recommendation may be that an adult avoids reading books that make people anxious, such as books about climate change.

[0085] In some embodiments, the action module 304 may use subsequent sensor data to add to the baseline of the behavior pattern. For example, the action module 304 corrects the behavior pattern using subsequent sensor data after the user follows the recommendations and the target result is achieved.

[0086] In some embodiments, instead of the behavior module 304 that determines the behavior pattern of a set of sleep events of a target person and applies rules to determine recommendations for achieving a target result, the sleep application 103 includes a machine learning module 306 that generates clusters of behavior patterns and trains a machine learning model to output recommendations for achieving a target result. In some embodiments, the machine learning module 306 includes a set of instructions executable by the processor 335 to train a machine learning model to output recommendations for achieving a target result. In some embodiments, the machine learning module 306 may be stored in the memory 337 of the computing device 300 and be accessible and executable by the processor 335.

[0087] In some embodiments, the machine learning module 306 implements supervised learning by using a training data set of initial sensor data along with parameters such as temperature, pressure, humidity, light, sound, motion, and thermal images as a function of time. The initial sensor data is labeled with recommendations for achieving a target result. In some embodiments, the machine learning module 306 implements unsupervised learning by using a training data set of initial sensor data and does not include labeled recommendations for achieving a target result. In some embodiments, the training data set is further grouped according to attributes of the target person such as age, gender, weight, location, attachment style, target result, etc. It may be advantageous to further group the data according to age since the sleep behavior of an infant is very different from that of a person in their 80s, for example. In some embodiments, the training data includes images of the physical environment and / or audio of the physical environment. In some embodiments, the machine learning module 306 implements a deep learning machine learning model.

[0088] In some embodiments, the machine learning module 306 generates per-parameter clusters based on data similarity. For example, if the training data is labeled, the clusters may be for temperature, pressure, humidity, light, sound, motion, and thermal images as a function of time, and may be further grouped based on attributes such as age, gender, weight, location, attachment style, target outcome, etc. In another example, when the training data is unlabeled, the clusters may be for temperature, pressure, humidity, light, sound, motion, and thermal images as a function of time.

[0089] The output of the machine learning model during training is a recommendation that can help achieve the target behavior. For example, in supervised learning, the model during training may be given time series data from various sensors, the target behavior is not achieved during a first period, and the target behavior is achieved during a second period. The model can learn from the data various parameters that changed between the two periods. Each changed parameter can be a potential recommendation as part of a larger intervention. Based on such data, one or more parameters of the machine learning model can be adjusted. Then, the model may be given only the data for the first period (during which the target behavior was not achieved) and may generate recommendations for changes to achieve the target behavior. The recommendations may be compared to the ground truth corrections that worked to achieve the target behavior (e.g., lower the sound level, play white noise, turn off the light, etc.). Feedback is provided to the machine learning model, for example, to adjust the weights for one or more nodes (if the model is implemented using a neural network). Training the model to make predictions for a large dataset produces a trained machine learning model, and then the trained machine learning model can be used to create recommendations for a subject in real life.

[0090] When the machine learning module 306 trains a machine learning model, the trained machine learning model receives the initial sensor data of the target person as well as the attributes of the target person. The machine learning model outputs recommendations for achieving the target result. In some embodiments, the machine learning model may output the targeted actions and recommendations on how to achieve the targeted actions.

[0091] In some embodiments, the machine learning module 306 receives feedback in the form of information as to whether the recommendations were followed and, if so, whether implementing the recommendations led to achieving the target action. If the recommendations were followed but did not help achieve the target action, in some embodiments, the machine learning module 306 modifies the parameters of the machine learning model accordingly.

[0092] The user interface module 308 generates a user interface. In some embodiments, the user interface module 308 includes a set of instructions executable by the processor 335 to generate the user interface. In some embodiments, the user interface module 308 may be stored in the memory 337 of the computing device 300 and be accessible and executable by the processor 335.

[0093] The user interface module 308 displays a user interface that enables a user to input information about a target person. The information about the target person may include name, age, gender, weight, attachment style, etc. If the target person is a baby, the user interface may include additional information related to the baby, such as options for recording breastfeeding (breast milk or formula), diaper changes, etc. In some embodiments, the user interface module 308 generates a user interface for collecting information about bedtime routines that can be used to complement the analysis of the behavior module 304 of the behavior pattern. For example, a parent may describe that the bedtime routine includes breastfeeding the baby in a rocking chair until the baby falls asleep. The behavior module 304 may use this information to verify the sensor data identified by the motion sensor 250 that identifies the mother moving in a repetitive motion when breastfeeding the baby in the rocking chair part of the bedtime preparation while the baby and the same location.

[0094] In some embodiments, the user interface module 308 generates a user interface for setting user preferences that include one or more target results. For example, the user interface module 308 may include questions for the user about whether there are preferred temperatures, preferred humidity, preferred brightness, preferred sound levels, etc. in the physical environment where the target person is sleeping. In some embodiments, the user interface may include suggestions such as "The American Medical Association recommends that sleep is best achieved in a room with a temperature of 68-72 degrees Fahrenheit. Is that okay?" In some embodiments, the user preferences include privacy settings. For example, the user can specify that they do not want images to be captured by the camera 260 due to medical concerns, military concerns, or other privacy issues. However, the user can verify that they are more comfortable based on the sensor data from the thermal imaging sensor 255.

[0095] In some embodiments, the user interface module 308 includes suggestions for target outcomes in a drop-down menu. Other options are possible, such as text fields where the user can enter data. Referring to FIG. 10A, an exemplary user interface 1000 for configuring sleep preferences is shown. In this example, the user interface 1000 includes information already entered about the target subject, such as age and medical condition, and the target outcomes are listed as a drop-down menu 1002. In this example, the user selected the target outcomes by checking the boxes next to the following items: shorten bedtime preparation, prevent nighttime awakenings, increase sleep duration, gradually stop napping, and an option to add additional target outcomes to a text field.

[0096] In some embodiments, the user interface module 308 includes options for suggesting targeted actions to the user. For example, the machine learning module 306 may receive initial sensor data as input and output targeted actions, as well as recommendations for achieving the targeted actions, based on the similarity of the initial sensor data to clusters of parameters.

[0097] In some embodiments, the user interface module 308 also includes options for a user different from the target subject to add targeted actions. For example, a parent may watch TV instead of going to sleep after the baby has fallen asleep. The parent may benefit from a notification from the user interface module 308 and be able to go to bed earlier than usual.

[0098] FIG. 10B shows an exemplary user interface 1025 for providing recommendations for achieving a targeted action. In this example, the recommendations are to turn off the TV at 9:00 p.m., ensure that the child is in bed within 15 minutes of 8:00 p.m., and use blackout curtains. Each recommendation is data-driven based on the action module 304 identifying an action pattern and a machine learning model outputting the action pattern or recommendation to determine the recommendation.

[0099] In some embodiments, the user interface module 308 generates a summary of sleep changes over time for the target person and, if the user is different from the target person, for that user.

[0100] FIG. 10C shows an exemplary user interface 1050 for providing a health check to the user. In this example, the user interface module 308 generates a comparison of the time the target person, i.e., the baby, sleeps as compared to the parent. By showing the sleep times together, the comparison serves as a visual aid to inform the user (grandparent, caregiver, spouse, etc.) of ways to help improve the sleep of the primary nighttime caregiver. In this example, the comparison helps the user recognize that they need more sleep and the times when additional sleep is possible, and in some cases, follow recommendations to achieve the target result or modify the target result to include less wakefulness.

[0101] FIG. 10D shows an exemplary user interface 1075 for providing a tangible reward for following a recommendation, according to some embodiments described herein. In this example, since the user has not followed the recommendation, the user interface module 308 offers a $10 gift card to provide an additional incentive for the user to follow the recommendation.

[0102] Tangible rewards can be useful for older target subjects. For example, a sports enthusiast can benefit from the goal result of getting enough sleep by following recommendations such as consuming sufficient protein and maintaining a regular bedtime.

[0103] In some embodiments, the user interface module 308 generates a visual analysis including a sleep summary for the user every morning to provide the user with metrics for gauging progress. FIG. 10E shows an exemplary user interface 1090 for providing a sleep graph of a target subject according to some embodiments described herein. In this example, the baseline sleep graph shows that the bedtime preparation lasted for two hours, sleep occurred from 9:30 to 11:00 PM, the baby experienced a night awakening by the parent from 11:00 PM to 12:00 AM, sleep occurred from 12:00 AM to 1:00 AM, the baby experienced a night awakening from 2:00 AM to 3:00 AM, sleep occurred from 3:00 AM to 5:00 AM, the baby woke up at 5:00 AM, but the parent was in the room with the baby and tried to sleep with the baby to extend the night while the parent was asleep.

[0104] The sleep shown in the graph is independent sleep, and independent sleep is when the target person sleeps without the help of any other person in the physical environment. In some embodiments, the processing module 302 determines the time when the target person slept independently, the time when movement occurred, etc. The black sections are a mixture of sleep-wake and adult interaction. The black sections indicate when the baby woke up and are analyzed by the action module 304 to determine what caused the sleep-wake. For example, in the case of a baby, sleep-wake may be caused by interaction with the parent, but as the baby gets older, sleep-wake may be caused by something else, such as the child accessing a tablet. The action module 304 can identify different causes of sleep-wake based on different types of sensor data. For example, the action module 304 differentiates thermal sensor data that identifies whether there are different people in the room or the child is looking at a tablet, whether there are sounds made by two people in the room or the child is tapping on the tablet, etc.

[0105] The user interface 1090 also shows the progress of sleep improvement because during the previous night, the bedtime was shortened by 30 minutes and the baby slept more than 3 hours. The summary does not include the sleep graph for the user, but the user's sleep is also measured and the summary also includes information about the parent's additional sleep. This is advantageous because when the parent wakes up because of the baby, the parent may have trouble getting back to sleep and may be doing things other than sleeping at night, such as walking around the living room, watching TV, and replying to emails.

[0106] In this case, since sleep has improved, the user interface includes a button 1092 for changing the target result. If the target result was to shorten the bedtime preparation and increase sleep by three hours, the goal has been achieved and the user can prepare to set a new goal. Examples of new goals may include falling asleep alone, achieving independent sleep throughout the night, waking up at a certain time in the morning, generalizing new behaviors to other caregivers and settings, and setting goals over time to monitor whether they are being maintained.

[0107] Figure 10F shows an exemplary user interface 1095 that includes a checklist of recommendations, according to some embodiments described herein. The checklist can be particularly useful for reminding the user about activities that are not easily tracked. For example, this checklist includes ensuring the target person's active outdoor time, discussing the target person's transition to bed, always providing enough food for the goal setup, preparing the bedroom for sleep, and reading a book designed to teach the target person about sleeping alone and discussing the book.

[0108] In some embodiments, when the user provides an input (e.g., mouse arrow, finger, etc.), the user interface includes options for providing additional information about the subject. For example, clicking on "Riley's active outdoor time" can link to an article about the science of sleep and its connection to activity. In another example, clicking on "Always provide enough food for Riley" can link to a user interface that allows the user to enter information about what Riley ate and can provide additional analysis on whether the amount of food was sufficient for a target person of Riley's age. In some embodiments, information about food intake may be used by the behavior module 304 to determine the target person's eating patterns, and as a result, additional recommendations may be generated based on the user input.

[0109] Figure 11A shows an exemplary graph 1100 of bedtime preparation as a night function, based on the baseline phase, the phase when following recommendations, and the phase when following new recommendations, according to some embodiments described herein. In this example, two lines represent data determined by sensor data from the sleep hub 120 and data reported by the user. The action module 304 can determine that it takes longer to get the target person ready for bed than the information from self-reporting because the action module 304 is using different definitions and / or the information from self-reporting is less reliable.

[0110] Initial sensor data is collected to create a baseline for the first five days, but the user is not implementing any recommendations. As a result, bedtime preparation is quite long. The action module 304 generates recommendations for the user to follow from the night of the 5th day to the night of the 12th day, but the recommendations are not successful, and the only significant improvement is the length of bedtime preparation. As a result, the action module 304 generates new recommendations for the user to follow from the night of the 13th day to the night of the 21st day. The new recommendations are even more effective in reducing bedtime preparation.

[0111] Figure 11B shows an exemplary graph 1150 of blocks where the target result is achieved or not achieved during the phases shown in Figure 11A, according to some embodiments. In this example, the filled blocks represent nights when the target result was achieved ("met"), and the white blocks represent nights when the target result was not achieved ("unmet"). The graph shows three target results, namely, bedtime preparation of less than 90 minutes (as shown in Figure 11A), interference actions of less than 2 minutes, and no occurrence of night awakenings. In this example, the action module 304 generates multiple recommendations to address all of the target results.

[0112] The user interface module 308 generates a user interface that includes information about personal information collected by the application, the user's right to delete personal information collected by the application, the right to opt out of the sale of personal information, and information about the user's right not to be discriminated against if the user chooses to exercise any of these rights. This information may be provided before the user enters any information.

[0113] The sleep assistant 310 provides sleep assistance to the user. In some embodiments, the sleep assistant 310 includes a set of instructions executable by the processor 335 to provide sleep assistance. In some embodiments, the sleep assistant 310 is stored in the memory 337 of the computing device 300 and may be accessible and executable by the processor 335.

[0114] In some embodiments, the sleep assistant 310 provides an interface for the sleep hub 120 when the user and / or the target person attempts to access functions on the sleep hub 120. For example, the sleep assistant 310 may enable the user to record voices such as stories, songs, etc. for playback during bedtime. In some embodiments, the sleep assistant 310 is connected to other services such as a music streaming application, a meditation application, etc. In another example, the user can request the sleep hub 120 to start a white noise machine via the sleep assistant 310. In some embodiments, the white noise machine continues to generate sound until the action module 304 determines that sleep has ended. In some embodiments, the sleep assistant 310 includes an option to set a timer for the white noise machine or a manual option to switch off the white noise machine.

[0115] In some embodiments, the sleep assistant 310 includes prompts for the user regarding the sleep events the user is experiencing. The sleep assistant 310 may apply the behavioral principles in an action-oriented mode designed to teach the user how to develop healthy sleep habits based on the support of scientific and medical experts. In some embodiments, the sleep assistant 310 asks the user for feedback on how the user has received the process, detects patterns, and provides advisory notifications to ensure a peaceful night. In some embodiments, the sleep assistant 310 includes a data repository with additional information about the science of sleep. For example, the sleep assistant 310 can instruct the user interface module 308 to generate a user interface for the user to search for specific articles, view sections of frequently asked questions, etc.

[0116] In some embodiments, the sleep assistant 310 includes options for triggering a conversation with a sleep consultant. Access to the sleep consultant can be through live counseling videos, through chat, etc. In some embodiments, the sleep assistant 310 also includes a chatbot option for answering common questions about sleep by accessing a database with frequently asked questions.

[0117] Exemplary flowchart FIG. 12 is a flowchart showing an exemplary method for providing recommendations to achieve a target result. The method shown in flowchart 1200 can be implemented by the computing device 300 of FIG. 3.

[0118] Method 1200 may start at block 1202. At block 1202, initial sensor data is received from a set of sensors in a physical environment over a period of time, and the set of sensors includes at least one sensor selected from a set of a temperature sensor, a pressure sensor, a humidity sensor, an optical sensor, an acoustic sensor, a thermal imaging sensor, and a motion sensor. Block 1204 may follow block 1202.

[0119] At block 1204, the initial sensor data is aggregated to identify target subjects in the physical environment, one or more persons near the target subject, and attributes of objects. Block 1206 may follow block 1204.

[0120] At block 1206, a behavior pattern of a set of sleep events of the target subject is determined based on the initial sensor data. Block 1208 may follow block 1206.

[0121] At block 1208, recommendations are generated based on the behavior pattern to achieve the target result of the target subject in the physical environment. Block 1210 may follow block 1208.

[0122] At block 1210, the recommendations are provided. Block 1212 may follow block 1210.

[0123] At block 1212, it is determined whether the recommendations have been followed. If the recommendations have not been followed, block 1214 follows block 1212. At block 1214, an offer of a reward if the recommendations are followed thereafter is made.

[0124] If the recommendations are followed, block 1216 follows block 1212. In block 1216, it is determined whether the target result has been achieved. If the target result has not been achieved, block 1218 follows block 1216. In block 1218, the recommendations are modified. For example, if the modification was to change the temperature in the room to reduce night-time awakenings, the action module 304 may change the recommendations for different factors, such as reducing the number of times the user enters the room after the baby has gone to sleep.

[0125] If the target result has been achieved, block 1220 may follow block 1216. In block 1220, the user is asked whether they want to define a new target result.

[0126] The various embodiments described herein include obtaining data from various sensors within a physical environment, analyzing such data, generating recommendations, and providing a user interface. Data collection is carried out in compliance with applicable laws only when there is specific user permission. The data is stored in compliance with applicable laws, including anonymizing or otherwise modifying the data to protect user privacy. The user is provided with clear information about data collection, storage, and use, and options to select the types of data that may be collected, stored, and utilized. Further, the user controls the devices on which the data may be stored (e.g., client device only, client device and server device, etc.) and the devices on which data analysis is carried out (e.g., client device only, client device and server device, etc.). The data is utilized for the specific purposes described herein. Without clear user permission, the data is not shared with third parties.

[0127] In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In some instances, in order to avoid obscuring the description, structures and devices are shown in block diagram form. For example, embodiments may have been described above primarily with respect to user interfaces and specific hardware. However, embodiments may be applicable to any type of computing device capable of receiving data and commands, as well as any peripheral device that provides services.

[0128] References in this specification to "some embodiments" or "some instances" mean that a particular feature, structure, or characteristic described in connection with the embodiment or instance may be included in at least one implementation of the specification. Appearances of the phrase "in some embodiments" in various places in this specification are not necessarily all referring to the same embodiment.

[0129] Some portions of the detailed descriptions above are presented from the perspective of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the data processing arts. An algorithm is here, generally, considered to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0130] However, it should be borne in mind that all of these terms and similar terms should be associated with appropriate physical quantities and are nothing more than convenient labels applied to these quantities. In particular, unless otherwise specified, as will be apparent from the following description, throughout this specification, descriptions using terms including "processing" or "computing" or "calculating" or "determining" or "displaying" or the like refer to actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memories of the computer system and converts it into other data similarly represented as physical quantities in the computer system memory or registers or other such information storage devices, transmission devices, or display devices.

[0131] Embodiments of this specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium including, but not limited to, any type of disk including optical disks, ROM, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic cards or optical cards, flash memories including USB keys with non-volatile memories, or any other type of medium suitable for storing electronic instructions.

[0132] This specification can take the form of some overall hardware embodiments, some overall software embodiments, or some embodiments including both hardware elements and software elements. In some embodiments, this specification is implemented in software including, but not limited to, firmware, resident software, microcode, etc.

[0133] Furthermore, this specification can take the form of a computer program product accessible from a computer-usable medium or a computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this specification, a computer-usable medium or a computer-readable medium can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0134] A data processing system suitable for storing or executing program code includes at least one processor directly or indirectly coupled to memory elements through a system bus. The memory elements can include local memory used during actual execution of the program code, a mass storage device, and a cache memory that provides at least some temporary storage of at least some program code to reduce the number of times the code must be retrieved from the mass storage device during execution.

Description of Reference Numerals

[0135] 100 Network environment, environment 101 Cloud server 103, 103a, 103b Sleep application 105 Network 115, 115a, 115b, 115n User device 120 Sleep hub 125, 125a, 125b User 127 Activity Tracker 130 Night Light 199 Database 205 Physical Environment 210 Target Person 215 People Near the Target Person 224 I / O Interface 225 Temperature Sensor 230 Pressure Sensor 235 Humidity Sensor 240 Light Sensor 245 Sound Sensor 250 Motion Sensor 255 Thermal Imaging Sensor 260 Camera 265 Speaker 300 Computing Device 302 Processing Module 304 Action Module 306 Machine Learning Module 308 User Interface Module 310 Sleep Assistant 318 Bus 322 Signal Line 324 Signal Line 326 Signal Line 328 Signal Line 330 Signal Line 335 Processor 337 Memory 339 I / O Interface 341 Display 345 Storage Device 400 Block Diagram 500 Block Diagram 600 Block Diagram 605 Sound Sensor Data 700 Block Diagram 800 Block Diagram 805 Motion Sensor Data 900 Block Diagram 905 Initial Sensor Data 1000 User Interface 1002 Drop-down menu 1025 User interface 1050 User interface 1075 User interface 1090 User interface 1092 Button 1095 User interface 1100 Graph 1150 Graph 1200 Method

Claims

1. Receiving initial sensor data from one or more sensors of a sensor set in a physical environment during a time period, wherein the sensor set includes at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor; Determining a behavior pattern of a set of sleep events of a target person based on the initial sensor data; Generating recommendations based on the behavior pattern to achieve a target result of the target person in the physical environment; And providing the recommendations. A method implemented by a computer, comprising:

2. Determining whether the recommendations are followed; And determining whether a target action has been achieved in response to a determination that the recommendations are followed. The method according to claim 1, further comprising:

3. Updating the behavior pattern based on subsequent sensor data in response to a determination that the target action has been achieved; And updating the target result based on the updated behavior pattern. The method according to claim 2, further comprising:

4. Determining whether the recommendations are followed; And offering a reward in case of following the recommendations in response to a determination that the recommendations are not followed. The method according to claim 1, further comprising:

5. The sensor set includes the thermal imaging sensor, and the method includes: Detecting the target person, one or more persons near the target person, and one or more objects in the physical environment based on the initial sensor data; Determining a distance between the target person and the one or more persons based on the initial sensor data; Determining a movement pattern of the target person within the physical environment; Determining one or more movement patterns corresponding to the one or more persons; Determining one or more movement patterns of the one or more objects; further comprising; the step of determining the behavior pattern is based on the movement pattern of the target person, the one or more movement patterns corresponding to the one or more persons, and the one or more movement patterns of the one or more objects; The method according to claim 1.

6. the sensor set includes the sound sensor, and the method includes; detecting a sound level within the physical environment based on the initial sensor data; detecting the sound of the target person, the sound of one or more persons near the target person, and other sounds within the physical environment based on the initial sensor data; removing specific sounds with a filter; determining a sound pattern; further comprising; the step of determining the behavior pattern is based on the sound pattern; The method according to claim 1.

7. the sensor set includes the motion sensor, and the method includes; detecting the motion of the target person, one or more persons near the target person, and one or more objects within the physical environment based on the initial sensor data; determining a target person movement pattern, a person movement pattern, and one or more object movement patterns; further comprising; The step of determining the behavior pattern is based on the target subject movement pattern, the human movement pattern, and the one or more object movement patterns. The method according to claim 1.

8. The step of determining the behavior pattern includes the step of determining an attribute associated with at least one sleep event selected from bedtime preparation, nocturnal awakening, napping preparation, napping awakening, and sleep setting. The method according to claim 1.

9. The step of providing a user interface that requests user preferences for the target result, wherein the target result is defined based on the user preferences. The method according to claim 1, further comprising.

10. Receiving subsequent sensor data during a subsequent time period; Based on a comparison of the subsequent sensor data and the behavior pattern, determining that a certain action is likely to cause a nocturnal awakening or a napping awakening; Providing a warning that the certain action is likely to cause the nocturnal awakening or the napping awakening; The method according to claim 1, further comprising.

11. The recommendations are provided to a user different from the target subject, and the method includes Determining a behavior pattern of a set of sleep events of the user based on the initial sensor data; The method according to claim 1, further comprising.

12. Receiving the initial sensor data associated with the user to identify the length of time when the user is asleep; Providing to the user a user interface including the length of time when the user is asleep compared to when the target subject is asleep; The method according to claim 11, further comprising **Claim 13** providing the initial sensor data as an input to a trained machine learning model; using the trained machine learning model to output the recommendations for achieving the target result The method according to claim 1, further comprising **Claim 14** One or more processors and a memory coupled to the one or more processors, the memory having instructions stored thereon, wherein when the instructions are executed by the processor, the processor is caused to receive initial sensor data from one or more sensors of a set of sensors in a physical environment over a period of time, the set of sensors including at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor; determine a baseline for one or more of a set of sleep events of a target subject based on the initial sensor data; A computing device that performs operations including **Claim 15** The operations further include determining a behavior pattern of a set of sleep events of a target subject based on the baseline and the initial sensor data for the one or more of the set of sleep events; generating recommendations based on the behavior pattern to achieve a target result of the target subject in the physical environment; providing the recommendations; determining whether the recommendations are followed; in response to determining that the recommendations are followed, determining whether a target action has been achieved The computing device according to claim 14, further comprising **Claim 16** the operation being in response to a determination that the target action has been achieved, updating the action pattern based on subsequent sensor data; updating the target result based on the updated action pattern and further comprising the computing device according to claim 15.

17. the operation being determining whether the recommendations have been followed; in response to a determination that the recommendations have not been followed, offering a reward for following the recommendations thereafter and further comprising the computing device according to claim 15.

18. A non - transitory computer - readable medium having instructions stored thereon, which, when executed by one or more computers, cause the one or more computers to perform operations, the operations being receiving initial sensor data from one or more sensors of a set of sensors in a physical environment during a time period, the set of sensors including at least one of a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, a sound sensor, a thermal imaging sensor, and a motion sensor; determining an action pattern of a set of sleep events of a target subject based on the initial sensor data; generating recommendations based on the action pattern to achieve a target result of a target subject in the physical environment; providing the recommendations and comprising a non - transitory computer - readable medium.

19. the operation being determining whether the recommendations have been followed; in response to a determination that the recommendations have been followed, determining whether a target action has been achieved The computer-readable medium according to claim 18, further comprising **Claim 20** wherein the operation updating the action pattern based on subsequent sensor data in response to a determination that the target action has been achieved; and updating the target result based on the updated action pattern The computer-readable medium according to claim 19, further comprising

Images