Method and system for achieving lighting response

Abstract

The present invention relates to predictive lighting from device screens based on user profiles. Systems, methods, and computer-readable storage media are provided to determine a user's mood, infer an appropriate lighting scheme, and then implement the lighting scheme on all devices within a predetermined proximity of the user. In addition, when a user begins a task, the device can track the user and use lighting from nearby screens to provide functional lighting.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201580031595.X, filed on May 12, 2015. Technical Field

[0003] The disclosed embodiments generally relate to lighting systems for use in homes, offices, and other indoor lighting environments. Background Art

[0004] The surrounding environment can have a large functional and emotional impact on the user. Most of the environment is indoor lighting. Traditionally, lighting is part of the house and is separate from the ecosystem of electronic devices around the user. Therefore, the lighting around the user is usually "dumb". The lighting is not personalized and is not optimized for the user based on the user's functional needs or emotional state. Even in networked home lighting systems, most of the devices are manual or manually programmed. For example, some such lighting systems may allow the user to set a timer so that the light will come on at a specific time, or respond to motion sensing. In addition, the system utilizes traditional lighting sources (such as light bulbs) and tracks the lighting, which has limited capabilities. These systems also do not take into account the user's emotional state or mental state. It would be beneficial to provide a mechanism and method to overcome the limitations and shortcomings of current home lighting. Summary of the invention

[0005] The present invention overcomes the above-mentioned limitations and disadvantages by providing methods, systems and computer-readable storage media for determining a user's mood, inferring an appropriate lighting scheme, and then implementing the lighting scheme on all devices within a predetermined proximity of the user, including conventional lighting devices (such as lamps, track lighting and overhead lighting) and non-conventional lighting devices (such as mobile phones, tablet computers, televisions, and clocks).

[0006] The following provides an overview of the present invention in order to provide a basic understanding of some aspects of the present invention. This overview is not an exhaustive review of the present invention. This overview is not intended to identify key / important elements of the present invention or to describe the scope of the present invention. Specifically, this overview has provided some concepts of the present invention in a simplified form as a prelude to a more detailed description that will be provided later.

[0007] Different implementations of systems, methods, and devices within the scope of the appended claims each have multiple aspects, no single one of which is solely responsible for the desirable attributes described herein. Without limiting the scope of the appended claims, some salient features are described herein. After considering this discussion, and particularly after reading the section entitled "Detailed Description," one will understand how the features of the various implementations may be used.

[0008] One aspect of the present disclosure is a method for adaptive lighting. The method occurs at a computer system having one or more processors and a memory storing programs executed by the one or more processors. A lighting map of lighting devices within a predefined area is accessed. The lighting devices include one or more non-conventional lighting devices. Each lighting device has one or more lighting characteristics, such as a state (e.g., on / off state and current color setting), a screen size (if the device is non-conventional, such as a smartphone, a television, or a clock radio), and a lighting capability (e.g., maximum brightness). The lighting devices can be controlled via one or more networks (such as Bluetooth, Wi-Fi, and a cellular network). For each lighting device, the map indicates at least the location / positioning and current state of the lighting device. In some embodiments, the map also indicates the orientation of the device.

[0009] In some embodiments, real-time contextual information is obtained for the user. For example, the information may be obtained from a set of networked sensors (which may include one or more of a camera, a microphone, a network router, and a local or indoor weather station) and the contextual information may also be obtained from the user's device. In some embodiments, the contextual information includes the user's location within a predefined area (e.g., in a residence, the room where the user is located), the time of day, and one or more indications of user movement, sounds near the user's location, ambient light levels near the user's location, calendar events, web interactions, and device interactions. Then, based on the contextual information, a user task and a lighting requirement associated with the user task are inferred. In addition, based on the user's location, a first subset of lighting devices located near the user's location is determined from the lighting map. The subset of lighting devices includes at least one non-conventional lighting device, such as a mobile phone, a tablet computer, a television, or a clock. Then, the lighting characteristics of the subset of lighting devices are controlled via one or more networks to provide a response to the lighting requirement. In some embodiments, the response is performed during a predefined time period to achieve a lighting effect that changes over time to provide a desired lighting effect.

[0010] Some embodiments provide a centralized lighting controller system including one or more central processing units (CPUs) for executing programs and also including a memory storing programs executed by the CPUs. The programs include instructions for executing any of the embodiments of the adaptive lighting method described above. Some embodiments of the centralized lighting controller system also include program instructions for executing additional options discussed below.

[0011] Still other embodiments provide a non-transitory computer-readable storage medium storing one or more programs configured to be executed by a computer. The program includes instructions for executing any embodiment of the above-mentioned adaptive lighting method. Some embodiments of the non-transitory computer-readable storage medium also include program instructions for executing additional options discussed below.

[0012] Thus, these methods, systems, and storage media provide new, more intuitive, and more efficient ways to provide lighting to users utilizing non-traditional lighting fixtures. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To better understand the above-described aspects of the present invention as well as additional aspects and implementations thereof, reference should be made to the following "Detailed Description" in conjunction with the following drawings, in which like reference numerals represent corresponding parts throughout all figures.

[0014] Figure 1A is a block diagram illustrating a distributed computer system according to some implementations.

[0015] Figure 1B is a block diagram illustrating a system for lighting based on a user's situation according to some implementations.

[0016] Figure 2 is a block diagram illustrating an example server according to some implementations.

[0017] Figure 3 is a block diagram illustrating an example client according to some implementations.

[0018] Figure 4 is a block diagram illustrating a data structure of a lighting map according to some implementations.

[0019] Figure 5 is a flow chart representing a method of providing predictive lighting according to some embodiments.

[0020] Figure 6 is a schematic diagram / blueprint of an example suite in which a user may reside, according to some implementations.

[0021] Like reference numerals designate corresponding parts throughout the figures. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the current embodiments. However, it will be apparent to one of ordinary skill in the art that the current various embodiments may be practiced without these specific details. In other instances, well-known methods, processes, components, and networks are not described in detail in order to avoid unnecessarily obscuring various aspects of the embodiments.

[0023] The screen size of smart devices is growing rapidly. Smartphones with screens of 4" or larger and with full color display capabilities are now available. In addition, smart TVs are becoming more and more popular and screen sizes are about to reach 84 inches. At this size, smart TVs now have a huge impact on the lighting of a room. Tablets, mobile phones, and clocks with display screens can also have an impact on the lighting of a room, especially when light levels are low. As screens continue to grow (both in size and usage), the lighting impact of these screens will also grow. In addition, the screen of the device is often the only task lighting available. For example, people often use their mobile phones as a flashlight, but this manual process is less than ideal.

[0024] Some home automation products allow a user's smartphone to control lighting and HVAC (heating, ventilation, and air conditioning) in a house. However, most of these devices are manual. For example, some home automation products allow manual remote control of home lighting. Some home automation products allow users to set timers so that lights come on at specific times. Some devices have algorithms that learn patterns based on the user's manual adjustments to the thermostat. However, these systems are not configurable to provide predictive lighting by using conventional and non-conventional lighting devices.

[0025] Instead, some embodiments described herein relate to a computing system that determines the mood or lighting needs of a user (or multiple different users in different lighting areas of a house or business), infers an appropriate lighting scheme / plan, and then implements the lighting scheme on all devices within a predetermined proximity of the user. In addition, as the user begins a task, environmental sensors (including cameras and motion sensors) and smart devices (including devices for lighting, such as smartphones with GPS and accelerometer capabilities) can track the user and provide functional lighting using lighting from nearby screens or conventional lights, as long as these lighting devices can accept network control of their lighting capabilities (directly or indirectly).

[0026] In some embodiments, as described in more detail below, a large display screen (e.g., a flexible roll-to-roll display) can occupy a larger proportion of a room. In this case, the screen is the primary light source, rather than a traditional free-standing floor lamp. Thus, the lighting available from each screen can be extensively customized. For example, screen lighting properties, including any of the countless brightness, shape, color, size, and motion of light displayed on the screen. The light can travel with the user or alternate between bright and dark like a heartbeat.

[0027] Figure 1A 1 is a block diagram illustrating a distributed computer system 100 according to some embodiments. The distributed computer system 100 includes a plurality of lighting devices and sensors associated with one or more users. The lighting device may be a non-traditional lighting device (such as a television, a video projector, a smart phone, a tablet computer, a computer monitor, a laptop computer, an audio component, or an appliance). The lighting device 110 may also be a traditional lamp or ceiling light. In some embodiments, each lighting device 110a-n is a networked electronic device with a controllable lighting element. For example, each lighting device 110 may independently have network capabilities or each lighting device 110 may be connected to a power transformer, a charging device, a controllable power outlet, an adapter at a power outlet, a power switch, or a light socket that can be used to control other "dumb" (e.g., non-networked) lamps or appliances by, for example, turning the adapter on or off. In some embodiments, the device 110 having a large amount of computing and communication capabilities can perform at least some operations associated with the lighting application 102 to provide control over the lighting behavior of other devices.

[0028] In some embodiments, the devices and sensors also include a centralized lighting controller 101 that receives information from the lighting devices and sensors and sends commands to the lighting devices 110 via an application program interface. In some embodiments, the lighting controller 101 is configured to understand when a user is using any device running a common operating system (e.g., Android), and then coordinate operations with the device using the command and data format associated with the operating system. In some embodiments, the lighting controller 101 is also configured to perform at least some operations associated with the lighting application 102, such as analyzing data from sensors and lighting devices and predicting the user's tasks. In other embodiments, the lighting controller application 101 only receives information from other lighting devices 110 and sends some or all of the information to the server's lighting application 102 for analysis operations (such as user task prediction). The user's devices and sensors (directly or through the centralized lighting controller 101) communicate with the server system (s) 200 through a communication network 130 (such as the Internet, other wide area networks, local area networks, metropolitan area networks). The server system(s) 200 include a lighting application 102 that receives information from lighting fixtures and sensors (directly or indirectly), predicts a user's tasks, and sends commands to lighting fixtures 110 via an application program interface. In some embodiments, the lighting application 102 interacts with a variety of other systems to obtain additional information about the user's activities and current status, including a calendar system 140, an email system 150, and other systems (e.g., a search system, a social networking system, an entertainment system for watching videos, movies, etc.). The server system lighting application 102 may also reference a lighting fixture database that serves as a reference for lighting characteristics of a variety of different conventional and non-conventional fixtures.

[0029] Figure 1B A system for effective lighting based on a user's situation is illustrated. There are three main aspects of the system that can be implemented by a lighting application 102 executing on a server (e.g., in one or more servers coupled to one or more users and their devices and sensors via the Internet 130): determining a user situation 104, determining an appropriate lighting response 106, and controlling devices proximate to the user 108 to implement the appropriate response. In various embodiments, the operations illustrated as being performed by the lighting application 102 may be performed by a lighting controller (e.g., a centralized lighting controller 101, 102) located indoors (e.g., in a user's home or business) for a particular user. Figure 1A ) is performed. In some embodiments, the operation is performed by the server lighting application 102 and the indoor lighting controller (101, Figure 1AIn the following description, no specific system configuration is required and the described embodiments are intended only as non-limiting examples.

[0030] according to Figure 1B First, determine the user's situation 104. In this step, determine what the user is likely to do next and the profile of the user's overall mood. This step is based at least in part on information determined from the user's device and / or sensors. In some embodiments, information is obtained from web activity history 112 from the following examples, such as: user's interaction with email (e.g., web-based email activity), search history, social network activity, calendar, and alarm clock. In some embodiments, information is obtained from the user's device interaction 114, such as: Wi-Fi hotspot activity, connection to a cellular network, accelerometer movement of the device (e.g., from a handheld or wearable device). In some embodiments, obtain environmental information 116, such as: geographic location (e.g., from GPS), location, time of day (from device clock), weather (from sensors or from location-based weather sources), ambient lighting (device based on screen state and / or ambient lighting of the room based on sensors), motion sensor data (e.g., from accelerometers or other motion sensors that sense the movement of the device or the movement of the user or other objects near the device). In some implementations, device capability information 118 is also obtained, such as: screen size, orientation (eg, from the device's gyroscope), current state (on / off and current brightness setting).

[0031] Using this information, tasks are predicted by using a task prediction algorithm 120. Tasks may include: user needs to wake up, user needs to calm down (e.g., sleep), user needs to go to a meeting, user is sleeping, user is reading, user is watching TV, user needs to use the bathroom, etc.

[0032] according to Figure 1B , then determining an appropriate lighting response 106. Now that the user's tasks 120 (and, therefore, their needs) have been determined, the next action is to combine the user tasks with the lighting rules 122 and the machine learning information 124. The appropriate lighting response includes an appropriate lighting model. The model takes the situation and creates an appropriate lighting environment by using a combination of conventional networked lighting devices (e.g., different types of light bulbs) and unconventional networked lighting devices (e.g., electronic devices with light-emitting display screens). The output of the model is recommended light levels for different proximity to the user over time.

[0033] according to Figure 1B , and then detect and control nearby devices 108. To this end, an API (202, Figure 2), which allows the lighting application 102 to control certain aspects of various devices. For example, in some embodiments, the lighting application 102 interacts with the lighting controller 101 that controls the display of the user device 110. Examples of lighting control include: color temperature, brightness, display on / off, and in some embodiments, displaying specific symbols or messages. Thus, optimizing the behavior 126 of the lighting device close to the user, for example, Figure 5 To explain in more detail, lighting characteristics of nearby lighting devices are controlled to meet the determined lighting needs of the user.

[0034] The API is configured to allow devices to report a number of key characteristics so that the lighting application 102 can generate a map of the devices and use them appropriately:

[0035] Distance / location to the user

[0036] Screen size

[0037] Screen orientation (which way it's facing)

[0038] Screen type (radiative, reflective, e-ink)

[0039] Current state (on, off, brightness, etc.)

[0040] Using this information, the system can tune the lighting near the user for maximum benefit both emotionally and functionally.

[0041] Example 1: Emotion

[0042] Jacob wakes up at 7 a.m. His bedroom TV is dimly lit and shows a nice warm-colored outdoor image. The image makes him feel happy and relaxed. Out of habit, he turns on his phone. The screen is also dimmed and all colors have turned warm. He lies in bed for a while. At 8 a.m., the phone alarm goes off, telling him that he has a meeting at 9 a.m. He takes a nap. By now, the brightness on his TV has increased and illuminates the entire room. The colors are becoming more and more yellow and white. When the alarm goes off again, his phone is at full brightness. The TV is also at full brightness, and the lights in the room have even turned on automatically. His wall-sized screens (e.g., digital photo frames and clocks) are also turned on. Then, Jacob looks at the calendar event and realizes that he doesn't need to go to the meeting. He replies "no" to the meeting. The screen near him continues to be on, but turns back to a softer color and brightness.

[0043] In the above example, the lighting application 102 first receives information from devices and sensors in Jacob's residence, which provides Jacob's location, time of day, and devices in use via an API. The lighting application 102 determines that Jacob is still in bed at 7 a.m. (and his calendar shows that he has no schedule before 9 a.m.). The lighting application 102 determines that he should get up at 8 a.m. and then sends commands to lighting devices (conventional and non-conventional) close to Jacob via an API to adjust the screen brightness and color temperature of these devices, thereby moving from a state of keeping at night to turning on in the morning and providing gradual brightening. When Jacob actively interacts with his devices, such as turning on his phone at 7 a.m., taking a nap at 8 a.m., and replying "no" to the meeting at 9 a.m., the lighting application 102 adjusts its lighting scripts to take into account Jacob's current state. For example, when Jacob replies "no" to the 9 a.m. meeting, the lighting application 102 sends instructions to dim the device screen and return to a softer color. In some embodiments, the lighting device application 102 sends instructions to the local lighting controller 101 to control the appropriate client device, however, in other embodiments, all or some functions of the lighting application 102 are executed locally at the lighting controller 101 .

[0044] Example 2: Night Light

[0045] Mary wakes up at 2 a.m. to use the bathroom. She doesn't want to wake her husband, even though he's a light sleeper. She gets out of bed quietly. Immediately, the floor lamp near her feet lights up, allowing her to put on her slippers. Then, the corner of her TV glows, pointing to the bathroom with a dim arrow (perhaps with an explanatory message). As she walks toward the TV and reaches the hallway, the bathroom radio turns on its screen, causing a soft beam of light to emanate from the bathroom. She walks into the bathroom without turning on a light. When she lies back down in bed, the radio, TV, and floor lamp automatically turn off.

[0046] In the above example, the lighting application 102 first receives information from devices and sensors in Mary's residence, which provide Mary's location, time of day, and devices in use via an API. The lighting application 102 determines that Mary is asleep in bed. The lighting application 102 determines that she should be sleeping, and sends commands to lighting devices (conventional and non-conventional) close to Mary via an API to turn off or turn on dim sleep mode brightness and color temperature. Then, when sensors (such as motion sensors) determine that Mary has quietly gotten out of bed, the lighting application 102 adjusts its lighting script to take into account Mary's current state and her intended tasks. For example, the lighting application 102 sends instructions to the device screen to provide lighting and directional guidance along her path, and then, after she has returned to bed, provides instructions to the device to turn off the device screen. As discussed with the previous example, in some embodiments, the lighting application 102 sends instructions to the local lighting controller 101 to control the appropriate client device, however, in other embodiments, all or some functions of the lighting application 102 are performed locally by the lighting controller 101.

[0047] Figure 2 2 is a block diagram illustrating a server 200 according to some embodiments. The server 200 typically includes one or more processing units (CPUs) 202, one or more network or other communication interfaces 210, a memory 212, and one or more communication buses 214 for interconnecting these components. The communication bus 214 optionally includes circuits (sometimes referred to as chipsets) that interconnect and control the communication between the system components. The server 200 optionally includes a user interface (not shown) that includes a display device and one or more input devices, such as a keyboard, a mouse, a touch-sensitive display screen, or other pointing devices. The memory 212 includes a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state memory device; and may include a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 212 optionally includes one or more storage devices that are far away from the (multiple) CPUs 202. The memory 212, or alternatively, the non-volatile memory device(s) within the memory 212, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 212 or the computer-readable storage medium of the memory 212 stores the following programs, modules, and data structures, or a subset thereof:

[0048] • Operating system 206, which includes processes for handling various basic system services and performing hardware-related tasks.

[0049] A network communication module 208, which is used to connect the server 200 to other computers via one or more communication network interfaces 210 (wired or wireless) and one or more communication networks (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0050] A lighting device database 170 that serves as a reference for lighting characteristics of a variety of different conventional and non-conventional devices (the lighting device database 170 may be remote from the server 200 and accessible by the server 200 via a network connection);

[0051] A lighting application 102 that receives information from lighting devices and sensors, predicts the user's tasks, and sends commands to lighting devices 110 via an application program interface (API) 204 to implement a lighting plan that is appropriate for the context. As noted above, in some embodiments, the lighting application 102 can develop and manage multiple lighting plans for multiple users in different lighting areas of the same house or business. In some embodiments, the lighting application 102 can develop and manage multiple lighting plans for multiple different lighting areas of the same house or business. In some embodiments, the lighting application includes at least some of the following modules and data structures:

[0052] o Lighting map module 220, which creates and later accesses lighting maps 228 for lighting devices (such as for Figure 5 described in more detail);

[0053] o A context information determination module 222 for obtaining information 230 associated with the context of one or more users, such as, for each user, location, time of day, etc. (e.g., from Figure 1B Components 112, 114, and 116);

[0054] o Task prediction / task inference module 224, the task prediction / task inference module 224 includes a task prediction / task inference module 224 for determining Figure 1B Task prediction algorithm for described user task / situation 104 / 232; and

[0055] o Lighting control / lighting response module 226, which is configured to respond to the lighting control / lighting response module 226 according to the lighting control / lighting response module 226. Figure 1BThe lighting plan 106 / 234 developed by the lighting application 102 is described to control the lighting characteristics of at least some of the lighting devices close to the user. In some embodiments, the lighting plan includes one or more lighting events, each lighting event involving a combination of lighting operations performed at one or more lighting devices via one or more lighting commands, as indicated by the API 204. For example, the example lighting plan 234-1 includes three lighting events LE1, LE2, and LE3, wherein the lighting event LE1 is implemented via three lighting commands LC1, LC2, and LC3 that can be issued to each or more lighting devices.

[0056] Each of the elements identified above is typically stored in one or more of the previously mentioned memory devices and corresponds to a set of instructions for performing the functions described above. It is not necessary to implement the modules or programs (i.e., sets of instructions) identified above as separate software programs, processes, or modules, and therefore, various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 212 stores a subset of the modules and data structures identified above. In addition, memory 212 may store additional modules and data structures not described above.

[0057] Figure 3300 (such as a user device, or lighting controller 101) according to some embodiments. The client 300 typically includes one or more processing units (CPUs) 302, one or more network or other communication interfaces 310, a memory 312, and one or more communication buses 314 for interconnecting these components. The communication bus 314 optionally includes circuits (sometimes referred to as a chipset) that interconnect and control communication between system components. The client 300 optionally includes a user interface 304, which includes a display device 306 (e.g., a display screen) and one or more input devices 308, such as a keyboard, a mouse, a touch-sensitive display screen, or other pointing devices. In some embodiments, the client 300 includes a gyroscope and / or an accelerometer 316 for providing information about the current orientation of the device. In some embodiments, the client 300 also includes a GPS 320 for determining and providing location information about the client device. The client device may include a sensor 318 for sensing movement of a user (or other object) near the client device 300, for sensing sound, and / or for sensing ambient light proximate to the client device. Memory 312 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 312 optionally includes one or more storage devices remote from the CPU(s) 302. Memory 312, or alternatively, non-volatile memory devices(s) within memory 312, includes non-transitory computer-readable storage media. In some embodiments, memory 312 or a computer-readable storage medium of memory 312 stores the following programs, modules, and data structures, or a subset thereof:

[0058] • Operating system 322, which includes processes for handling various basic system services and performing hardware-related tasks.

[0059] A network communication module 324 that connects the client 300 to other computers via the one or more communication network interfaces 310 (wired or wireless) and one or more communication networks (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0060] One or more device applications 325, which are used to control various functions on the device, such as:

[0061] o A lighting controller application 101 that exists only in some embodiments, which receives information from lighting devices and sensors and sends commands to lighting devices 110 via an application program interface (API) 204. In some embodiments, the lighting controller also predicts the user's tasks, however, in other embodiments, the lighting controller application 101 only receives information from other lighting devices 110 and sends some or all of this information to the server's lighting application for analysis operations (such as user task prediction);

[0062] o Device status module 326, which provides information about on / off status, brightness / darkness of the screen, current user activity (e.g., cell phone, email, or other Internet activity);

[0063] o a display screen control module 328 for receiving instructions from the server 300 and implementing the instructions by controlling the display 306 as instructed (e.g., implementing a lighting plan by displaying colors, information, arrows, etc.);

[0064] o a clock 330 for tracking the current time and providing the current time to the server 300;

[0065] o Calendar module 322, which is used to track user appointments and calendar events;

[0066] o an email module 334 for sending and receiving email messages;

[0067] o a handset module 336 for voice and text messaging;

[0068] o an Internet browsing module 338 for connecting to Internet pages to search, browse, and perform social networking activities;

[0069] o Device data 340, such as device capabilities and other stored data; and

[0070] o Display characteristics 342, such as orientation, number of pixels, color range, etc.

[0071] Each of the elements identified above is typically stored in one or more of the previously mentioned memory devices and corresponds to a set of instructions for performing the functions described above. It is not necessary to implement the modules or programs (i.e., sets of instructions) identified above as separate software programs, processes, or modules, and therefore, each subset of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 312 stores a subset of the modules and data structures identified above. In addition, memory 312 may store additional modules and data structures not described above.

[0072] although Figure 2 A "server system 200" is shown and Figure 3 A "client system 300" is shown, but the purpose of these figures is more of a functional description of the various features present in the server collection rather than a schematic diagram of the structure of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, in Figure 2 Some items shown individually in the figure may be implemented on a single server and a single item may be implemented by one or more servers. The actual number of servers used to implement server system 200 and how features are distributed among these servers will vary from one embodiment to another and will generally depend in part on the amount of data traffic that the system must handle during peak usage periods and during average usage periods.

[0073] Figure 4 406 is a block diagram illustrating a data structure of a lighting map 400 associated with a user according to some embodiments. In some embodiments, the lighting map 400 includes at least a device name 402 or a device ID or other mechanism for identifying a specific device associated with a user. The location 404 of the device (e.g., GPS location) is included. Some devices, such as televisions, clocks, and digital photo frames, remain relatively static. Before utilizing other devices (such as mobile phones, tablet computers, and wearable devices (e.g., smart watches)), they should be queried / confirmed to meet the user's lighting needs. The orientation 406 of the device can also be recorded. In some embodiments, the orientation includes the direction in which the display screen of the device is facing. In some embodiments, the state 408 of the device (e.g., on / off) is also recorded, as is the degree of brightness / darkness. Additional information 410 can also be stored in the lighting map.

[0074] Figure 51 is a flow chart representing a method 500 for providing adaptive lighting and predictive lighting according to some embodiments of the present invention. The method 500 is generally governed by instructions stored in a computer-readable storage medium and executed by one or more processors. The method occurs at a computer system having one or more processors and a memory storing programs executed by the one or more processors. For example, the lighting method 500 can be executed by the lighting application 102 ( 102 ) at the server 200, the centralized lighting controller 101 in the room, the user device, or some combination of the three. Figure 1A , 1B ) to execute.

[0075] A lighting map of lighting devices within a predefined area is accessed (502). In some embodiments, the locations of the lighting devices in the lighting map are described with reference to rooms of a structure occupied by a user. Each lighting device has one or more lighting characteristics, such as an on / off state and intensity. In addition, the lighting devices can be controlled via one or more networks, such as Bluetooth technology, Wi-Fi, and cellular networks. In some embodiments, for each lighting device, the accessed map indicates at least the location and state of the lighting device. In some embodiments, the map also indicates the orientation of the device.

[0076] The lighting device includes one or more non-conventional lighting devices (504). For example, the non-conventional lighting device is typically a multifunctional light-emitting device. In some embodiments, each non-conventional lighting device is a networked electronic device with a controllable light-emitting display screen. In some embodiments, the non-conventional lighting device is selected from: a television, a video projector, a smart phone, a tablet computer, a computer monitor, a laptop computer, an audio component, or an appliance. In other embodiments, each non-conventional lighting device is a networked electronic device with a controllable lighting element. For example, it can also be a power transformer, a charging device, or a power socket with a controllable status light.

[0077] Real-time contextual information is obtained for the user (506). For example, the information is obtained from a set of networked sensors. In some embodiments, the contextual information includes the current user location within a predefined area, the time of day, and one or more indications of user movement, sounds proximate to the user's location, ambient light levels proximate to the user's location, calendar events for the user, and / or device interactions by the user, and / or web interactions by the user (508).

[0078] Based on the context information, the user task and the lighting requirements associated with the user task are inferred (510). For example, the user task may be to wake up at a specific time. Other tasks include: walking somewhere, reading, sleeping, watching TV, using a computer, writing, drawing, cooking, eating, etc.

[0079] In addition, based on the user location, a first subset of lighting devices located proximate to the user location is determined from the lighting map (512). In some embodiments, at least one of the first subset of lighting devices is responsive to a command sent according to an API configured to specify lighting characteristics (including lighting on / off state, and one or more of lighting hue, saturation, and brightness). In addition, the subset of lighting devices includes at least a first non-conventional lighting device (514). In some embodiments, the first non-conventional lighting device is configured to execute an operating system that can control display content and lighting characteristics (including display on / off state, and one or more of lighting hue, saturation, and brightness) for a plurality of addressable pixels of the display.

[0080] Then, the lighting characteristics of the subset of lighting devices are controlled via one or more networks to meet the user's lighting needs (516). As explained in the previous figures, the control of the lighting devices is performed by the server lighting application 102, the lighting controller 101 at the user's location, or a combination of the two. The lighting devices are controlled during a predefined time period. For example, as described above in the night light and alarm clock example, the light from various lighting devices will be illuminated at different times and the intensity will vary over time to meet the user's lighting needs. In some embodiments, controlling the lighting characteristics of the lighting devices includes controlling the display content of at least one of the unconventional lighting devices to provide a meaningful indicator to the user related to the inferred task. For example, the meaningful indicator may include a text message, a graphical display, or a direction indicator. In some embodiments, an accompanying audio message may also be provided. For example, the meaningful indicator may guide the user to a destination, provide an alarm sound, provide information about an upcoming meeting, etc.

[0081] In some embodiments, the inferred task involves the user participating in a specific activity at a specific place. For example, the inferred task may include: the user is sleeping at night, getting up in the morning, eating in the kitchen, watching TV, or reading in the living room. For such a task, controlling the lighting characteristics of a subset of lighting devices includes the following steps. First, the user's current location is identified. A lighting map is consulted to identify a first set of lighting devices (including at least one unconventional lighting device) that are close to the user's current location. Based on the user's current location, a command is sent to a corresponding lighting device in the first set of lighting devices to instruct the first set of lighting devices to provide an illumination with lighting characteristics consistent with the inferred task and context information. For example, if the user is going to sleep (e.g., determined by the controller, indicating that it is late at night, there is no router activity, the user is turning off the bedside lamp, and there is a rustling sound in the bed), the lighting controller identifies the illuminated lights (including any device screens and illuminated power status lights) close to the user and sends commands to those devices via one or more computer networks to dim those lights to a level consistent with the user's preferences (e.g., from complete darkness to a dimly lit place with a tranquil color (e.g., dim blue or purple)). The controller may also identify a lighting device that can be used as a night light (if the user desires to use the lighting device as a night light according to the user preferences), and then send a command to the device via one or more computer networks to set the illumination of the device to a level consistent with the user preferences. Note also that any lighting plan implemented by the lighting controller may include a time element that allows the controller to gradually dim or brighten the lights in the room over time.

[0082] In some embodiments, the inferred task involves a user moving from a current location (e.g., a first room) to a destination (e.g., a second room). For example, the user movement may occur in a dark environment and therefore require path guidance. When the task involves the user moving from the current location to the destination along the path, a subset of lighting devices along the path is controlled based on updated information about the user's current location along the path (518). In some embodiments, controlling the lighting characteristics of the subset of lighting devices includes the following steps. First, the user's current location is identified. A physical path for the user to the destination in the context of a lighting map is also identified. The lighting map is consulted to identify a first set of lighting devices (including at least one unconventional lighting device) that are close to the user's current location, the user's physical path, and the destination. Then, by using an environmental sensor, the user's current location is sensed and updated. Based on the user's current location, a command is sent to each lighting device in the first set of lighting devices to instruct the first set of lighting devices to provide lighting from the current location along the path to the destination, and the lighting characteristics are consistent with the inferred task and context information.

[0083] In some embodiments, a lighting pattern is prepared as follows. A network command is sent to one or more lighting devices to cause each of the one or more lighting devices to perform a respective identifiable sequence of lighting events. For example, a first example sequence for a first device may involve blinking a status light on and off according to a repeating short, short, long sequence. A second example sequence for a second device is a slow dimming of the status light followed by three quick blinks. Positioning is performed using a camera in the context of the lighting pattern position of the corresponding lighting device by identifying the identifiable sequence of lighting events. In other words, the position of the first device is determined using the camera by identifying the first sequence.

[0084] In some embodiments, in addition to or as an alternative to the video-based techniques discussed above, non-video-based positioning techniques are used to identify the location of lighting devices, especially those that are highly mobile (such as mobile phones or tablet computers). For example, in some embodiments, audio and / or radio frequency triangulation is used to provide a location map of the corresponding lighting devices based on information received from multiple other static devices (such as routers, televisions, and smart thermostats, etc.).

[0085] In some embodiments, the server lighting application 102 requires prior user consent (explicit or implicit) for access to the user's lighting devices and sensors. Therefore, only information authorized by the user can be used for predictive lighting. This protects user privacy and complies with relevant privacy laws or contractual provisions. Therefore, with the viewer's consent (e.g., by virtue of user settings in the home lighting controller 101 application or prior explicit consent to send data from their lighting devices and sensors to the server), any private activity data is used in creating predictive lighting.

[0086] In addition, in some embodiments, the type of each lighting device is identified by using visual matching technology using a database of camera images and images of lighting devices (including images of unconventional lighting devices). In other embodiments, the device is identified by querying. For example, querying networked devices to obtain information that identifies their types, and then storing the received information about the device type. In some embodiments, after determining the type of the device, accessing the database of lighting devices to determine the lighting characteristics of the identified lighting device. In other embodiments, the lighting characteristics of the identified and located lighting device are determined by recording the lighting response of the lighting device to the lighting command sequence. One benefit of recording the lighting response is that the system can then take into account the current orientation of the device. For example, whether the tablet device has a screen facing downward (and therefore it is rarely used or not used to meet the identified lighting needs associated with the inferred user task). Similarly, in some embodiments, the lighting characteristics of the identified and located lighting device are regularly determined by recording the lighting response of the lighting device to the lighting command sequence.

[0087] Thus, in some embodiments, the system determines whether the lighting demand is met by controlling the lighting characteristics of a first set of lighting devices. Then, in response to determining that the lighting demand is not met, the lighting characteristics of a second set of lighting devices are controlled. In other words, an alternative lighting plan is developed based on the use of a different set of lighting devices to meet the lighting demand.

[0088] Figure 5 The various operations shown generally correspond to instructions stored in a computer memory and a non-transitory computer-readable storage medium. The computer-readable storage medium generally includes a magnetic or optical disk storage device, a solid-state storage device (such as flash memory), or (multiple) other non-volatile memory devices. The computer-readable instructions stored in the computer-readable storage medium are source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Specifically, in Figure 5 Many of the operations shown in Figure 2 20 , the context information determination module 222 , the task prediction / task inference module 224 , and the instructions in the lighting control / lighting response module 226 of the server system 200 shown in FIG.

[0089] Figure 6is a schematic diagram / blueprint of an example suite inhabited by a user according to some embodiments. In some embodiments, the inferred user task involves the user moving from a current location (e.g., first room 602) to a destination (e.g., second room 604). For example, the user movement may occur in a dark environment and therefore require path guidance. When the task involves the user moving from the current location 602 to the destination 604 along the path 606, the lighting devices 110 along the path are controlled based on the updated information about the user's current location along the path. Specifically, according to Figure 5 As described in operation 518 , controlling lighting characteristics of a subset of lighting fixtures 102 is performed.

[0090] The terms used in the description of the embodiments herein are only for describing specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, unless the context clearly indicates otherwise, the singular forms "one", "an", and "the" are also intended to include plural forms. It is also to be understood that the words "and / or" used herein represent and encompass any and all possible combinations of one or more of the associated listed items. It is further to be understood that the words "comprises" and / or "comprising", and the words "includes" and / or "including", when used in this specification, refer to the existence of the features, wholes, steps, operations, elements and / or parts described, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their groups.

[0091] It is also understood that although the words "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without changing the meaning of the description, as long as all occurrences of the first element are consistently renamed and all occurrences of the second element are consistently renamed. The first element and the second element are two elements, but they are not the same element.

[0092] As used herein, the word "if" may be interpreted as "when..." or "when..." or "in response to...", depending on the context. Similarly, the phrase "if it is determined that..." or "if (the stated condition or event) is detected..." may be interpreted as "upon determining that..." or "in response to determining that..." or "upon detecting (the stated condition or event)..." or "in response to detecting (the stated condition or event)...", depending on the context.

[0093] For the purpose of illustration, the above description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments have been selected and described in order to best illustrate the principles of the invention and its practical application, so that those skilled in the art can best utilize the invention and the various embodiments, and various modifications may be made according to the specific use envisioned.

Claims

1. A method performed at a computer system having one or more processors and a memory storing a program executed by the one or more processors, the method comprising: receiving real-time current context information of a user in a home environment from a plurality of devices and networked sensors via one or more networks, wherein the real-time current context information includes at least one of a scheduled event or a sensed motion, In response to the real-time current context information: Inferring a task of the user and a lighting requirement associated with the task, wherein inferring the task comprises predicting the task using a task prediction algorithm; determining a lighting response for the task that will satisfy the lighting need, wherein determining the lighting response for the task comprises combining the task with lighting rules and machine learning information, and wherein each lighting response comprises a model that outputs recommended lighting at different proximity to the user over time; and identifying a subset of a plurality of lighting fixtures from a lighting map based on proximity of the subset to the user to implement the lighting response; and Lighting characteristics of the subset are controlled via the one or more networks to achieve the lighting response.

2. The method according to claim 1, wherein: The real-time current context information further includes at least one of sounds proximate to the user's location, ambient light levels proximate to the user's location, device interactions by the user, and web interactions by the user.

3. The method according to claim 1, wherein: The subset includes non-conventional lighting devices.

4. The method according to claim 3, wherein: The non-conventional lighting devices include: televisions, video projectors, smart phones, tablet computers, computer monitors, laptop computers, audio components or electrical appliances.

5. The method according to claim 3, wherein: The non-conventional lighting device includes a controllable lighting element.

6. The method according to claim 5, wherein: The controllable lighting element is selected from: a power transformer, a charging device, a power socket with a controllable status light, a power switch adapter and a light socket adapter.

7. The method according to claim 3, wherein: The non-conventional lighting fixtures respond to commands sent via the one or more networks according to an API configured to specify lighting characteristics, including lighting on / off status, and one or more of lighting hue, saturation, and brightness.

8. The method according to claim 3, wherein: Controlling the lighting characteristics includes controlling display content of the non-conventional lighting device to provide meaningful indicators to the user, wherein the display content is relevant to the task, and wherein the meaningful indicators include text messages or directional indicators.

9. The method according to claim 1, wherein: Accessing a lighting map identifying a plurality of lighting fixtures in a home environment includes, for each of the lighting fixtures, identifying a location of the lighting fixture with reference to a room in the home environment.

10. The method according to claim 1, wherein: The plurality of lighting devices include a plurality of unconventional lighting devices, each of the unconventional lighting devices includes a display screen, and the method further includes: For each of the non-conventional lighting devices, information of one or more of a screen size, a direction the screen faces, a screen type, and a current screen on / off status is received via the one or more networks.

11. The method according to claim 1, further comprising: Determining that one or more of the plurality of lighting devices are devices running on a common operating system and coordinating with one or more of the plurality of lighting devices using data formats and commands associated with the common operating system, wherein the one or more networks include a local area network and the Internet.

12. A centralized lighting controller, comprising: one or more processors; as well as A memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 11.

Images