SYSTEM AND METHOD FOR PROVIDING IMMERSIVE LIGHTING INTERACTION - Patent application
The system uses sensor data from client devices to control lighting infrastructure, addressing the need for intuitive, immersive, and group-based lighting control beyond traditional GUIs, enhancing user interaction and engagement.
Patent Information
- Application Number
- JP2022508835
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-10-07
- Filing Date
- 2020-08-04
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2040-08-04
AI Technical Summary
Existing lighting control systems require users to interact with graphical user interfaces (GUIs) or preprogrammed sequences, distracting them from their environment and limiting interactive control options.
A system that uses sensor data from client devices, such as motion sensors, to control lighting infrastructure without the need for GUI interaction, allowing immersive and group-based lighting experiences.
Enables users to control lighting infrastructure intuitively through motion and group behavior, providing an immersive experience without the need for traditional input mechanisms, enhancing user engagement and interaction.
Smart Images

Figure 0007583511000001 
Figure 0007583511000002 
Figure 0007583511000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a system and method for controlling a lighting infrastructure by one or more users based on sensor data received by the users' mobile devices. [Background technology]
[0002] A lighting infrastructure, such as a bridge, a building, a monument, or other large-scale structure, is composed of multiple lighting devices. A lighting controller communicates with the lighting infrastructure to control each lighting device. To control the lighting infrastructure, a user manipulates the controls using traditional input mechanisms such as a touch screen or other cursor on the client device. This requires the user to look at a user interface and manipulate one or more user interface controls. This may involve a graphic user interface (GUI) that includes buttons, sliders, switches, checkboxes, menus, or text fields that allow the user to enter their input controls. A disadvantage of this approach is that it requires the user to divert attention from other activities and focus on the GUI. If the user is forced to focus on looking at the user interface of the client device, the user cannot enjoy his or her environment. There are systems that address the situation for controlling the lighting infrastructure that do not require the user to look at a GUI.
[0003] The prior art lacks the ability to allow a user to control a lighting infrastructure without using a GUI. For example, prior art systems allow a user to reserve a time slot for operating the lighting infrastructure. Once a reservation is made, the user can use a mobile application installed on a client device to operate a touch screen to change the lighting controls of the lighting infrastructure, e.g., the color or intensity of the lights. Specifically, the user can select by touching a particular color displayed on a GUI generated by the mobile application and control the lighting infrastructure to display the selected color. Such a solution requires the user to operate a GUI to control the lighting infrastructure.
[0004] Other prior art systems include lighting infrastructures controlled by pre-programmed animation sequences, where no user input is required. Rather, specific sequences stored in a computer memory are read and converted into lighting signals to control the lighting infrastructure. In other prior art, more advanced lighting controls are based on real-time data, including, for example, weather data. Weather data obtained over a network is converted into corresponding lighting controls. However, such solutions do not allow for user interaction. Summary of the Invention [Problem to be solved by the invention]
[0005] The present disclosure overcomes the problems of the prior art by providing an immersive lighting experience where no GUI is used to control the lighting infrastructure, and includes embodiments that allow group behavior to control the lighting infrastructure. [Means for solving the problem]
[0006] One aspect of the invention relates to a client device configured to communicate with a lighting controller that controls a lighting infrastructure. The client device controls the lighting infrastructure via a computing system, such as a server. The client device includes a processor, a sensor, and a communication interface configured to communicate with the computing system. The client device also includes a memory that stores computer instructions executable by the processor. The client device uses these instructions to receive sensor samples from the sensor. The client device then classifies a subset of the sensor samples to generate a classification of the sensor samples. The client device maps the classification to a corresponding control instruction and sends the control instruction to the computing system. The computing system is configured to send control signals to the lighting controller to control the lighting infrastructure according to the control instructions.
[0007] According to one embodiment, the sensor includes a motion sensor generating sensor samples, which may be acceleration samples or gyroscope samples, and the classification further includes a motion classification specifying a type of motion.
[0008] According to another embodiment, the sensor is a wireless accessory of the client device, hi another embodiment, the client device includes a watch, a smartphone, or a wearable device.
[0009] According to another embodiment, a client device includes a long short term memory (LSTM) classifier including a plurality of cells, each cell configured to receive a corresponding sensor sample from a subset of the sensor samples.
[0010] Another aspect of the invention relates to an immersive group lighting experience in which group behaviors are translated into control signals to control a lighting infrastructure. According to one embodiment, a computing system is configured to receive a plurality of classifications from a plurality of client devices, determine whether the received types of classifications exceed a threshold amount, and send control signals to a lighting controller to control a lighting infrastructure according to the control signals corresponding to the received types of classifications. [Brief description of the drawings]
[0011] Further details, aspects and embodiments of the present invention are described, by way of example only, with reference to the following drawings, in which elements are illustrated for simplicity and clarity and are not necessarily drawn to scale, and in which elements corresponding to elements already described may have the same reference numerals. [Figure 1] 1 illustrates a schematic diagram of an example embodiment of an immersive lighting system; [Diagram 2] 2 illustrates a schematic diagram of an example of communication between a client device and a computing system in the embodiment of the immersive lighting system of FIG. 1; [Diagram 3] 2 illustrates various data transmitted and calculated in the embodiment of the immersive lighting system of FIG. 1; [Figure 4] 2 illustrates an example of a classifier in the embodiment of the immersive lighting system of FIG. 1. [Diagram 5] 2 illustrates data used to analyze group behavior in the embodiment of the immersive lighting system of FIG. 1; [Figure 6] 2 illustrates a schematic diagram of an example of communication between a lighting controller and a computing system in the embodiment of the immersive lighting system of FIG. 1; [Figure 7a] 2 is a flow chart illustrating example functions performed by a client device in the embodiment of the immersive lighting system of FIG. 1. [Figure 7b] 2 is a flowchart illustrating example functions performed by a computing system in the embodiment of the immersive lighting system of FIG. 1. [Figure 8] 2 illustrates a schematic diagram of an embodiment of a computing system in the immersive lighting system of FIG. 1; [Figure 9] 2 illustrates a schematic diagram of an embodiment of a client device in the immersive lighting system of FIG. 1; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] The embodiments illustrated in the drawings and described in detail herein are to be considered as illustrative of the principles of the invention and are not intended to limit the invention to the specific embodiments shown and described herein.
[0013] In the following, for ease of understanding, elements of the embodiments are described in terms of operation, however, it will be apparent that each element is configured to perform the functions that are described as being performed by it.
[0014] Moreover, the invention is not limited to those embodiments, but the invention resides in each and every novel feature or combination of features described herein or recited in mutually different dependent claims.
[0015] FIG. 1 illustrates a schematic diagram of an example of an embodiment of an immersive lighting system. FIG. 1 illustrates a networked environment including a lighting infrastructure 100 controlled by a lighting controller 103. The lighting infrastructure 100 may be a bridge, a building, a monument, a stadium, an arena, or other large-scale structure. The lighting infrastructure 100 is composed of a number of lighting devices. The lighting devices include one or more light sources, such as, for example, light-emitting diodes (LEDs). The lighting devices can be grouped together and controlled as a group. The lighting devices are installed in the lighting infrastructure 100 to provide lighting on a large scale. The lighting controller 103 may be a device that controls the lighting devices installed in the lighting infrastructure. In this regard, the lighting controller 103 communicates with the lighting devices of the lighting infrastructure using any communication protocol.
[0016] The lighting controller 103 communicates with other system components via a network 105. The network 105 may include the Internet, an intranet, an extranet, a wide area network (WAN), a local area network (LAN), a wired network, a wireless network, or other suitable network, or any combination of two or more such networks.
[0017] Further, the immersive lighting system of FIG. 1 includes a computing system 106 connected to the network 105. The computing system 106 may include, for example, a server computer or any other system providing computing power. Alternatively, the computing system 106 may employ multiple computing devices that may be configured, for example, in one or more server or computer banks or other arrangements. Such computing devices may be located in a single installation or distributed in many different geographic locations. For example, the computing system may include multiple computing devices that together may include hosted computing resources, grid computing resources and / or any other distributed computing arrangement. In some cases, the computing system 106 may correspond to elastic computing resources, where the allotted capacity of processing, network, storage, or other computing-related resources may change over time. The computing system 106 may implement one or more virtual machines that use the resources of the computing system 106.
[0018] The computing system 106 includes a database 109. Various data are stored in the database 109 or other memory accessible to the computing system 106. The database 109 may represent one or more databases 109. The data stored in the database 109 includes user accounts 112, session data 115, group behavior data 118, and lighting maps 120. The user accounts 112 include information about users, including usernames, passwords, authentication information, and / or login credentials. The purpose of the user accounts 112 is to register users who intend to access the immersive lighting system to control the lighting infrastructure 100. In this regard, the user accounts 112 provide information for authenticating and authorizing users who wish to access the system. The session data 115 includes information about established active sessions with the immersive lighting system. The group behavior data 118 includes real-time information about multiple users for making decisions related to how to control the lighting infrastructure 100. The lighting map 120 includes information that clusters one or more LEDs of the lighting infrastructure 100 into corresponding cluster identifiers such that each LED can be controlled independently. For example, a control signal that controls the lighting infrastructure 100 to apply only to a cluster identifier thereby independently controls one or more LEDs.
[0019] Various applications and / or other functions may execute on the computing system 106, including, for example, a client interface application 124 and a controller interface application 127. The client interface application 124 provides an interface to client devices, as described below. The controller interface application 127 provides an interface to the lighting controller 103.
[0020] Also, as mentioned above, multiple client devices 134 operated by different users are connected to the network 105. The client devices 134 are portable so that they can be carried by the users and easily transported to the lighting infrastructure 100. The client devices may be, for example, mobile phones, tablets, wearable devices (e.g., wristwatches), or any other portable mobile devices that function as clients. The client devices 134 include a sensor 137, a classifier 141, and a communication interface 144. The sensor may be a wireless accessory of the client device 134 or may be integrated into the housing of the client device 134. The sensor may be a motion sensor, a heat sensor, a microphone, a light sensor, a camera, or a biometric sensor. The classifier 141 may include a software component that analyzes the sensor data and classifies it based on detecting patterns. The communication interface 144 allows the client devices 134 to communicate with the computing system 106 over the network 105. The client device 134 is configured to control the lighting infrastructure 100 by communicating with a computing system, which communicates with the lighting controller 103 .
[0021] We now describe how the various components of Fig. 1 are configured to realize a group lighting system. A user using a client device 134 wants to interact with the lighting infrastructure 100. For example, if the lighting infrastructure is installed in a stadium hosting sporting events, the user may want to control the stadium lighting to reflect the user's mood. Depending on how the user moves the client device, the lighting infrastructure 100 changes the lighting recipe. Different movements correspond to different lighting control instructions or different lighting animations, e.g., changing color, hue, intensity, brightness, etc. Allowing the user to control the lighting infrastructure 100 without using traditional inputs, such as touch screen controls, realizes an immersive experience.
[0022] A session is established between the computing system 106 and the user, and the occurrence of the session is stored as session data 115. The session may be established by the client device 134 requesting authorization. The computing system 106 may invite the client device 134 to make a request.
[0023] According to an embodiment of the present invention, once the session is established, the client device 134 is configured to send control instructions for controlling the lighting infrastructure 100, the control instructions being derived from sensor data. In this case, the sensor 137 of the client device 134 generates sensor samples. These sensor samples are analyzed by the classifier 141 to generate classifications. The types of classifications are mapped to corresponding control instructions. The communication interface 144 of the client device 134 sends the control instructions to the client interface application 124 of the computing system 106. The computing system 106 interprets the control instructions and generates corresponding control signals. The controller interface application 127 of the computing system 106 sends the control signals to the lighting controller 103 so that one or more LEDs of the lighting infrastructure 100 are controlled according to the control instructions originating from the client device 134.
[0024] According to another embodiment, the immersive experience may be an immersive group experience where group behavior dictates how the lighting infrastructure 100 is controlled. Synchronous group behavior may trigger lighting changes in the way the lighting infrastructure 100 displays light.
[0025] In this embodiment, the computing system 106 communicates with a number of client devices 134. Each client device 134 includes a sensor that generates sensor data. The sensor data from each client device 134 is classified using a respective classifier 141. The classifications produced by the classifiers 141 are transmitted to the client interface application 124. These classifications are stored as group behavior data 118. The computing system 106 analyzes the group behavior data to determine whether a threshold amount of classifications of the sensor data from the multiple client devices is met. For example, the classifiers 141 may be configured to classify the motion sensor data into a number of classifications, one of which may include a motion of standing up. If the quantification of the types of classifications for the motion of “standing up” received by the computing system 106 exceeds a threshold amount, the computing system may send a corresponding control signal to the lighting controller such that the lights of the lighting infrastructure 100 are controlled accordingly. In this regard, classification may be quantified by the number of classifications of a particular type received in a period of time, the proportion of classifications of a particular type based on all classifications received, or the frequency, periodicity, or rate at which a particular type of classification is received in a period of time.
[0026] Functions described as being performed locally (e.g., on the client device 134) may be distributed to be performed remotely (e.g., by the computing system 106). For example, the classifier 141 may be implemented on the computing system 106 instead of on the client device 134. In this case, the client device 134 sends the sensor data to the computing system 106 for subsequent classification. Alternatively, classification of the sensor data may be performed initially by the client-side classifier 141 and then completed by a classifier resident on the computing system 106.
[0027] Fig. 2 shows a schematic diagram of an example of communication between a client device 134 and a computing system 106 in the immersive lighting system of Fig. 1. The client device 134 sends a control request 201 to the computing system 106. According to an embodiment, the control request 201 includes user data 203 and location data 206. The user data 203 may include information identifying a user of the client device 134, such as, for example, a user identifier. The location data 206 may include location coordinates of the client device 134 that issued the control request 201. For example, the client device 134 may include a location module that obtains location coordinates or a relative position of the client device 134. The location module may be a Global Navigation Satellite System (GNSS) module that obtains geographic coordinates. The location module may be configured to detect a distance or a relative position with respect to the lighting infrastructure 100. When issuing the control request 201, the client device 134 accesses the location module to embed the location coordinates or the location data in the control request 201. Although FIG. 2 shows the client device 134 sending the request 201 , the control request 201 may also be solicited by the computing system 106 .
[0028] When the computing system 106 receives the control request 201, the computing system 106 may authenticate the request. For example, the computing system 106 cross-references the user account 112 and the user data 203 to determine whether the client device 134 that issued the control request 201 is associated with an authorized user. In this manner, the computing system 106 obtains information about the user that issued the control request 201.
[0029] Upon authenticating the user, the computing system 106 sends a permission 209 to the client device 134. The permission 209 authorizes the client device 134 to issue subsequent control commands 212 to the computing system 106. By granting the permission 209 to the client device, a session is established and the session data 115 is updated accordingly. During the session, one or more control commands 212 are issued by the client device 134. Alternatively, when using group behaviors to control the lighting infrastructure 100, the computing system 106 may derive one or more control commands 212 based on classifications of sensor data received from multiple client devices 134. As described in more detail below, the control commands 212 are derived from the sensor data as opposed to a user manipulating a GUI.
[0030] For example, depending on how the sensor 137 senses the user moving the client device 134, a corresponding control command 212 is generated. For example, moving the sensor 137 upward increases the intensity and moving the sensor 137 downward decreases the intensity. The sensor 137 may generate a motion sensor output that is classified by the classifier 141 to generate the corresponding control command 212.
[0031] In another embodiment, the sensor is a camera and the control commands 212 are derived from the output of the camera. For example, a facial or pattern recognition program may be applied to a digital image obtained from the camera to generate corresponding control commands 212. For example, a smile may correspond to a turn-on-light control command and a frown may correspond to a turn-off-light control command.
[0032] The client device 134 may issue one or more control commands 212 as long as the session persists. In one embodiment, the user may end the session and send a terminate session message 215 to the computing device. After the session is terminated, the computing system 106 will not process subsequent control commands 212 from the client device 134 unless a new session is established.
[0033] Figure 3 illustrates various data that are transmitted and calculated in the embodiment of the immersive lighting system of Figure 1. Figure 3 illustrates an embodiment of a client device 134 that receives sensor data from a sensor 137 to generate classifications 312 and corresponding control instructions 212.
[0034] The sensors 137 convert physical inputs, such as light, motion, heat, or electricity, into computer-readable sensor data. The sensor data may be formatted as a series of sensor samples 308, with each sample time including one or more sensor readings. The embodiment of FIG. 3 illustrates a motion sensor 137 that periodically generates motion sensor samples 308. The motion sensor samples 308 may include accelerometer readings along a particular axis (e.g., x-axis, y-axis, z-axis) and / or gyroscope readings along a particular axis (e.g., x-axis, y-axis, z-axis).
[0035] The client device 134 includes a classifier 141 that receives sensor data, such as a stream of sensor samples 308. The classifier 141 is configured to analyze the sensor samples 308 and generate a classification 312. In this regard, the classifier 141 determines whether the sample data matches one of a finite set of classifications. In the case of a motion sensor 137, the classifier 141 may be configured to determine whether the sensor samples 308 correspond to a motion classification 312, such as upward motion, downward motion, shaking motion, clockwise rotation, counterclockwise rotation, sideways motion, etc. The example of FIG. 3 illustrates six different classifications, A-F, of the sample data. If the sample data does not match a predefined classification 312, the classifier 141 determines that no classification exists for the analyzed sample data.
[0036] The client device 134 then maps the determined classification 312 to a corresponding control command 212. According to one embodiment, the client device 134 stores a map that converts the classification 312 to the control command 212. In this regard, classification A is mapped to control command A, classification B is mapped to control command B, and so on. By way of example, classification A may be an upward movement and classification B may be a downward movement. Control command A may be an instruction to increase the light intensity and control command B may be an instruction to decrease the light intensity. If the sensor moves upward, an instruction to increase the light intensity is generated, and if the sensor moves downward, an instruction to decrease the light intensity is generated. The user may manipulate the client device 134 to generate a control command that correlates with the movement.
[0037] Figure 4 illustrates an example of a classifier 141 in the embodiment of the immersive lighting system of Figure 1. Figure 4 illustrates an embodiment in which the classifier 141 is implemented as a Long Short-Term Memory (LSTM) classifier 141. The LSTM classifier 141 is trained with training data consisting of a set of training samples.
[0038] The LSTM classifier 141 is composed of multiple LSTM cells 403a-n. The first LSTM cell feeds its output to the next LSTM cell in the sequence. Each LSTM cell 403 receives as raw input a corresponding sensor sample 308 from the sequence of sensor samples 308. If the LSTM classifier 141 is configured to analyze 10 samples (e.g., n=10) to make a classification decision, then 10 LSTM cells 403 are used, with each LSTM cell receiving a corresponding sensor sample 308. The number of LSTM cells (n) may be selected based on the longest time series in the training data.
[0039] The output of the LSTM classifier 141 is analyzed to generate a probability that the input sensor sample falls into each classification. As shown in FIG. 4, each classification 312, labeled A-F, has a corresponding probability score. The probability score indicates how likely the sensor sample 308 input into the classifier 141 falls into the classification 312. FIG. 4 shows an example where classification F has the highest probability score of 89%, while the other classifications 312 have significantly smaller scores. If the probability score exceeds a threshold score, the corresponding classification 312 is applied to the input sensor sample 308.
[0040] Figure 5 illustrates data used to analyze group behavior in an embodiment of the immersive lighting system of Figure 1. Figure 5 illustrates an embodiment in which a computing system 106 is configured to analyze group behavior to create an immersive group lighting experience. In this embodiment, the computing system 106 communicates with multiple client devices 134 operated by respective users 504. Each client device 134 includes a sensor 137 that transmits sensor data to a classifier 141 resident on the client device 134. The classifier 141 generates a classification 312.
[0041] After the client device 134 determines the classification 312, the classification is sent to the computing system 106. The example of FIG. 5 shows five users. Users 1, 2, and 3 each independently generate a classification 312 for their respective sensor data, and they all correspond to the same classification 312, which is labeled Classification A. This means that users 1, 2, and 3 exhibited similar behaviors captured by their respective sensors 137. User 4 generates a different classification 312, which is labeled Classification D. This means that user 4's behavior is different from users 1-3. User 5 does not generate a classification. This may be because there is no sensor data to analyze or the analyzed sensor data does not fall into the classification 312.
[0042] As shown in FIG. 5, the classifications 312 are received asynchronously from different users over a varying time period. The classifications 312 are received and stored as group behavior data 118. The group behavior may be collected and analyzed for a varying time window. For a particular time window, the computing system 106 records the classifications received from multiple users 504 and stores the classifications as group behavior data 118. This group behavior data 118 is analyzed to provide metrics such as the frequency of a particular type of classification. In the example of FIG. 5, for a particular time window, classification A occurred three times, classifications B and C did not occur, and classification D occurred once. The group behavior data 118 may also express the received classifications as percentages to indicate their relative frequency. Here, classification A occurred 75% of the time out of all classifications 312 received for the particular time window. The computing system 106 generates a control signal based on the group behavior data 118. For example, the control signal may be mapped to classification A, which has the highest relative frequency. In other embodiments, a threshold is applied to generate the control signal. For example, if a classification type exceeds a threshold frequency, a control signal mapped to that classification type is generated. If the threshold frequency is 70%, a control signal mapped to classification A may be generated since the frequency of classification A is 75%. As another example, if the occurrence of a particular classification type exceeds a threshold count, a corresponding control signal is generated. If the threshold count is 10, no control signal is generated if the count of the most frequent classification type is below 10. The use of a threshold amount (e.g., threshold frequency or threshold count) can encourage more users to interact with the immersive lighting system or modify how they interact with the client device 134.
[0043] To further elaborate on this example, the computing system 106 determines whether the type of classification received exceeds a threshold amount. For example, the threshold amount may be a predetermined number of a particular classification type received within a time window. The threshold may also be defined as a percentage. For example, the threshold amount may be characterized as whether at least 50% of the classifications 312 received are of a particular type.
[0044] If the received classification type exceeds a threshold amount, the computing system 106 determines a corresponding control signal that is mapped to the classification type, and then the computing system 106 sends the control signal to the lighting controller 103 to control the lighting infrastructure 100 according to the control signal.
[0045] FIG. 5 illustrates an embodiment that may take place in a sports arena. Each client device 134 may classify movements according to a finite set of classifications 312. One classification 312 may correspond to a "stand up" movement where the user stands up, and another classification may correspond to a "sit down" movement where the user sits down. Upon detecting these classified movements, the client device 134 transmits the classifications 312 to the computing system 106. The computing system 106 receives and stores the classifications 312 as group behavior data 118. If the number of classifications of a particular type exceeds a threshold amount, a corresponding control signal is generated. For example, if the group behavior suggests that many people are standing up, the corresponding control signal may be to play a predefined animation sequence, increase the light intensity, or change the color output. This may encourage users to participate more as a group to affect how the lighting infrastructure 100 outputs light.
[0046] Fig. 6 shows a schematic example of communication between the lighting controller 103 and the computing system 106 in the embodiment of the immersive lighting system of Fig. 1. The computing system 106 may store a statically or dynamically generated lighting map 120. The lighting map is sent to the lighting controller 103 so that the lighting controller 103 can map received control signals 605 to specific lights or clusters of lights. For example, if the lighting infrastructure 100 is installed in a stadium that is divided into 20 sections, the lighting map 120 may organize each light in the lighting infrastructure 100 into a designated section. This allows the lighting infrastructure 100 to have lights that are independently controlled. If there is an update to the lighting map 120, the updated lighting map 120 is sent to the lighting controller 103.
[0047] As previously discussed, the computing device 106 receives the control instructions 212 from a particular client device 134 and generates a corresponding control signal 605 to direct the lighting infrastructure 100 in accordance with the control instructions 212. In this regard, the control signal 605 is a transmission that embodies the control instructions 212 originating from the client device 134. The control signal 605 may also be generated by the computing system 106 based on the group behavior data 118.
[0048] In addition to specifying how to control the lights, the control signal 605 includes an identification of the lights to be controlled. Upon receiving the control signal 605, the lighting controller 103 uses the lighting map 120 to apply the control signal 605 to the lights identified in the control signal.
[0049] 7a is a flow chart illustrating an example of functions performed by the client device 134 in the embodiment of the immersive lighting system of FIG. 1. At 716, the client device 134 receives the sensor samples 308. The sensor 137 may be built into the client device 134 or may be an accessory to the client device 134. At 719, the client device 134 inputs the sensor samples 308 to the classifier 141. The sensor samples may be input as a continuous stream such that the classifier 141 listens continuously for sensor data. Alternatively, the sensor samples 308 are sent to the classifier 141 in response to a trigger. The trigger may be, for example, a button, a voice command, or other user initiated instruction.
[0050] At 722, the client device 134 determines a classification 312. A classifier 141 implemented on the client device 134 analyzes the sensor sample 308 to determine the extent to which the sensor sample 308 matches a predefined classification 312. If a classification 312 can be made, at 725, the client device 134 determines a control command based on the classification 312. For example, the client device 134 maps the classification 312 to a control command 212. In this regard, each classification type corresponds to a certain control command 212.
[0051] At 728, the client device 134 sends the control instructions 212 to the computing system 106. The computing system 106 receives the control instructions 212 and generates corresponding control signals 605 to control the lighting infrastructure 100 in accordance with the control instructions 212.
[0052] Figure 7b is a flow chart illustrating an example of functions performed by the computing system 106 in the embodiment of the immersive lighting system of Figure 1. Figure 7b illustrates an embodiment for implementing an immersive group lighting experience. At 745, the computing system 106 receives classifications 312 from multiple client devices 134. The classifications 312 may be stored as group behavior data 118. Each classification, at an individual level, represents a particular sensor pattern.
[0053] When analyzing the aggregation of sensor patterns, the computing system may generate a corresponding control signal 605. At 748, the computing system determines whether a threshold amount of classifications is exceeded. For example, if the number of a particular type of classification 312 exceeds a threshold number or percentage (compared to other types of classifications received), at 751, the computing system 106 determines a corresponding control signal 605. The control signal includes instructions indicating how the lights in the lighting infrastructure 100 should be controlled. At 755, the computing system 106 transmits the control signal to the lighting infrastructure.
[0054] Fig. 8 illustrates a schematic of one embodiment of the computing system 106 in the immersive lighting system of Fig. 1. The computing system 106 includes one or more computing devices 800. Each computing device 800 includes at least one processor circuit, e.g., having a processor 803, a memory 806, and a communication interface 809, each of which is coupled to a local interface 812 or bus. Each computing device 800 may include, e.g., at least one server computer or similar device. The communication interface 809 may include hardware, e.g., a network interface card, a modem, a transceiver, or a radio, and / or software, e.g., a software module that encodes / decodes communication packets for transmission and reception. The local interface 812 may include, e.g., a data bus 812 with an associated address / control bus or other bus structure as may be understood.
[0055] Stored in the memory 806 are both data and several components executable by the processor 803. Among other things, stored in the memory 806 and executable by the processor 803 are the client interface application 124 and the controller interface application 127. Also stored in the memory 806 may be a database 109 and other data. Additionally, an operating system may be stored in the memory 806 and executable by the processor 803.
[0056] The communication interface 809 is configured to communicate with the lighting controller 103 and the plurality of client devices 134. The processor 803 uses the communication interface to establish communication with components external to the computing system 106. For example, the processor 803 may send instructions to the communication interface 809 to cause transmission of data to the lighting controller 103 or the client devices 134. Similarly, data received from the communication interface 809 is forwarded to the processor 803.
[0057] Figure 9 illustrates generally one embodiment of a client device 134 in the immersive lighting system of Figure 1. The client device 134 includes a processor 903, a memory 906, and a communication interface 144, each of which is coupled to a local interface 912 or bus. Additionally, the client device 134 includes a sensor 137. The sensor 137 may be a wired, wireless, or integrated component with respect to the client device 134.
[0058] The communications interface 144 may include hardware, such as, for example, a network interface card, modem, transceiver, or radio, and / or software, such as, for example, software modules that encode / decode communication packets for transmission and reception. The local interface 912 may include, for example, a data bus with an associated address / control bus or other bus structure as may be appreciated.
[0059] Stored in the memory 906 are both data and certain components executable by the processor 903. Among other things, stored in the memory 906 and executable by the processor 903 is a classifier 141. Additionally, an operating system or firmware may be stored in the memory 906 and executable by the processor 903.
[0060] The communication interface 144 is configured to communicate with the computing system 106. The processor 903 uses the communication interface to establish communication with components external to the computing system 106. For example, the processor 903 may send instructions to the communication interface 144 to cause the transmission of control instructions 212 or classifications 312. Similarly, data received from the communication interface 144 is forwarded to the processor 903.
[0061] With respect to both Figures 8 and 9, it should be understood that there may be other applications stored in memory 806, 906 and executable by processor 803, 903 as may be understood. If any of the components described herein are implemented in the form of software, any of a number of programming languages may be employed, such as, for example, C, C++, C#, Objective C, Java, JavaScript, Perl, PHP, Visual Basic, Python, Ruby, Flash, or other programming languages.
[0062] Some software components are stored in the memory 806, 906 and are executable by the processor 803, 903. In this respect, the term "executable" refers to a program file in a form that can ultimately be executed by the processor 803, 903. An example of an executable program may be, for example, a compiled program that can be loaded into a random access portion of the memory 806, 906 and converted into machine code in a format that can be executed by the processor 803, 903, source code that can be represented in a suitable format such as object code that can be loaded into a random access portion of the memory 806, 906 and executed by the processor 803, 903, or source code that can be interpreted by another executable program to generate instructions in the random access portion of the memory 806, 906 to be executed by the processor 803, 903, etc. The executable program may be stored in any portion or component of the memory 606, including, for example, random access memory (RAM), read-only memory (ROM), a hard drive, a solid-state drive, a USB flash drive, a memory card, an optical disk such as a compact disk (CD) or digital versatile disk (DVD), a floppy disk, a magnetic tape, or other memory component.
[0063] Memory 806, 906 is defined herein to include both volatile and non-volatile memory and data storage components. A volatile component is one that does not retain a data value upon loss of power. A non-volatile component is one that retains data upon loss of power. Thus, memory 806, 906 may include, for example, random access memory (RAM), read-only memory (ROM), a hard disk drive, a solid state drive, a USB flash drive, a memory card accessed via a memory card reader, a floppy disk accessed via an associated floppy disk drive, an optical disk accessed via an optical disk drive, a magnetic tape accessed via a suitable tape drive, and / or other memory components, or a combination of any two or more of these memory components. Further, RAM may include, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM), and other such devices. The ROM may include, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other similar memory device.
[0064] Also, the processor 803, 903 may represent multiple processors 803, 903 and / or multiple processor cores, and the memory 806, 906 may represent multiple memories 806, 906, respectively, operating in parallel processing circuits. In such a case, the local interface 812, 912 may be a suitable network facilitating communication between any two of the multiple processors 803, 903, between any processor 803, 903 and any of the memories 806, 906, or between any two of the memories 806, 906, etc. The local interface 812, 912 may include additional systems designed to regulate this communication, including, for example, performing load balancing. The processor 803, 903 may be an electrical or other available structure.
[0065] The software applications or programs described herein may be embodied in software or code executed by general purpose hardware as described above, but may alternatively be embodied in dedicated hardware, or a combination of software / general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each may be implemented as a circuit or state machine employing any of several technologies or a combination thereof. These technologies may include, but are not limited to, discrete logic circuits having logic gates for performing various logical functions upon application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field programmable gate arrays (FPGAs), or other components, etc. Such technologies are generally well known to those skilled in the art and therefore will not be described in detail herein.
[0066] The foregoing detailed description has set forth some of the many forms that the present invention can take. The above examples are merely illustrative of some possible embodiments of the various aspects of the present invention, and equivalent alternatives and / or modifications will occur to those skilled in the art upon reading and understanding the present invention and the accompanying drawings. In particular, with respect to the various functions performed by the above-described components (devices, systems, etc.), the terms used to describe such components (including references to "means") are intended, unless otherwise indicated, to correspond to any component, such as hardware or combination of hardware, that performs the specified function of the described component (i.e., is functionally equivalent) even if it is not structurally equivalent to the disclosed structure that performs that function in the illustrated implementation of the disclosure.
[0067] Although certain features of the invention may be illustrated and / or described with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be advantageous and desirable for a given or particular application. Furthermore, reference to a singular component or item is intended to encompass two or more such components or items unless otherwise specified. Also, to the extent that the terms "comprises," "includes," "has," "having," "having" or variations thereof are used in the detailed description and / or claims, such terms are intended to be encompassed in an analogous manner to the term "comprising."
[0068] The present invention has been described with reference to preferred embodiments. However, modifications and variations will occur to those skilled in the art upon reading and understanding the foregoing detailed description. It is intended that the present invention be construed to include all such modifications and variations. It is only the claims, including all equivalents, that are intended to define the scope of the present invention.
[0069] In the claims, reference signs in parentheses refer to the reference signs in the drawings of exemplary embodiments or formulas of embodiments to enhance the clarity of the claims, and should not be construed as limiting the claims.
Claims
1. A mobile client device adapted to be carried by a user and configured to communicate with a computing system, the computing system transmitting control signals to a lighting controller for controlling a lighting infrastructure, the mobile client device comprising: A processor; A sensor; a communications interface configured to communicate with a computing system; A memory storing computer instructions that, when executed, cause the processor to: receiving a sensor sample from the sensor; classifying the subset of sensor samples to generate a classification of the sensor samples; and mapping the classification to corresponding control instructions for controlling the lighting infrastructure and sending the control instructions to a computing device, the computing device sending control signals to a lighting controller according to the control instructions; or transmitting the classifications to a computing system, the computing system receiving a plurality of classifications from a plurality of mobile client devices, selecting a classification type from the received plurality of classifications based on a relative frequency or a threshold amount of the classification type occurring in the received plurality of classifications, and transmitting a control signal to a lighting controller according to the classification type. Memory, , including mobile client devices.
2. The mobile client device of claim 1, wherein the classification corresponds to a classification type among a plurality of classification types.
3. The mobile client device of claim 1 , wherein the sensor includes a motion sensor, the sensor samples include acceleration samples or gyroscope samples from the motion sensor, and the classification includes a motion classification that specifies a type of motion.
4. The mobile client device of claim 1 , wherein the sensor is a wireless accessory of the mobile client device.
5. The mobile client device of claim 1 , wherein the mobile client device comprises a wristwatch.
6. The mobile client device of claim 1 , wherein the computer instructions include an LSTM classifier including a plurality of cells, each cell configured to receive a corresponding sensor sample from the subset of sensor samples.
7. The mobile client device of claim 6, wherein the output of the LSTM classifier is analyzed to generate a classification type probability score for each of a plurality of classification types, the classification type probability score indicating the likelihood that a subset of the sensor samples falls into that respective classification type.
8. A lighting infrastructure control system for controlling a lighting infrastructure, the lighting infrastructure control system comprising: A plurality of mobile client devices, each according to any one of claims 1 to 7; communicating with each of the plurality of mobile client devices to receive control instructions or classifications from each mobile client device; and sending a control signal to a lighting controller in accordance with the received control command; or receiving a plurality of classifications from the plurality of mobile client devices, selecting a classification type from the received plurality of classifications based on a relative frequency or a threshold amount of the classification type occurring in the received plurality of classifications, and transmitting a control signal to a lighting controller according to the classification type. a computing system configured to a lighting controller configured to communicate with the computing system to receive control signals for controlling the lighting infrastructure and to control the lighting infrastructure; Lighting infrastructure control systems, including:
9. An immersive lighting system comprising the lighting infrastructure control system of claim 8 and a lighting infrastructure.
10. 1. A computer-implemented method for controlling a lighting infrastructure, the method comprising: receiving a sensor sample from a sensor of a mobile client device carried by a user; classifying, by the mobile client device, the subset of sensor samples to generate a classification of the sensor samples; mapping, by the mobile client device, the classification to corresponding control instructions for controlling the lighting infrastructure; transmitting, by the mobile client device, the control instructions to a computing device; determining, by the computing device, a control signal in accordance with the control instructions; and sending, by the computing device, the control signal to a lighting controller for controlling the lighting infrastructure; or transmitting, by the mobile client device, the classification to a computing system; receiving, by the computing system, a plurality of classifications from a plurality of mobile client devices; selecting, by the computing system, a classification type from among the received plurality of classifications based on a relative frequency or a threshold amount of that classification type occurring in the received plurality of classifications. determining, by a computing device, a control signal corresponding to the classification type; and sending, by the computing system, a control signal to a lighting controller for controlling the lighting infrastructure; and A method comprising:
11. The computer-implemented method of claim 10 , wherein the classifying step includes generating a classification corresponding to a classification type among a plurality of classification types.
12. the sensor includes a motion sensor; the receiving step includes receiving acceleration samples or gyroscope samples from the motion sensor; The computer-implemented method of claim 10 , wherein the classifying step includes assigning a type of movement.
13. The computer-implemented method of claim 10 , wherein the sensor is a wireless accessory of the mobile client device.
14. The computer-implemented method of claim 10, wherein the steps performed by the mobile client device are performed on a wristwatch.
15. 11. The computer-implemented method of claim 10, wherein the classifying step is performed by an LSTM classifier including a plurality of cells, each cell configured to receive a corresponding sensor sample from the subset of sensor samples, and the computer-implemented method includes analyzing an output of the LSTM classifier to generate a probability score for each of a plurality of classification types, the probability score for the classification type indicating a likelihood that the subset of sensor samples falls into the respective classification type.
Citation Information
Patent Citations
Method, apparatus, and system for occupancy sensing
EP3517839A1
Light control method and lighting device using the light control method
JP2014535154A
Method for controlling a lighting device
JP2017507453A
Systems and methods for probabilistic semantic sensing in sensor networks
JP2017518587A
Control of Lighting
JP2017528888A
