Controller for sensing in an environment via first and second sensing systems and method thereof

Abstract

A method of sensing in an environment via first and second sensing systems, in which a first controller and a second controller are arranged to control the first and second sensing systems, respectively, and in which the first sensing system has a different type than the second sensing system and has a sensing range at least partially overlapping the second sensing system, wherein the first controller is further arranged for controlling a first actuation system that may affect sensing from the second sensing system at least in a sub-region of the environment, and wherein the method comprises: receiving, at the first controller, a request for interaction from the second controller, wherein the request includes adjusting actuation of the first actuation system to minimize an effect of the first actuation system on sensing of the second sensing system, assigning a priority value to the received request indicating a priority level of the received request, and determining whether to accept the request to adjust the actuation based on the assigned priority value.

Description

Technical Field

[0001] This invention relates to a method for sensing in an environment via first and second sensing systems. The invention also relates to a controller, system, and computer program product for sensing in an environment via first and second sensing systems. Background Technology

[0002] Smart home systems are deployed to monitor and / or control home properties such as lighting, climate, entertainment systems, and appliances. Smart systems include sensing, actuation, and / or control functions to describe and analyze conditions within the environment. Sensing enables the system to “see” its environment, machine learning / AI algorithms bring “intelligence that thinks / makes decisions based on what it sees,” and actuation elements provide the means to influence the environment based on “intelligent” decisions made by machine learning / AI algorithms.

[0003] When connected to the internet, smart (home) devices are an important component of the Internet of Things (IoT). While using the convention of smart “home” systems, the concept of connected / smart systems is also applied to other non-home environments, such as offices, retail stores, hospitals, public squares, stadiums, etc. Summary of the Invention

[0004] The inventors have recognized that interactions between different intelligent systems in a multi-intelligent-system environment are constantly increasing and evolving. Due to these increasing interactions, the sensing and actuation of different systems are interconnected and can influence each other (positively or negatively). Because these intelligent systems aim to improve the user experience in the environment, sensing and actuation (which is primarily based on context-aware insights derived from sensing data) to provide the best user experience in a multi-intelligent-system environment becomes challenging.

[0005] Therefore, the object of the present invention is to improve sensing in view of the interaction with other actuation / sensing systems in a multi-intelligent system environment.

[0006] According to a first aspect, the objective is achieved by a method of sensing in an environment via first and second sensing systems, wherein a first controller and a second controller are arranged to control the first and second sensing systems respectively, and wherein the first sensing system has a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system, wherein the first controller is also arranged to control a first actuation system that can affect the sensing from the second sensing system at least in a sub-region of the environment, wherein the method includes: receiving at the first controller a request for interaction from the second controller, wherein the request includes adjusting the actuation of the first actuation system to minimize the influence of the first actuation system on the sensing of the second sensing system, assigning a priority value to the received request indicating a priority level of the received request, and determining whether to accept the request to adjust the actuation based on the assigned priority value.

[0007] This method involves sensing within an environment. This environment can be an indoor environment, such as a home, hotel, nursing home, hospital, office, grocery store / shopping mall, entertainment center, etc.; or an outdoor environment, such as a street, parking lot, public square, urban landscaping system, stadium, etc.

[0008] Sensing is performed via first and second sensing systems, wherein a first controller and a second controller are arranged to control the first and second sensing systems respectively. The sensing systems may include, but are not limited to, presence sensors, light sensors, humidity sensors, air quality sensors (CO, pollutants, etc.), motion sensors, occupancy sensors (infrared (IR), passive infrared (PIR), ultrasonic, etc.), thermal sensors, electromagnetic sensors (e.g., radio frequency based sensing, radar sensing, WiFi Doppler), structured light sensors (e.g., ToF), lidar sensors, acoustic sensors, air quality sensors (CO, pollutants, etc.), video (security cameras, etc.), audio (microphones, etc.), etc.

[0009] The first sensing system is of a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system. In one example, the different sensor types include different sensing mechanisms for the same sensing application. For example, the first sensing system is associated with presence sensing via vision-based sensing, and the second sensing system is associated with the same presence sensing application, but via a different sensing mechanism (such as radio frequency-based sensing). In another example, the first sensing system is associated with gesture detection sensing via radar sensing, and the second sensing system is associated with the same gesture sensing application, but via a different sensing mechanism (such as WiFi radio frequency-based sensing). In yet another example, heat emitted from a phase-cut dimmer or high-power lighting device may negatively impact the sensing performance of a PIR or thermopile sensor due to its current lighting task. In yet another example, the different sensor types include sensing with a modified set of parameters of the same sensing mechanism. In yet another example, the different sensor types may include different sensing mechanisms for different sensing applications, such as presence sensing in the environment and temperature sensing.

[0010] A first controller is also arranged to control a first actuation system. The first actuation system may include attributes such as lighting, climate, entertainment systems, screens, projectors, TVs, and appliances such as HVAC systems, connected lighting systems, audio / video systems, blinds, etc. A second controller may be arranged to control a second actuation system. In one example, the first actuation system and the first sensing system are co-located. In another example, the first actuation system and the first sensing system are included in the same device or the same housing; for example, actuation and sensing are performed by the same device.

[0011] The first actuation system can influence sensing from a second sensing system, at least within a sub-region of the environment. For example, an illumination actuation system (e.g., lighting equipment arranged to illuminate the environment) can influence a vision-based sensing system (e.g., a camera-based sensing system). Based on the field of view of the sensing system and the range or field of view of the actuation system, the effect of actuation on sensing can be limited to a sub-region of the environment or extended to the entire environment. This effect can be negative, causing the actuation to impede the sensing capability of the sensing system, or positive, causing the actuation to help improve the sensing capability of the sensing system. The sub-region of the environment can (at least partially) overlap with a sensing range that at least partially overlaps.

[0012] Similarly, EMI from lamps or lighting control devices can interfere with the electronics of nearby radar sensors. For example, a WiFi-controlled phase-cut dimmer installed in a 2-gang concealed wiring box arrangement, adjacent to a flush-mounted non-lighting UI with radar sensing, may cause EMI interference to the radar sensor.

[0013] The method includes receiving a request at a first controller to interact with a second controller, wherein the request includes adjusting the actuation of a first actuation system to minimize the impact of the first actuation system on the sensing of a second sensing system. This request can be received from the second controller and / or (directly) from a user. The user can send the request using the second controller or using a user device (such as a mobile phone, tablet, etc.). The second controller can be configured to determine the (negative) impact of the first actuation system, and then the first controller can receive the request from the second controller to adjust the actuation such that the negative impact of the actuation can be minimized / eliminated. The second controller can request adaptation of the actuation within a specific range of a sub-region or over the entire area of ​​interference.

[0014] The method also includes assigning a priority value to the received request, which indicates the priority level of the received request. Because the method also includes determining whether to accept a request for adjusted actuation based on the assigned priority value, the prioritization approach minimizes the (negative) impact of actuation on the sensing system (e.g., interference suffered by a non-illuminated sensing system due to actuation of the illumination function of an illuminated sensing system), thus improving sensing in light of interactions with other actuation / sensing systems in a multi-intelligent system environment.

[0015] If the determination leads to acceptance of the request, the method may further include adjusting the actuation of the first actuation system based on the request.

[0016] The adjustment is performed to reduce the impact of the first actuation system on the sensing of the second sensing system. In one example, the adjustment is determined by the first controller based on a request and / or by the second controller. Therefore, by adjusting the actuation, sensing is further improved, considering interactions with other actuation / sensing systems in a multi-intelligent system environment.

[0017] If the determination results in the rejection of the request, the method may further include requesting the second controller to switch the sensing task of the second sensing system at least partially to the first sensing system in the at least partially overlapping sensing range.

[0018] The first controller can, for example, decide not to accept an adjustment request from the second controller if it deems the actuation more important or at least equally important than the sensing of the second sensing system. In this case, the first controller can provide the second controller with an opportunity to switch the sensing to the first sensing system controlled by the first controller. This switching involves taking over sensing from the second sensing system. In this example, the first and second sensing systems include the same sensing application but with different sensing mechanisms (e.g., presence sensing via vision-based sensing and presence sensing via radio frequency-based sensing; or gesture detection via radar sensing and gesture detection via radio frequency-based sensing). Because the sensing in the sub-region affected by the actuation of the second sensing system is taken over by the first sensing system, the overall sensing in the environment is improved.

[0019] If the determination results in the rejection of the request, the method may further include requesting the second controller to perform joint sensing within at least partially overlapping sensing ranges.

[0020] Attached to or replacing switching sensing, the first controller may require the second controller to perform joint sensing within at least partially overlapping sensing ranges. The final sensing result of the sensing event may include a fusion of the sensing results from the two sensing systems.

[0021] The first controller may be arranged to control a lighting system including at least one lighting device for providing illumination in an environment and a radio frequency-based sensing device, and wherein the second controller may be arranged to control a vision-based sensing system, wherein the radio frequency-based sensing has a sensing range that at least partially overlaps with the vision-based sensing, and wherein the illumination is capable of affecting the vision-based sensing system at least in a sub-region of the environment.

[0022] In a further developed embodiment, the actuation of the lighting equipment can be temporarily reduced to minimize interference from the second sensing system. For example, if the task of the radar or vision sensor is to occasionally detect the user's seated body posture, it can adjust the dynamic lighting scheme of the wall washer to reduce occasional light output and thus lower emitted EMI, thereby maximizing the sensing performance of the radar sensor in the other sensing system.

[0023] The first automatic system may include a lighting system, and a first controller may be arranged to control the lighting system. The first sensing system may include a radio frequency (RF) based sensing system. RF-based sensing is a sensing mechanism involving a wireless transceiver (or transmitter / receiver) arranged to transmit and receive radio frequency (RF) signals. These RF signals (which can also be used for radio communications) are affected by the presence / movement of a person within the sensing volume, for example, through reflection, absorption, scattering, etc. RF-based sensing uses this deviation of the RF signals to infer the presence / movement of a person. RF-based sensing also extends to other applications (such as location detection, fall detection, gesture detection, vital sign detection, etc.) that also rely on how RF signals are affected within the sensing volume. RF-based sensing may further include radar-based sensing. In one example, the first sensing system may include a time-of-flight (ToF) sensing system.

[0024] A lighting system may include at least one lighting device for providing illumination in an environment. In one example, the lighting system is a connected lighting system such that the lighting devices and / or other devices in the lighting system (such as lighting controllers, flush-mounted phase-cut dimmers, adapters, smart sockets, etc.) are equipped with wireless transceivers (or transmitters / receivers) for communicating with each other and / or with a first controller. In this example, RF signals transmitted between devices in the lighting system (such as at least one lighting device) can be used for radio frequency-based sensing. In this example, the actuation system and the sensing system are co-located, for example, within the same lighting device.

[0025] The second sensing system can be a vision-based sensing system, such as a camera-based sensing system. The radio frequency-based sensing can have a sensing range that at least partially overlaps with the vision-based sensing, and wherein illumination can affect the vision-based sensing system at least in a sub-region of the environment.

[0026] The method may also include assigning priority values ​​based on one or more of the following: the request for interaction, the characteristics of the requested interaction with the second controller, the characteristics of the second controller, and the number and / or type of sensing devices of the second sensing system.

[0027] The method may also include assigning a confidence value that indicates the confidence level of the second controller, and a priority value may be assigned based on the confidence value.

[0028] Trustworthiness values ​​can be determined based on at least one or more of the following: previous interactions with the first controller, security certificates of the second controller, etc. Priority values ​​can be advantageously assigned based on trustworthiness values; for example, a known, trusted second controller can be assigned a higher value compared to an unknown, suspicious second controller.

[0029] The method may also include assigning priority values ​​based on sensing tasks and / or sensing results from the second sensing system. For example, sensing tasks that are also valuable for the actuation and / or sensing of the first controller can be assigned higher priority values. Additionally or alternatively, sensing tasks that are critical to the user's needs / health (such as sensing related to health parameters, such as fall detection, vital sign detection, gait analysis, etc.) can be assigned higher priority values. Even for the same sensing task, sensing results can be assigned different sensing priority values; for example, a fall detection event can be assigned a higher priority value compared to no fall or near-fall event.

[0030] The method may also include assigning priority values ​​based on contextual information related to the environment. For example, contextual information may include the type of environment, such as indoor or outdoor environment, objects in the environment, whether a second controller with a sensing system is placed in a specific area of ​​the environment, the size of the environment, etc.

[0031] The method may further include determining whether to accept the request for adjustment actuation based on the user's past interactions or preferences with the first and second controllers.

[0032] For example, due to privacy concerns, a user might prefer radio frequency-based sensing over vision-based sensing, or the user might have experienced numerous false alarms from sensing systems compared to others, or the user might prefer a WiFi sensing system (which uses only the WiFi band) from a health perspective compared to other sensing systems (such as radar systems using the 60 GHz mm band). In such scenarios, the user could input their preferences or indicate any (good / bad) past interactions.

[0033] The decision to accept the request for adjustment actuation can be further based on the sensing performance of the first and second sensing systems, at least in a sub-region of the environment.

[0034] Sensing performance (e.g., fewer false alarms / missed alarms, higher resolution, better range, health preference, etc.) is a key parameter that can be included in the criteria used to determine whether to accept a request to adjust the actuation of the first actuation system. For example, if the first sensing system has poor sensing performance, the first controller will adjust the actuation of the first actuation system so that the second sensing system can continue sensing without any performance degradation. Alternatively, if the second sensing system has poor sensing performance, the first sensing system can request the second controller to switch sensing tasks or perform joint sensing via the first controller.

[0035] In this example, sensing performance can be evaluated based on the number of sensing nodes, the historical performance of the first and second sensing systems, the layout of sub-regions of the environment, physical objects in the sub-regions of the environment, and sensing interference in the sub-regions of the environment.

[0036] According to the second aspect, the objective is achieved by a controller for sensing in the environment via first and second sensing systems, wherein the first controller and the second controller are arranged to control the first and second sensing systems respectively, and wherein the first sensing system has a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system, wherein the first controller is also arranged to control a first actuation system that affects the sensing from the second sensing system at least in a sub-region of the environment, wherein the controller includes a processor arranged to perform the steps of the method according to the first aspect (or at least control the execution of the steps of the method according to the first aspect).

[0037] According to the third aspect, the objective is achieved by a system for sensing in an environment via first and second sensing systems, wherein a first controller and a second controller are arranged to control the first and second sensing systems respectively, and wherein the first sensing system has a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system, wherein the first controller is also arranged to control a first actuation system that affects the sensing from the second sensing system at least in a sub-region of the environment, wherein the system includes at least the first controller and the controller according to the second aspect.

[0038] According to the fourth aspect, this objective is achieved by a computer program product including instructions that, when executed by a computer, cause the computer to perform the steps of the method of the first aspect. This objective can also be achieved by a computer program product including instructions that, when executed by a processor of a controller, cause the processor to perform the steps of the method of the first aspect.

[0039] It should be understood that computer program products, controllers, and systems may have similar and / or the same embodiments and advantages as the methods described above. Attached Figure Description

[0040] Referring to the accompanying drawings, the above and additional objects, features, and advantages of the disclosed systems, devices, and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of the systems, devices, and methods, in which: Figure 1 An embodiment of a system for sensing in the environment via first and second sensing systems is illustrated schematically and exemplary. Figure 2An embodiment of a controller for sensing in the environment via first and second sensing systems is illustrated schematically and exemplary. Figure 3 A flowchart illustrating an embodiment of a method for sensing in the environment via first and second sensing systems is shown schematically and exemplary.

[0041] All figures are schematic and not necessarily to scale, and generally only show the parts necessary to illustrate the invention, where other parts may be omitted or merely suggested. Detailed Implementation

[0042] Figure 1 Embodiments of a system 100 for sensing in environment 101 via a first sensing system 110a-d and a second sensing system 120 are illustrated schematically and exemplary. In the exemplary figures, the first sensing system includes a radio frequency (RF) based sensing system performed by the lighting devices 110a-d. In one example, RF signals transmitted between the (connected) lighting devices 110a-e can be used for RF-based presence sensing, motion sensing, and other sensing applications. In another example, a radar sensor (24 GHz radar, UWB radar, 60 GHz radar, WiFi Doppler radar) can be integrated into the lighting devices 110a-e for the presence and motion of user 130 in environment 101, and for other sensing applications in environment 101. In yet another example, a time-of-flight (TOF) sensor can be integrated into the lighting devices 110a-e. These sensing modes can coexist or exist individually in the lighting devices 110a-e.

[0043] In one example, user device 131 (such as a mobile phone, tablet, etc.) may be part of the sensing system. Other sensing applications may involve healthcare applications, such as fall detection, vital sign monitoring, etc. In another example, other sensor types (such as PIR, acoustic sensors, vibration sensors, etc.) are embedded in lighting devices 110a-d. In one example, actuation of lighting devices 110a-e may be based on the monitored sensing task.

[0044] Lighting fixtures 110a-d are devices or structures arranged to emit light suitable for illuminating environment 101, providing or substantially contributing to illumination of a scale sufficient for that purpose. Lighting fixtures 110a-d include at least one light source or lamp, such as an LED-based lamp, a gas discharge lamp, or an incandescent bulb, (or optionally) plus any associated bracket, housing, or other such enclosure. Each of the lighting fixtures 110a-d can take any of a variety of forms, such as a ceiling light, a wall-mounted light, a wall washer, or a freestanding light (and the light fixtures do not necessarily have to be of the same type). In this exemplary figure, lighting fixtures 110a-c are ceiling light fixtures, and lighting fixture 110d is a floor lamp. Any number and type of lighting fixtures 110a-d can be present in environment 101.

[0045] For the connected lighting system (connected lighting devices 110a-d), the lighting devices 110a-d may further include a wireless transceiver or transmitter / receiver (not shown) for wireless communication according to a wireless communication protocol (such as Wi-Fi, Bluetooth, Zigbee, Thread, etc.). In one example, radio frequency signals transmitted for network communication can be used for RF sensing. Additionally or alternatively, additional radio frequency signals may be transmitted / received for the purpose of radio frequency-based sensing.

[0046] The second sensing system 120 is exemplarily shown as a vision system 120. The vision system 120 includes a camera for sensing tasks. This camera may be a 2D camera, a stereo camera, or a depth-sensing (ranging) camera (e.g., a time-of-flight camera). Sensing tasks for the first and second sensing systems may involve sensing characteristics of the user 130 in environment 101 or characteristics of environment 101 / the atmosphere in environment 101. Sensing tasks may include presence detection, motion detection, activity detection, interaction detection between objects, interaction detection between a person and an object, fall detection, vital sign detection, atmospheric condition detection, etc.

[0047] The first controller 110 and the second controller 120 can be respectively arranged to control the first sensing systems 110a-d and the second sensing system 120. In the exemplary figure, the first controller 110 is a lighting controller and is shown external to the lighting devices 110a-d, while the second controller 120 is shown integrated into the second sensing system 120. This choice is not unique, and the first controller 110 can be integrated into one or more lighting devices 110a-d, and / or the second controller 120 can be external to the second sensing system 120.

[0048] The first sensing system 110a-d may be of a different type than the second sensing system 120. For example, the first sensing system 110a-d may include a radio frequency-based sensing system arranged to sense, for example, the presence of user 130 in environment 101. The second system 120 may also be arranged to sense the presence of user 130, but uses a vision-based sensing system (such as camera 120) for presence sensing. The first sensing system 110a-d and the second sensing system 120 may have at least partially overlapping sensing ranges, such that within the at least partially overlapping sensing ranges, both sensing systems, for example, detect the presence of user 130.

[0049] The first controller 110 can also be arranged to control a first actuation system that can influence sensing from a second sensing system, at least in a sub-region of the environment. In the exemplary figures, the actuating devices are lighting devices 110a-d. In one example, the first sensing systems 110a-d and the first actuation systems 110a-d are co-located. Sensing and actuation can be performed by the same device (e.g., lighting devices 110a-d) (e.g., sensing via an integrated wireless transceiver or transmitter / receiver). In other words, the sensing element (e.g., a wireless communication module) and the actuation element (e.g., a light source) can be contained in the same device or housing.

[0050] Light emitted by lighting devices 110a-b can affect a second sensing system, such as vision-based system 120. When lighting devices 110a-d do not provide sufficient illumination to light the environment 101, the performance of camera sensing 120 may be severely degraded. Alternatively, glare, shadows, or high-intensity light can also negatively impact the sensing performance of camera sensing 120. Illumination effects can also affect the performance of camera sensing by causing people to move away from the camera (e.g., an elderly person will not sit there if the area within the camera's field of view is only dimly illuminated).

[0051] It is known that lighting infrastructure introduces EMI into the built environment, depending on the installation environment and the product. In many commercially available sensor designs, the impact of EMI on sensing performance is overlooked. For example, as touch sensing devices become increasingly thinner (with fewer or thinner substrates) and capacitive touch sensing technology is being implemented in more and more new environments, new challenges arise for the robustness of touch sensors. It is known that many high-sensitivity touch sensors face the challenge of insufficient EMI (electromagnetic interference) immunity levels.

[0052] Capacitive touch displays can be mounted on walls, close to high-power lighting fixtures, to provide upward lighting effects for ceilings or high-intensity colored wall grazing light effects. This specific touchscreen design, deployed in a room, can exhibit low immunity to EMI noise. Capacitive touch sensors are susceptible to conductive coupling (which occurs when the light and the victim touch display are connected in direct contact via wires). Conducted EMI is naturally the most significant source of noise for capacitive touch sensors.

[0053] In addition, touch sensors may be affected by the following: inductive coupling (which occurs when, for example, a changing magnetic field of an LED driver causes a voltage change on the victim's touch display); capacitive coupling (which occurs when a changing electric field causes a voltage change on the victim's touch display); and radiative coupling (which occurs when the distance between the LED driver and the victim's capacitive display is typically greater than one wavelength and the LED light emits electromagnetic waves that are received by the victim's LED display, which acts as an antenna).

[0054] It is well known that WiFi sensing can perform fine-grained gesture recognition. For example, when a user is simply typing on a keyboard with their fingers using a commercial WiFi chip, WiFi sensing can recognize the typed letters from a distance. In one example, when a first lighting system receives a request to adjust the actuation of the lighting system (e.g., to minimize EMI radiation from the lighting fixtures and thereby improve the touch / gesture sensing accuracy of a nearby capacitive touchscreen display), a second controller at least partially switches between the user's hand gesture detection and the WiFi sensing performed by the light. This switching can be a complete switch, such that the user's hand is detected entirely by WiFi sensing, rather than by touch / gesture detection performed via capacitive sensors and / or radar sensors on the LCD display.

[0055] In another example, the first sensing system can be flush-mounted in a first concealed wiring box, and the second sensing system can be flush-mounted in a second concealed wiring box. The first and second concealed wiring boxes can be arranged in a duplex configuration. Therefore, the first and second sensing systems are very close to each other and may therefore interfere with each other. For example, the first sensing system could be embedded in a phase-cut dimmer module, which could lead to EMI or harmonic interference with the second sensing system, especially when the phase-cut dimmer is operating at low light levels. Note that the phase-cut dimmer module is characterized by WiFi radio for home automation but can also perform WiFi sensing. The second sensing system could be a radar system, which is susceptible to interference from the EMI emissions of the phase-cut dimmer system.

[0056] In a further developed embodiment, actuating the lighting device at maximum light output level could involve actuating an active cooling device integrated into the lighting device, such as a fan integrated into a high-intensity spotlight. However, the noise from the active cooling device will degrade the audio sensing of another sensing system located near the lighting device. Similarly, the fan may cause mechanical vibrations that affect non-lighting sensing systems.

[0057] Similarly, whenever mechanical stress relaxation occurs due to temperature changes in lighting equipment, changes in the light output of the lighting equipment can lead to mechanical stress in the lighting system, resulting in audio noise (e.g., crackling noise) or vibration events. These audio / vibration events can interfere with other sensing systems. Likewise, thermal gradients caused by actuated lighting equipment can lead to mechanical stress at the location of another sensor, which may degrade the sensing performance of non-lighting sensing systems located near the lighting equipment.

[0058] Figure 2 An embodiment of controller 210 is illustrated schematically and exemplary for sensing in the environment via first sensing systems 110a-d and second sensing system 120. In one example, controller 210 is included in first controller 110, or vice versa; for example, first controller 110 is controller 210, and the designations for controller 210 and first controller 110 may be used interchangeably. In another example, controller 210 may be included in second controller 120, or vice versa.

[0059] Controller 210 may include an input unit 214 and an output unit 215. The input unit 214 and output unit 215 may be included in a transceiver (not shown), or the input 214 may be included in a receiver and the output 215 may be included in a transmitter, arranged to receive (input unit 214) and transmit (output unit 215) radio frequency signals or any wireless signals for communication with the first controller 110 and / or the second controller 120 according to any suitable wireless communication protocol (such as Bluetooth, Zigbee, Wi-Fi, Thread, etc.). The input unit 214 and output unit 215 may communicate wiredly according to any suitable wired communication protocol (such as Power over Ethernet, Power Line Communication, etc.).

[0060] The controller 210 may further include a memory 212, which may be arranged to store communication IDs of the first controller 110, the second controller 120, the lighting devices 110a-d, and / or any actuation / sensing devices. The controller 210 may include a processor 213, which is arranged to perform the steps of the method according to the first aspect or at least control the execution of the steps of the method according to the first aspect.

[0061] Controller 210 may be implemented in a separate unit (such as a wall panel, desktop computer terminal, or even a portable terminal (such as a laptop, tablet, or smartphone)) separate from the first controller 110, the second controller 120, the lighting devices 110a-d, and the user equipment 131. Alternatively, controller 210 may be incorporated into the same unit as the first controller 110, the second controller 120, and the user equipment 131, and / or in the same unit as one of the lighting devices 110a-d. Furthermore, controller 210 may be implemented in or away from environment 101 (e.g., on a server); and controller 210 may be implemented in a single unit or in a distributed functional form distributed across multiple separate units (e.g., a distributed server comprising multiple server units at one or more geographical locations, or distributed control functionality distributed across the first controller 110, the second controller 120, the user equipment 131, and the lighting devices 110a-d). Furthermore, the controller 210 may be implemented as software stored in a memory (which includes one or more memory devices) and arranged for execution on a processor (which includes one or more processing units), or the controller 210 may be implemented as dedicated hardware circuitry, or configurable or reconfigurable circuitry such as a PGA or FPGA, or any combination thereof.

[0062] In order for the controller 210 to receive or transmit communication signals, communication may be implemented by any suitable wired or wireless means, such as a local (short-range) RF network, such as Wi-Fi, ZigBee, Bluetooth or Thread network, Power over Ethernet, Power line communication, or any combination of these and / or other means.

[0063] Figure 3 A flowchart illustrating an embodiment of a method 300 for sensing in environment 101 via first sensing systems 110a-d and second sensing system 120 is shown schematically and exemplary. First controller 110 and second controller 120 may be respectively arranged to control first sensing systems 110a-d and second sensing system 120. First controller 110 and second controller 120 may be integrated into sensing systems / devices 110a-d, 120, or alternatively may be external to sensing systems / devices 110a-d, 120.

[0064] The first sensing system 110a-d has a different type from the second sensing system 120. This different type can include the same sensing task, such as presence sensing performed via different sensing mechanisms, like camera-based sensing and radio frequency-based sensing. This different type can also include different sensing tasks.

[0065] The first sensing system 110a-d may have a sensing range that partially overlaps with the second sensing system 120. Thus, for example, in at least one sub-region of environment 101, the first sensing system 110a-d and the second sensing system 120 (e.g., via different sensing mechanisms) detect the same sensing event.

[0066] The first controller 110 may also be arranged to control the first actuation systems 110a-d, which may influence sensing from the second sensing system 120 at least in a sub-region of the environment 101. The first actuation system 110a-d and the first sensing system 110a-d may be co-located. In another example, the first actuation system 110a-d and the first sensing system 110a-d may be performed by the same system or the same device. At least one sub-region may (at least partially) overlap with at least a portion of the sensing range of the two sensing systems 110a-d, 120. In one example, the second controller 120 may also be arranged to control a second actuation system (not shown).

[0067] In one example, a first controller 110a-d may be arranged to control a lighting system 110a-d, which includes at least one lighting device 110a-d for providing illumination in an environment 101 and for radio frequency-based sensing, and wherein a second controller 120 is arranged to control a vision-based sensing system, wherein the radio frequency-based sensing has a sensing range that at least partially overlaps with the vision-based sensing 120, and wherein the illumination may affect the vision-based sensing system 120 in at least a sub-region of the environment 101.

[0068] Method 300 may include receiving, at a first controller 110, a request 310 of an interaction from a second controller 120, wherein the request includes adjusting the actuation of the first actuation systems 110a-d to minimize the impact of the first actuation systems 110a-d on the sensing of the second sensing system 120. The first actuation systems 110a-d may positively or negatively affect the sensing from the second sensing system 120. In another example, the first actuation systems 110a-d may also affect the sensing of the first sensing system 110a-d. The second controller 120 may determine the impact of the first actuation systems 110a-d on the sensing of the second sensing system 120. This determination may be precise (e.g., providing the probability of false alarms / accuracy loss), or it may be a coarse estimate of the effect, such as whether the first actuation systems 110a-d affect the second sensing system 120. Additionally or alternatively, the first controller 110 may determine the impact of the first actuation systems 110a-d on the sensing of the second sensing system 120. Based on how the impact is determined, for example, the first sensing system 110a-d and / or the second sensing system 120 may have a threshold, beyond which the second controller 120 is arranged to request the first controller 110 to adjust the actuation of the first actuation system 110a-d to minimize the impact of the first actuation system 110a-d on the sensing of the second sensing system 120.

[0069] The second controller 120 can communicate with the first controller 110 using any wireless and / or wired communication means utilizing suitable wireless / wired communication protocols. Both the first controller 110 and the second controller 120 include appropriate circuitry (not shown) to perform the wireless / wired communication. In one example, the second controller 120 directly transmits a request to the first controller 110. Alternatively or additionally, the second controller 120 may transmit a request to the first controller 110 via user equipment 131 or any other device / means.

[0070] Method 300 may further include assigning a priority value 320 to the received request, indicating a priority level of the received request. Method 300 may further include assigning the priority value 320 based on one or more of the request for interaction, the requested interaction feature with the second controller 120, and features of the second controller 120.

[0071] In one example, the characteristics of the second controller 120 may include a confidence property, and the method 300 may further include assigning a confidence value that indicates the confidence level of the second controller 120, and assigning a priority value based on the confidence value.

[0072] Additionally or alternatively, method 300 may further include assigning a priority value 320 based on the sensing task and / or sensing results of the second sensing system 120. Also additionally or alternatively, method 300 may further include assigning a priority value 320 based on contextual information related to the environment 101.

[0073] Method 300 may further include determining whether to accept the adjustment actuation request based on the assigned priority value. Method 300 thus allows the first controller 110 to make a decision to accept or reject the request to adjust the first actuation system 110a-d. The first controller 110 and the second controller 120 may be arranged in a master-slave configuration such that the second controller 120 can be the master controller, and therefore, requests from the second controller 120 are assigned priority. Or in other words, the first controller 110, as the slave controller 110, (always) accepts requests from the second controller 120.

[0074] As an alternative to a master-slave configuration, the first controller 110 and the second controller 120 may have equal permissions, or they may not have defined permissions. For example, the first controller 110 and the second controller 120 may come from different manufacturers or suppliers, and they may not know any priority ordering of each other in the environment, and may also not know each other's work (e.g., machine learning algorithms).

[0075] If determination 330 leads to acceptance of the request, method 300 may further include adjusting the actuation of the first actuation systems 110a-d based on the request. This adjustment may be determined 330 by the first controller 110 and / or the second controller 120. The adjustment is determined to minimize / eliminate the (negative) impact of the first actuation systems 110a-d on the sensing of the second sensing system 120.

[0076] If it is determined that 330 results in the rejection of the request, then method 300 may further include requesting the second controller 120 to switch the sensing task of the second sensing system 120 at least partially to the first sensing system 110a-d in the at least partially overlapping sensing range.

[0077] Additionally or alternatively, if it is determined that 330 results in the rejection of the request, method 300 may further include requesting the second controller 120 to perform joint sensing in at least partially overlapping sensing ranges.

[0078] In one example, before the first controller 110 proposes to take over sensing from the second controller 120, the first controller 110 may check the user 130's past use of the first sensing system 110a-d and the second sensing system 120 mobile applications on the user's mobile device 131, and / or whether the first sensing system 110a-d and / or the second sensing system 120 mobile applications are currently open on the user's mobile device 131. If the user uses the first sensing system 110a-d mobile applications more frequently, or if the first sensing system 110a-d mobile applications are currently open while the second sensing system 120 mobile applications are closed, the first controller 110 may share these insights with the second controller 120, along with the reasons why the first controller 110 should take over sensing, since the first sensing system 110a-d mobile applications (currently) are more suitable for providing feedback to the user 130 regarding the sensing insights.

[0079] In one example, the first controller 110 may be arranged to operate in an interpretable mode and a non-interpretable mode, and the first controller 110 in interpretable mode may be arranged to transmit interpretability information related to the control decisions of the first controller 110 to the second controller 120. In one example, the second controller 120 may also be arranged to transmit interpretability information to the first controller 110, for example, in an interpretable mode. The first controller 110 and the second controller 120 may include or be arranged to use a machine learning model to determine at least some control decisions for controlling the corresponding actuation and / or sensing systems. The machine learning model may include a mathematical function or representation of the relationship between (one or more) inputs and (one or more) outputs. The model is the result of a machine learning algorithm applied to a training dataset. The model is typically a parameterized mathematical formula, where the parameters are learned through a machine learning algorithm. Given input data, the model may directly generate classification labels or regression values, or it may generate the probability of each possible value (input).

[0080] Explainability information may include information related to the determination of control decisions by the machine learning model, such as how the machine learning model works and why it determines a particular control decision. For example, how the sensing system draws conclusions about sensed events based on raw data. Explainability information may include transparency and / or interpretability information related to the determination of control decisions and the machine learning model. Explainability information provides an understanding of the machine learning model and the control decisions determined by the model. For example, this information may include the degree to which cause and effect can be observed within the actuation and / or sensing system. Or, in other words, it is the degree to which what can be predicted given changes in the input or machine learning algorithm parameters. Additionally or alternatively, this information may include the degree to which the internal mechanisms of the machine learning model can be explained in human-like terms. Transmitting explainability information to the second controller 120 can help the second controller 120 understand why the machine learning model of the first controller 110 makes a particular control decision. In one example, the explainability information does not necessarily need to be in human-like terms because the information is shared with the second controller 120 and not directly with the user 130. The information may be shared with the user 130 via the second controller 120. Alternatively, this information may be shared with both the second controller 120 and the user 130, or only with the user 130.

[0081] In one example, as part of a negotiation process, the first controller 110 may propose to the second controller 120 that they evaluate their respective sensing performance before making a final decision on how to reallocate sensing tasks. This evaluation can be performed through parallel co-sensing (e.g., simultaneously by WiFi sensing from the first sensing systems 110a-d and optimal computer vision-based sensing from the second sensing system 120). Alternatively, sensing can be performed in an A / B mode (for Mondays and Wednesdays, the second sensing system 120 performs all lighting control based on its computer vision sensing data, and for Tuesdays and Thursdays, the first sensing systems 110a-d perform all lighting control). Over the weekend, the first and second controllers 110 and 120 jointly evaluate which of the two sensing scenarios is more satisfying for the user 130. Based on these findings, the two systems reconfigure their co-sensing in environment 101.

[0082] Before proposing co-sensing in a room, the machine learning model of the first controller 110 can identify which type / number of sensing nodes the machine learning model of the second controller 120 can use for context-aware sensing of, for example, environment 101. The machine learning model of the first controller 110 can then estimate whether the first controller 110 or the second controller 120 is actually in a position to make a better decision. For example, for the first room in environment 101, the second controller 120 has access to two camera devices, while the first sensing systems 110a-d can achieve optimal performance with a smaller number of sensing nodes in the same room. Therefore, the first controller 110 will comply with the original request from the second controller 120 to adjust lighting actuation and will not propose co-sensing. However, for the second room in environment 101 with many pieces of furniture, the second controller 120 has access to a single camera, while the first sensing systems 110a-d have WiFi lights in the same room, ensuring optimal WiFi sensing performance. Therefore, the first controller 110 will propose co-sensing and take over the sensing task from the second sensing system 120. In this example, as described above, priority values ​​can be assigned based on contextual information of environment 101 and / or the available sensing resources of the sensing system.

[0083] Additionally or alternatively, the first actuation system 110a-d can present a light scene including multiple lights, and the second controller 120 can request adjustment of the light settings of a subset of the lighting devices 110a-d. However, the first controller 110 can decide not to comply with the original request from the second controller 120; a better alternative is to intentionally select the actual lighting settings of the room, such that... • Computer vision AI running on the second controller 120 can monitor the first part of the room. Meanwhile, the lights in the second part of the room are still displaying their original depth-dimming lighting settings, which hinders the computer vision of the second controller 120. To compensate for the impaired computer vision of the second sensing system 120, the lighting system is configured with RF sensing to optimally monitor the second part of the room.

[0084] The lighting system 110a-d in the third part of the room can be configured such that the third part can be monitored by a combination of computer vision from a suboptimal second sensing system and RF sensing from a suboptimal first sensing system. However, sufficient overall sensing performance can be achieved by fusing the two sensor modes in the third part of the room.

[0085] Therefore, in this example, the first sensing system 110a-d and the second sensing system 120 are orchestrated together to coordinate the interaction between light settings, the computer vision performance of the second controller 120 in different areas of the room, the RF sensing performance of the lighting devices 110a-d in different areas of the room, and the human-centered lighting experience of the user 130.

[0086] In one example, the second actuation system of the second controller 120 can influence the first sensing and / or the first actuation systems 110a-d. For example, actuating the second sensing system 120 (e.g., a home monitoring camera) could have its own floodlight source (the second actuation system), which is controlled independently of the sensing system 120. However, when the user 130 is approaching home, the floodlight of the second controller 120 might negatively impact the first actuation (lighting) systems 110a-d's 'welcome home lighting scene' in the garden, for example. In this case, the first controller 110 contacts the second controller 120 to coordinate how the second sensing system actuates its light to the camera to suit the overall garden lighting experience for the user 130. For example, the first controller 110 could request from the second controller 120 to use only IR light instead of visible light for its monitoring task. However, IR light has a limited sensing range compared to the visible floodlight of the second controller 120. However, the first controller 110 proposes to perform RF sensing with its garden lights in the darker parts of the sidewalk, so that the second controller 120 receives advance notification of the visitor's arrival.

[0087] Continuing the example, when the first controller 110 (e.g., based on geofencing data or everyday life insights) infers that the homeowner is approaching the house, the first controller 110 may request the second controller 120 to use only IR light so as not to interfere with the garden light scene presented by the lighting devices of the first actuation systems 110a-d. On the other hand, if the second controller 120 has reason to believe that the person approaching the house is not the homeowner but is likely a stranger, regardless of the request of the first controller 110, the second controller 120 will not comply with the request of the first controller 110 because the second controller 120 wants to use visible floodlight to indicate to the approaching person that he is "within the camera's field of view".

[0088] In various examples, the second sensing system 120 may include AR / VR headsets that utilize computer vision to understand the context of the environment 101 in which they are operating. Let's assume a Metaverse rock band rehearsal is displayed through its headset. A Metaverse rock concert consists of a first guitarist, a second drummer (who shares a room with the first performer in the real world), and a third bassist (virtually represented as an avatar). The first performer's AR / VR headset uses computer vision to monitor the second performer's body movements within the room.

[0089] However, the first performer's AR / VR headset may not be able to perform its computer vision sensing tasks as well as the first actuation system 110a-d, for example, the entertainment lights create low-light or overly glaring lighting conditions for the AR / VR headset's image sensors.

[0090] In such an example, the first controller 110 can negotiate with the second controller 120 (e.g., an AR / VR headset) regarding the light settings of the first actuation system (lighting systems 110a-d) to improve AR / VR computer vision and thus better understand the context of the room. Specifically, the second controller 120 and the first controller 110, for example, agree on which areas of the room and / or which human activities the AR / VR headset's computer vision is responsible for, while the WiFi / radar sensing of the first sensing system 110a-d is responsible for context-aware sensing in which room area. For example, the first sensing system 110a-d could be responsible for detecting the second drummer's emotions (via monitoring the second drummer's vital signs), while the computer vision is only responsible for monitoring his large body movements. Based on this consensus, the first controller 110 then adjusts the lights in the area to be monitored by the AR / VR headset accordingly.

[0091] When the computer program product is run on the processing unit of the computing device (such as the processor 213 of the controller 210), the method 300 can be executed by the computer program code of the computer program product.

[0092] It should be noted that the above embodiments are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims.

[0093] In the claims, any reference numerals placed between parentheses should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or steps other than those stated in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer or processing unit. In an apparatus claim enumerating several means, several of these means may be embodied by the same hardware item. The mere fact that certain measures are referenced in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

[0094] Various aspects of this invention can be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer-readable storage device and executable by a computer. The instructions of this invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes. The instructions may be provided as a complete executable program, a partial executable program, as a modification (e.g., an update) of an existing program, or as an extension (e.g., a plugin) of an existing program. Furthermore, some processing of this invention may be distributed across multiple computers or processors or even the “cloud.”

[0095] Storage media suitable for storing computer program instructions include all forms of non-volatile memory, including but not limited to EPROM, EEPROM, and flash memory devices, disks such as internal and external hard drives, removable disks, and CD-ROMs. Computer program products may be distributed on such storage media or made available for download via HTTP, FTP, email, or through a server connected to a network such as the Internet.

Claims

1. A method for sensing in an environment via a first sensing system and a second sensing system, wherein a first controller and a second controller are arranged to control the first sensing system and the second sensing system respectively, and wherein the first sensing system has a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system. The first controller is further configured to control a first actuation system, which is capable of influencing sensing from the second sensing system, at least in a sub-region of the environment. The method includes: The first controller receives a request for interaction with the second sensing system, wherein the request includes adjusting the actuation of the first actuation system to minimize the impact of the first actuation system on the sensing of the second sensing system. Assign a priority value to the received request, indicating the priority level of the received request. The decision to accept or reject the adjustment actuation request is based on the assigned priority value.

2. The method of claim 1, wherein if the determination results in accepting the request, the method further includes adjusting the actuation of the first actuation system based on the request.

3. The method according to any one of the preceding claims, wherein if the determination results in rejection of the request, the method further comprises requesting the second controller to at least partially switch the sensing task of the second sensing system to the first sensing system within the at least partially overlapping sensing range.

4. The method according to any one of the preceding claims, wherein if the determination results in rejection of the request, the method further comprises requesting the second controller to perform joint sensing within the at least partially overlapping sensing range.

5. The method according to any one of the preceding claims, wherein the first controller is arranged to control a lighting system, the lighting system comprising at least one lighting device for providing illumination in an environment, and one or more of radar, time-of-flight (ToF) sensing, and radio frequency (RF) based sensing, and wherein the second controller is arranged to control a vision-based sensing system, wherein one or more of the radar, ToF, and RF based sensing have a sensing range that at least partially overlaps with the vision-based sensing, and wherein the illumination can affect the vision-based sensing system at least in a sub-region of the environment.

6. The method according to any one of the preceding claims, wherein the method further comprises assigning a priority value based on one or more of the following: the request for interaction, the characteristics of the requested interaction with the second controller, the characteristics of the second controller, and the number and / or type of sensing devices of the second sensing system.

7. The method according to any one of the preceding claims, wherein the method further comprises assigning a confidence value indicating the confidence level of the second controller, and wherein a priority value is assigned based on the confidence value.

8. The method according to any one of the preceding claims, wherein the method further comprises assigning a priority value based on the sensing task and / or sensing result of the second sensing system.

9. The method according to any one of the preceding claims, wherein the method further comprises assigning a priority value based on contextual information related to the environment.

10. The method according to any one of the preceding claims, wherein the method further comprises determining whether to accept the request for adjustment actuation based on the user's past interactions or preferences with the first controller and the second controller.

11. The method according to any one of the preceding claims, wherein determining whether to accept the request for adjustment actuation is further based on the sensing performance of the first sensing system and the second sensing system at least in a sub-region of the environment.

12. The method of claim 11, wherein sensing performance is evaluated based on the number of sensing devices of the first sensing system and / or the second sensing system, the historical performance of the first sensing system and the second sensing system, the layout of sub-regions of the environment, physical objects in the sub-regions of the environment, and sensing interference in the sub-regions of the environment.

13. A controller arranged for controlling a first sensing system in an environment, wherein the first sensing system has a sensing range that at least partially overlaps with that of a second sensing system. The controller is also configured to control a first actuation system, which influences sensing from a second sensing system at least in a sub-region of the environment. The controller includes a processor, which is configured to perform the following steps: Receive a request for interaction with the second sensing system, wherein the request includes adjusting the actuation of the first actuation system to minimize the impact of the first actuation system on the sensing of the second sensing system. Assign a priority value to the received request, indicating the priority level of the received request. The decision to accept or reject a request for adjustment actuation is based on the assigned priority value.

14. A system for sensing in an environment via a first sensing system and a second sensing system, wherein a first controller and a second controller are arranged to control the first sensing system and the second sensing system, respectively, and wherein the first sensing system has a different type from the second sensing system and has a sensing range that at least partially overlaps with the second sensing system. The first controller is further configured to control a first actuation system, which influences sensing from the second sensing system at least in a sub-region of the environment. The system includes: Second controller, The first controller according to claim 13.

15. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1-12.

Images