Adaptation of physical environment to XR environment

Abstract

There is provided techniques for controlling environment effectors in a physical environment. The environment effectors are controlled by a smart home controller. The method is performed by a controller. The method comprises identifying a first physical environment state for a first user in the physical environment based on the first user's interaction with an HMD device in the physical environment. The method comprises identifying a second physical environment state for a second user in the physical environment based on the second user's activity in the physical environment. The method comprises determining a control signal for the smart home controller to control the environment effectors to produce a physical environment state in the physical environment. The control signal is based on a combination of the first physical environment state and the second physical environment state.

Description

[0001] ADAPTATION OF PHYSICAL ENVIRONMENT TO XR ENVIRONMENT

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, a controller, a system, a computer program, and a computer program product for controlling environment effectors in a physical environment.

[0004] BACKGROUND

[0005] The use of devices for extended reality (XR) is increasing. In this respect, Head- Mounted Display (HMD) devices designed for XR environments typically isolate its user from seeing the real-world (i.e., the physical) environment. This is to enable the user to have larger XR fields-of-view, better picture quality, and no visual disturbances while experiencing the XR environment. This is at the cost of the XR environment being decoupled, by means of vision and sound, from the physical environment. That is, an ideal, fully immersive XR environment does not compromise itself to fit to any physical environment.

[0006] Eyes take time to adapt to dark or bright light, especially when the transition between different light conditions is fast. A user in a dark room wearing an HMD device with a bright display would be effectively blind and need to wait up to 5 minutes to see details in the dark room when removing the HMD device. Correspondingly, a user in a very bright room wearing an HMD device displaying a dark XR environment would also be effectively blind when removing the HMD device. The same applies to audio input; hearing also takes time to accommodate to significantly different loudness levels which could be the case when a user removes the HMD device and where the sound level in the XR environment is very different from the sound level in the physical environment.

[0007] There could be situations where a user wearing an HMD device rapidly needs to remove the HMD device, for example due to an interruption in the physical environment (e.g., interaction from another person, unexpected noise, phone call, walking into an unexpected physical obstacle, etc.). This does not allow time for the HMD device to slowly prepare the user for whatever real-world conditions the user is in. As a result, if the XR environment and the physical environment are mismatched with respect to each other (e.g., in terms of brightness levels, sound levels, etc.), the user may not be able to react appropriately to the interruption. Moreover, rapidly changing brightness levels, sound levels, etc. are unpleasant and uncomfortable for the user.

[0008] Different ways to mitigate the above issues will be presented next.

[0009] One way to mitigate the above issues could be to change the XR environment in the HMD device (e.g., by adapting the brightness level and / or sound level to fit the physical environment (e.g., as suggested in US9996977B2). However, this, compromises the XR experience since the image and / or sound of the XR environment is degraded and thus degrades the overall user experience when wearing the HMD device.

[0010] One way to mitigate the above issues could be to explicitly, and intentionally, change a smart home illumination and sound ambience through an explicit control interface shown in the HMD device (e.g., as suggested in US20200393952A1). That is, a user interface of smart home controller is presented at the display of the HMD device, and the user is enabled to via the user interface, manually adjust the smart home illumination and sound ambience. However, this requires significant extra effort on part of the user to have the foresight and take the time to perform such manual adjustment. At least such manual adjustment of the smart home illumination and sound ambience to the XR environment presented in the HMD device needs to be made such that the physical environment has been adapted to the XR environment when the user needs to remove the HMD device.

[0011] However, manually adjusting the smart home illumination and sound ambience to the XR environment might cause disturbances to users in the physical environment that are not wearing any HMD device, or even users wearing an HMD device but that are experiencing some other XR environment with other brightness levels and / or sound levels.

[0012] In view of the above, one option is to allow one single user to control the smart home illumination and sound ambience (for example by implementing the technology suggested in US9996977B2 in one single HMD) or letting the users not wearing any HMD control of the smart home illumination and sound ambience. Another option is to allow each user to contend for control of the smart home illumination and sound ambience, for example by implementing the technology suggested in US9996977B2 per individual user that is wearing an HMD device. However, none of these options resolve the issue at hand.

[0013] Hence, there is still a need for an improved adaptation of physical environment to one or more XR environments.

[0014] SUMMARY

[0015] An object of embodiments herein is to address the above issues.

[0016] A particular object of embodiments herein is to provide efficient adaptation of a physical environment to one or more XR environments.

[0017] A particular object is to enable adaptation of a physical environment to one or more XR environments that provides a compromise among the users.

[0018] According to a first aspect there is presented a method for controlling environment effectors in a physical environment. The environment effectors are controlled by a smart home controller. The method is performed by a controller. The method comprises identifying a first physical environment state for a first user in the physical environment based on the first user’s interaction with an HMD device in the physical environment. The method comprises identifying a second physical environment state for a second user in the physical environment based on the second user’s activity in the physical environment. The method comprises determining a control signal for the smart home controller to control the environment effectors to produce a physical environment state in the physical environment. The control signal is based on a combination of the first physical environment state and the second physical environment state.

[0019] According to a second aspect there is presented a controller for controlling environment effectors in a physical environment. The environment effectors are controlled by a smart home controller. The controller comprises processing circuitry. The processing circuitry is configured to cause the controller to identify a first physical environment state for a first user in the physical environment based on the first user’s interaction with an HMD device in the physical environment. The processing circuitry is configured to cause the controller to identify a second physical environment state for a second user in the physical environment based on the second user’s activity in the physical environment. The processing circuitry is configured to cause the controller to determine a control signal for the smart home controller to control the environment effectors to produce a physical environment state in the physical environment. The control signal is based on a combination of the first physical environment state and the second physical environment state.

[0020] According to a third aspect there is presented a computer program for controlling environment effectors in a physical environment. The computer program comprises computer code which, when run on processing circuitry of a controller, causes the controller to perform actions. One action comprises the controller to identify a first physical environment state for a first user in the physical environment based on the first user’s interaction with an HMD device in the physical environment. One action comprises the controller to identify a second physical environment state for a second user in the physical environment based on the second user’s activity in the physical environment. One action comprises the controller to determine a control signal for the smart home controller to control the environment effectors to produce a physical environment state in the physical environment. The control signal is based on a combination of the first physical environment state and the second physical environment state.

[0021] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0022] According to a fifth aspect there is presented a system for controlling environment effectors in a physical environment. The system comprises a controller configured to perform the method according to the first aspect and thereby to determine the control signal. The system comprises a smart home controller configured to control the physical environment by providing an actuation signal based on the control signal to the environment effectors in the physical environment. Advantageously, these aspects provide efficient adaptation of a physical environment to one or more XR environments.

[0023] Advantageously, these aspects enable adaptation of the physical environment to one or more XR environments such that that it provides a compromise among the users, and thus without severely degrading the experience of neither users interacting with HMD devices nor users not interacting with any HMD device.

[0024] Advantageously, by adapting the physical environment to the one or more XR environments, these aspects mitigate the above-noted issues where a user wearing an HMD device rapidly needs to remove the HMD device, thus reducing the risk that the user is temporary blinded (or deafened) when removing the HMD device.

[0025] Advantageously, by adapting the physical environment to the one or more XR environments, these aspects reduce the risks of excessively rapid changes (such as light flashes etc.) occurring in the physical environment as the user removes the HMD device.

[0026] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0027] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0030] Fig. 1 is a schematic diagram illustrating a system according to embodiments;

[0031] Fig. 2 is a flowchart of methods according to embodiments; Fig. 3 schematically illustrates different states according to embodiments;

[0032] Fig. 4 is a schematic diagram showing structural units of a controller according to an embodiment;

[0033] Fig. 5 is a schematic diagram showing functional modules of a controller according to an embodiment; and

[0034] Fig. 6 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.

[0035] DETAILED DESCRIPTION

[0036] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0037] Fig. i is a schematic diagram illustrating a system 100 where embodiments presented herein can be applied. The system 100 comprises a controller no. As will be disclosed, the controller no is configured to control environment effectors 152A- I:152B-M in a physical environment 160. In general terms, the controller 110 is configured to determine a control signal 180 for a smart home controller 140, and the smart home controller 140 controls the environment effectors 152A-1:152B-M by providing an actuation signal igoArigoB that is based on the control signal 180 to the environment effectors 152A-1:152B-M. In this respect, the controller no might, or might not, have access information of the environment effectors 152A-1:152B-M, such as the type of environment effectors 152A-1:152B-M, the placement of the environment effectors 152A-1:152B-M in the physical environment 160, etc. Especially the latter case, the control signal 180 provided by the controller no to the smart home controller 140 might not be identical to the actuation signals igoArigoB that are provided by the smart home controller 140 to the environment effectors 152A-I:152B-M. For example, the smart home controller 140 might for this purpose be configured to implement a mapping between the control signal 180 provided by the controller no to the smart home controller 140 and the actuation signals igoAngoB that are provided by the smart home controller 140 to the environment effectors 152A-I:152B-M. This mapping is at least based on the afore-mentioned information of the environment effectors 152A-1:152B-M. For this and other purposes, the controller 110 and the smart home controller 140 are configured to communicate with each other over a communication interface which can be wireless or wired and utilize any communication protocol suitable for the communication between the controller no and the smart home controller 140. In some examples, the controller no is part of the smart home controller 140.

[0038] The physical environment 160 can be divided in different zones 150A, 150B, in Fig. 1 represented by Zone A and Zone B. Each such zone might correspond to an individual room in a building, to different parts of one and the same room, or to different floors in one and the same building, etc. The different zones 150A, 150B can have different ambient illumination levels, ambient hue levels, directed illumination levels, directed hue levels, ambient sound volume, directed sound levels, etc. as set by the environment effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B-M. For this reason, there can thus be different types of environment effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B-M, such as light sources, light panels, smart window blinds, media displays, media players, loudspeakers, etc. Each combination of ambient illumination level, ambient hue level, directed illumination level, directed hue level, ambient sound volume, directed sound level, etc. can be referred to as a physical environment state. Hence, depending on the settings of the effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B- M, there might be different physical environment states in the different zones 15 oA, 150B. In some aspects, each physical environment state corresponds to a scene, where each scene represents a predefined configuration that can be activated for a group of the environment effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B-M. In this way, a group of the environment effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B-M can be controlled as one unit. Thus, different groups of the environment effectors 152A-1, ..., 152A-N, 152B-1, ..., 152B-M can be controlled by means of using different physical environment states in the different zones 150A, 150B. It is assumed that at least two users are located in the physical environment 160 and that at least one of the users is interacting with an HMD device 120. Different ways in which the users might interact with HMD device 120 will be disclosed below. As will be further disclosed below, the physical environment state of the physical environment 160 will be impacted by how the users interact with the HMD device 120. For this and other purposes, the controller no and the HMD device 120 are configured to communicate with each other over a communication interface 170a which can be wireless or wired and utilize any communication protocol suitable for the communication between the controller no and the HMD device 120. In some examples, the controller 110, 400 is part of the HMD device 120.

[0039] As schematically illustrated, one or more sensors 130 might be present in the physical environment 160. These one or more sensors 130 might be configured to monitor the physical environment states of the physical environment 160 and provide feedback to the smart home controller 140 or the controller 110. For this and other purposes, the smart home controller 140 (or the controller no) and the HMD device 120 are configured to communicate with each other over a communication interface 170b which can be wireless or wired and utilize any communication protocol suitable for the communication between the smart home controller 140 (or the controller no) and the HMD device 120. Further in this respect, the one or more sensors 130 might be configured to monitor where in the physical environment 160 the users are located, gaze directions, etc. Depending on the one or more purpose of the sensors, 130, each sensor 130 might be equipped with different types of sensing means, such as cameras, light meters, microphones, and the like.

[0040] According to the schematic diagram of Fig. 1, the controller 110 comprises a modulator block 112, an adaptor block 114 and a compromiser block 116.

[0041] Aspects of the modulator block 112 will be disclosed next.

[0042] The modulator block 112 is configured to identify a respective preferred physical environment state (e.g., in terms of preferred ambient illumination level, preferred ambient hue level, preferred directed illumination level, preferred directed hue level, preferred ambient sound volume, preferred directed sound level, etc.) for each user in the physical environment 160. For this purpose, the modulator block 112 might receive input in terms of the position and gaze direction of the users within the physical environment 160, their use of media devices (such as television, mobile phones, computers, etc.), and their interaction with HMD devices 120. If an HMD device 120 is in use by a user or otherwise associated with the user, then inputs also may include properties of the image (brightness, hue, etc.) and sound as played out in the HMD device 120, the HMD device’s isolation factor, the currently running application category and state, etc. As non-limiting example, for a user that is wearing an HMD device 120, the preferred physical environment state can be modulated to continuously match the XR environment as rendered by the HMD device. As non-limiting example, for a user that is not wearing any HMD device 120, the preferred physical environment state can be modulated to continuously match the activity of the user, for example if the user is reading, working, watching television, listening to music, etc. Hence, the output from the modulator block 112 is a preferred physical environment state for each user in the physical environment 160.

[0043] Aspects of the adaptor block 114 will be disclosed next.

[0044] The adaptor block 114 is configured to identify in what way the user is interacting with the HMD device 120, for example if the user is starting to use, and thus starting to wear, the HMD device 120, if the user is wearing, the HMD device 120, or if the user is ending his / her use of the HMD device 120, and is thus removing the HMD device 120. For this purpose, the adaptor block 114 might receive input from the HMD device 120 and / or the sensors 130. For example, removal of the HMD device 120 can be detected e.g., via sensors (movement, internal cameras, etc.) in the HMD device itself or by external sensors 130 (e.g., room cameras). Picking-up the HMD device 120 can be detected e.g., via movement sensors in the HMD device itself, from hand controllers, touch-sensitive surfaces on the HMD, etc. or by external sensors 130 (e.g., room cameras). Further aspects of this will be disclosed below with reference to Fig. 3. In general terms, different physical environment states might be preferred depending on the interaction. The adaptor block 114 might therefore at least gradually impact which physical environment state that is preferred for a given user, depending on how this given user interacts with an HMD device 120. Hence, the output from the adaptor block 114 is an adaptation of the preferred physical environment state for each user interacting with an HMD device 120 in the physical environment 160. Aspects of the compromiser block 116 will be disclosed next.

[0045] In general terms, the compromiser block 116 is configured to find a combination of the preferred physical environment states that on the one hand maximizes the preferred state of each user but that on the other hand also minimizes interference between multiple preferred states of multiple users sharing the same physical environment 160. The compromiser block 116 is therefore configured to receive the preferred physical environment states for all the users and determine a resulting physical environment state based on which the environment effectors 152A-1, 152A-N, 152B-1, 152B-M are set. Hence, the output from the compromiser block 116 is a control signal 180 that is to be converted to actuation signals 190a: 190B that are provided as input to the environment effectors 152A-1:152B-M in the physical environment 160. As will be further disclosed below, the preferred physical environment states might be combined in a weighted fashion, where location of effect, user priority and cross-user effect might be part of the weighting. In this way, a number of users can be located within the physical environment 160, potentially spread out between different zones, performing various activities, some of which include using an HMD device 120, where the environment effectors 152A-1:152B-M produce a physical environment state 3403:340? in the physical environment 160 that is a compromise of the physical environment states as preferred by the individual users.

[0046] The embodiments disclosed herein in particular relate to techniques for controlling environment effectors 152A-1:152B-M in a physical environment 160. In order to obtain such techniques, there is provided a controller no, 400, 500, a method performed by the controller 110, 400, 500, a computer program product comprising code, for example in the form of a computer program, that when run on a controller no, 400, 500 causes the controller 110, 400, 500 to perform the method.

[0047] Fig. 2 is a flowchart illustrating embodiments of methods for controlling environment effectors 152A-1:152B-M in a physical environment 160. The environment effectors 152A-1:152B-M are controlled by a smart home controller 140. The methods are performed by the controller 110, 400. The methods are advantageously provided as computer programs 620. First, the controller no, 400 identifies which, possibly different, physical environment states 3403:340? that would best fit the activities of each individual user in the physical environment 160. In this respect, each of the physical environment states 340a: 340P might be associated with its own at least one setting of the environment effectors 152A-1:152B-M. Further aspects relating thereto will be disclosed below.

[0048] S102: The controller 110, 400 identifies a first physical environment state 340a:340P for a first user in the physical environment 160. The first physical environment state 340a:340P is identified based on the first user’s interaction with an HMD device 120 in the physical environment 160. Step S102 can be implemented by the modulator block 112 and the adaptor block 114.

[0049] S104: The controller no, 400 identifies a second physical environment state 340a:340P for a second user in the physical environment 160. The second physical environment state 34oa:34oP is identified based on the second user’s activity in the physical environment 160. Step S104 can be implemented by the modulator block 112, possibly also in combination with the adaptor block 114, depending on whether the second user is interacting with an HMD device 120 or not.

[0050] In this respect, the second physical environment state 340a:340P is either identical to the first physical environment state 340a:340P or different from the first physical environment state 340a:340P. Regardless, the physical environment 160 is controlled to produce a physical environment state 340a:340P in the physical environment 160 that, as good as possible, fits both the first user and the second user, as in step S106.

[0051] S106: The controller no, 400 determines a control signal 180 for the smart home controller 140 to control the environment effectors 152A-1:152B-M to produce a physical environment state 3403:340? in the physical environment 160. The control signal 180 is based on a combination of the first physical environment state 3403:340? and the second physical environment state 340a:340P. Step S106 can be implemented by the compromiser block 116. The smart home controller 140 might then control the physical environment 160 by providing an actuation signal igoArigoB based on the control signal 180 to the environment effectors 152A-1:152B-M in the physical environment 160.

[0052] In this way, the lighting, soundscape, etc. of the physical environment 160 can be continuously modulated as a compromise between a user of an HMD device 120 and other users (either using HMD device 120 or not using any HMD device 120, or any combination of users using and not using HMD devices 120) sharing the same physical environment 160. In turn, this method thereby enables the above-disclosed issues with respect to discomfort in situations where a user wearing an HMD device 120 rapidly needs to remove the HMD device 120 to be mitigated.

[0053] Embodiments relating to further details of controlling environment effectors 152A- I:152B-M in the physical environment 160 as performed by the controller no, 400 will be disclosed next.

[0054] Further aspects of how the controller no, 400 can identify the first physical environment state 3403:340? and the second physical environment state 3403:340? will be disclosed next.

[0055] As disclosed above, the first physical environment state 340a: 340P is identified based on the first user’s interaction with an HMD device 120 in the physical environment 160. In this respect, there could be different ways in which the first user interacts with the HMD device 120. Reference is here made to Fig. 3(a) which illustrates a state diagram 300a with four usage states 3ioa:3iod, each defining a different interaction with the HMD device 120. In usage state 310a the user is starting to use, and thus starting to wear, the HMD device 120. Depending on the technology used to detect if the user starts to interact (such as use and wear the HMD device 120) there can be different triggers for when usage state 310a is entered. In some non-limiting examples, usage state 310a is entered as soon as the user reaches out and grabs the HMD device 120. In other non-limiting examples, usage state 310a is entered only when the user puts the HMD device 120 over his / her head, and so on. In usage state 310b the user is using, and thus wearing, the HMD device. In usage state 310c the user is ending his / her use of the HMD device, and is thus removing the HMD device. In usage state 3iod the user is not using, and thus not wearing, the HMD device. In this respect, in usage state 3iod the user is not interacting with the HMD device 120. That is, in some embodiments, the first user’s interaction with the HMD device 120 is associated with a first usage state 310a representative of the first user starting to use the HMD device 120, a second usage state 310b representative of the first user using use the HMD device 120, and a third usage state 310c representative of the first user ending to use the HMD device 120. Each usage state 3ioa:3iod can be converted to a dynamic configuration input for a preferred physical environment state 3403:340?. Hence, the first physical environment state 3403:340? can be made dependent on which of the usage states 3ioa:3iod the HMD device 120 is in. That is, in some embodiments, the first user’s interaction with the HMD device 120 is associated with different usage states 3ioa:3iod, each representing a different type of the first user’s interaction with the HMD device 120, where the first physical environment state 340a:340P is dependent on which usage state 3ioa:3iod the HMD device 120 is in.

[0056] Further, the physical environment state 340a:340P for the first usage state 310a and the third usage stage 310c can be determined as a compromise of the physical environment state 34oa:34oP for the second usage state 310b and a physical environment state 340a:340P representative of the first user not using the HMD device 120. This can be used to provide a smooth transition from the fourth usage state 3iod to the second usage state 310b, as well as from the second usage state 310b to the fourth usage state 3iod, and thereby gradually effect a preferred state transition from an HMD-adapted state (i.e., the second usage state 310b) to a non- HMD-adapted state (i.e., the fourth usage state 3iod) and vice versa. The duration and behavior (linear or non-linear) of this transition can be pre-configured, or dynamically adapted based on other inputs from the physical environment 160 such that the HMD-adapted state is reached at the moment, or shortly before, the HMD device 120 is fully placed on the user’s head, and vice versa.

[0057] Further, as illustrated in Fig. 3(b), the HMD device 120 might be associated with different HMD states 32oa:32oK. Each of the HMD states 32oa:32oK is representative of its own HMD device settings. In this respect, there could be different types of HMD device settings, such as brightness level, hue level, sound level, XR isolation level, XR application category and state. Hence, each HMD state 32oa:32oK might be associated with its own combination of brightness level, hue level, sound level, XR isolation level, XR application category and state. The first physical environment state 340a: 340P might then be dependent on which of the HMD states 32oa:32oK the HMD device 120 is in.

[0058] Further, as illustrated in Fig. 3(c), the second user’s activity might be associated with different activity states 330a:330L. Each of the activity states 330a:330L represents a different type of the second user’s activity. In this respect, there could be different types of activities, such as position of the second user, gaze direction of the second user, use of any media device by the second user. Hence, each of the activity states 33Oa:33oL might be representative of its own types of user activities in terms of any, or any combination of: position of the second user, gaze direction of the second user, use of any media device by the second user. Further, as follows from the above, the second user is either using HMD device 120 or not using any HMD device 120. Therefore, in some embodiments, the second user’s activity is associated with a first activity state 330a:330L representative of the second user interacting with a further HMD device 120 and at least one second activity state 33oa:33oL representative of the second user not interacting with any HMD device 120. The second physical environment state 3403:340? might then be dependent on which activity state 33oa: 330L the second user is in.

[0059] Further aspects of the physical environment states 340a:340P will be disclosed next. Fig. 3(d) illustrates different physical environment states 340a: 340P. In some aspects, the physical environment states 34oa:34oP are related to different settings of the environment effectors 152A-1:152B-M. In particular, in some embodiments, each of the physical environment states 340a: 340P is be associated with its own at least one setting of the environment effectors 152A-I:152B-M in terms of any, or any combination of: ambient illumination level, ambient hue level, directed illumination level, directed hue level, ambient sound volume, directed sound level. In this respect, the physical environment states 340a:340P identified for each user might thus specify a specific setting of the environment effectors 152A-I:152B-M, such as a specific ambient illumination level, a specific ambient hue level, a specific directed illumination level, a specific directed hue level, a specific ambient sound volume, and / or a specific directed sound level.

[0060] Further aspects of how the controller no, 400 can determine the control signal 180 for the smart home controller 140 will be disclosed next. As disclosed above, the control signal 180 is determined based on a combination of the first physical environment state 340a: 340P and the second physical environment state 3403:340?. In this respect, the control signal 180 can be determined as a compromise of the physical environment state 340a: 340P preferred for the first user and the physical environment state 340a:340P preferred for the second user. In particular, in some embodiments, the combination of the first physical environment state 340a:340P and the second physical environment state 340a:340P yields a third physical environment state 340a:340P, and the control signal 180 is determined for the third physical environment state 340a:340P to be produced in the physical environment 160 by the environment effectors 152A-1:152B-M.

[0061] Further in this respect, there can be different ways for how the control signal 180 can be determined as a compromise of the physical environment state 340a:340P preferred for the first user and the physical environment state 340a:340P preferred for the second user. One way to achieve this is for the preferred physical environment states 34oa:34oP (one per user) to be combined in a weighted fashion. Therefore, in some embodiments, the first physical environment state 340a: 340P and the second physical environment state 340a:340P are weighted according to weighting factors when the control signal 180 is determined.

[0062] Further in this respect, the preferred physical environment states 340a: 340P for users wearing an HMD device 120 and the preferred physical environment states 340a:340P for users not wearing any HMD device 120 can be weighted differently. That is, in some embodiments, the weighting factors are different for HMD deviceusing users and non-HMD device-using users.

[0063] In general terms, one purpose of the weighting is to identify a third physical environment state 34oa:34oP that on average is as good as possible for all the users and that yield as little interference with respect to the environment states 340a:340P that are preferred per individual user. Therefore, in some embodiments, the weighting factors are determined as a compromise between optimizing the physical environment state 340a: 340P per individual user and minimizing interference among the physical environment states 340a:340P that are optimal for the individual users. Fig. 4 schematically illustrates, in terms of a number of structural units, the components of a controller 400 according to an embodiment. Processing circuitry 410 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 610 (as in Fig. 6), e.g. in the form of a storage medium 430. The processing circuitry 410 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0064] Particularly, the processing circuitry 410 is configured to cause the controller 400 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 430 may store the set of operations, and the processing circuitry 410 maybe configured to retrieve the set of operations from the storage medium 430 to cause the controller 400 to perform the set of operations. The set of operations maybe provided as a set of executable instructions.

[0065] Thus, the processing circuitry 410 is thereby arranged to execute methods as herein disclosed. The storage medium 430 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The controller 400 may further comprise a communications (comm.) interface 420 at least configured for communications with other entities, functions, nodes, and devices, as in the system 100 in Fig. 1. As such the communications interface 420 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 410 controls the general operation of the controller 400 e.g. by sending data and control signals to the communications interface 420 and the storage medium 430, by receiving data and reports from the communications interface 420, and by retrieving data and instructions from the storage medium 430. Other components, as well as the related functionality, of the controller 400 are omitted in order not to obscure the concepts presented herein.

[0066] Fig. 5 schematically illustrates, in terms of a number of functional modules, the components of a controller 500 according to an embodiment. The controller 500 of Fig. 5 comprises a number of functional modules; an identify module 510 configured to perform step S102, an identify module 520 configured to perform step S104, and a determine module 530 configured to perform step S106. The controller 500 of Fig. 5 may further comprise a number of optional functional modules, as represented by functional module 540. In general terms, each functional module 510:540 may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium 430 which when run on the processing circuitry makes the controller 400, 500 perform the corresponding steps mentioned above in conjunction with Fig 5. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules 510:540 may be implemented by the processing circuitry 410, possibly in cooperation with the communications interface 420 and / or the storage medium 430. The processing circuitry 410 may thus be configured to from the storage medium 430 fetch instructions as provided by a functional module 510:540 and to execute these instructions, thereby performing any steps as disclosed herein.

[0067] The controller no, 400, 500 maybe provided as a standalone device or as a part of at least one further device. For example, the controller 110, 400, 500 maybe provided in a smart home controller 140 or in an HMD device 120. further, a first portion of the instructions performed by the controller 110, 400, 500 may be executed in a first device, and a second portion of the of the instructions performed by the controller no, 400, 500 maybe executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the controller 110, 400, 500 maybe executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by a controller 110, 400, 500 residing in a cloud computational environment. Therefore, although a single processing circuitry 410 is illustrated in Fig. 4 the processing circuitry 410 may be distributed among a plurality of devices, or nodes. The same applies to the functional modules 510:540 of Fig. 5 and the computer program 620 of Fig. 6.

[0068] Fig. 6 shows one example of a computer program product 610 comprising computer readable storage medium 630. On this computer readable storage medium 630, a computer program 620 can be stored, which computer program 620 can cause the processing circuitry 410 and thereto operatively coupled entities and devices, such as the communications interface 420 and the storage medium 430, to execute methods according to embodiments described herein. The computer program 620 and / or computer program product 610 may thus provide means for performing any steps as herein disclosed.

[0069] In the example of Fig. 6, the computer program product 610 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 610 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 620 is here schematically shown as a track on the depicted optical disk, the computer program 620 can be stored in any way which is suitable for the computer program product 610.

[0070] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method for controlling environment effectors (152A-1:152B-M) in a physical environment (160), wherein the environment effectors (152A-1:152B-M) are controlled by a smart home controller (140), wherein the method is performed by a controller (no, 400), and wherein the method comprises: identifying (S102) a first physical environment state (3403:340?) for a first user in the physical environment (160) based on the first user’s interaction with a headmounted display, HMD, device (120) in the physical environment (160); identifying (S104) a second physical environment state (340a: 340P) for a second user in the physical environment (160) based on the second user’s activity in the physical environment (160); and determining (S106) a control signal (180) for the smart home controller (140) to control the environment effectors (152A-1:152B-M) to produce a physical environment state (340a:340P) in the physical environment (160), wherein the control signal (180) is based on a combination of the first physical environment state (340a:340P) and the second physical environment state (340a:340P).

2. The method according to claim 1, wherein the first user’s interaction with the HMD device (120) is associated with different usage states (3ioa:3iod), each representing a different type of the first user’s interaction with the HMD device (120), and wherein the first physical environment state (340a: 340P) is dependent on which usage state (3ioa:3iod) the HMD device (120) is in.

3. The method according to any preceding claim, wherein the HMD device (120) is associated with different HMD states (32oa:32oK), wherein each of the HMD states (32oa:32oK) is representative of its own HMD device settings in terms of any, or any combination of: brightness level, hue level, sound level, extended reality, XR, isolation level, XR application category and state, and wherein the first physical environment state (340a:340P) is dependent on which of the HMD states (32oa:32oK) the HMD device (120) is in.

4. The method according to any preceding claim, wherein the second user’s activity is associated with different activity states (330a: 330L), each representing adifferent type of the second user’s activity, and wherein the second physical environment state (3403:340?) is dependent on which activity state (330a:330L) the second user is in.

5. The method according to claim 4, wherein each of the activity states (330a: 330L) is representative of its own types of user activities in terms of any, or any combination of: position of the second user, gaze direction of the second user, use of any media device by the second user.

6. The method according to any preceding claim, wherein the second user’s activity is associated with a first activity state (33oa:33oL) representative of the second user interacting with a further HMD device (120) and at least one second activity state (330a: 330L) representative of the second user not interacting with any HMD device (120), and wherein the second physical environment state (3403:340?) is dependent on which activity state (330a:330L) the second user is in.

7. The method according to any preceding claim, wherein the first user’s interaction with the HMD device (120) is associated with a first usage state (310a) representative of the first user starting to use the HMD device (120), a second usage state (310b) representative of the first user using the HMD device (120), and a third usage state (310c) representative of the first user ending to use the HMD device (120), and wherein the first physical environment state (340a:340P) is dependent on which of the usage states (3ioa:3iod) the HMD device (120) is in.

8. The method according to claim 7, wherein the physical environment state (34oa:34oP) for the first usage state (310a) and the third usage stage (310c) is determined as a compromise of the physical environment state (340a:340P) for the second usage state (310b) and a physical environment state (34oa:34oP) representative of the first user not using the HMD device (120).

9. The method according to any preceding claim, wherein the combination of the first physical environment state (340a:340P) and the second physical environment state (340a:340P) yields a third physical environment state (340a:340P), and wherein the control signal (180) is determined for the third physical environment state (340a: 340P) to be produced in the physical environment (160) by the environment effectors (152A-1:152B-M).io. The method according to any preceding claim, wherein the first physical environment state (3403:340?) and the second physical environment state (340a:340P) are weighted according to weighting factors when the control signal (180) is determined. n. The method according to claim io, wherein the weighting factors are different for HMD device-using users and non-HMD device-using users.

12. The method according to claim io or n, wherein the weighting factors are determined as a compromise between optimizing the physical environment state (340a:340P) per individual user and minimizing interference among the physical environment states (340a:340P) that are optimal for the individual users.

13. The method according to any preceding claim, wherein each of the physical environment states (34oa:34oP) is associated with its own at least one setting of the environment effectors (152A-1:152B-M) in terms of any, or any combination of: ambient illumination level, ambient hue level, directed illumination level, directed hue level, ambient sound volume, directed sound level.

14. The method according to any preceding claim, wherein the controller (no, 400) is part of the smart home controller (140).

15. The method according to any of claims 1 to 13, wherein the controller (110, 400) is part of the HMD device (120).

16. A controller (no, 400) for controlling environment effectors (152A-1:152B-M) in a physical environment (160), wherein the environment effectors (152A-1:152B-M) are controlled by a smart home controller (140), the controller (110, 400) comprising processing circuitry (410), the processing circuitry being configured to cause the controller (110, 400) to: identify a first physical environment state (340a:340P) for a first user in the physical environment (160) based on the first user’s interaction with a head-mounted display, HMD, device (120) in the physical environment (160);identify a second physical environment state (340a: 340P) for a second user in the physical environment (160) based on the second user’s activity in the physical environment (160); and determine a control signal (180) for the smart home controller (140) to control the environment effectors (152A-1:152B-M) to produce a physical environment state (340a:340P) in the physical environment (160), wherein the control signal (180) is based on a combination of the first physical environment state (340a:340P) and the second physical environment state (340a:340P).

17. The controller (no, 400) according to claim 16, the processing circuitry (410) being configured to cause the controller (no, 400) to perform the method according to any of claims 2 to 15.

18. A system (100) for controlling environment effectors (152A-1:152B-M) in a physical environment (160), the system (100) comprising: a controller (110, 400) configured to perform a method according to any of claims 1 to 15 and thereby to determine a control signal (180); and a smart home controller (140) configured to control the physical environment (160) by providing an actuation signal (igoArigoB) based on the control signal (180) to the environment effectors (152A-1:152B-M) in the physical environment (160).

19. A computer program (320) for controlling environment effectors (152A-1:152B- M) in a physical environment (160), wherein the environment effectors (152A-1:152B- M) are controlled by a smart home controller (140), the computer program comprising computer code which, when run on processing circuitry (410) of a controller (no, 400), causes the controller (110, 400) to: identify (S102) a first physical environment state (340a:340P) for a first user in the physical environment (160) based on the first user’s interaction with a headmounted display, HMD, device (120) in the physical environment (160); identify (S104) a second physical environment state (340a:340P) for a second user in the physical environment (160) based on the second user’s activity in the physical environment (160); anddetermine (S106) a control signal (180) for the smart home controller (140) to control the environment effectors (152A-1:152B-M) to produce a physical environment state (340a: 340P) in the physical environment (160), wherein the control signal (180) is based on a combination of the first physical environment state (3403:340?) and the second physical environment state (340a:340P).

20. A computer program product (610) comprising a computer program (620) according to claim 19, and a computer readable storage medium (630) on which the computer program is stored.

Images