Real-world sensory stimuli in virtual worlds

Abstract

There is provided techniques for controlling how activation of a real-world sensory stimuli affects rendering of virtual worlds. A controller obtains an indication that a real-world sensory stimuli is scheduled to be activated at a real-world location by a sensory stimuli releasing device. The real-world location has a correspondence to a first virtual location in a first virtual world rendered at a first XR device and to a second virtual location in a second virtual world rendered at a second XR device. According to the indication, that the real-world sensory stimuli is scheduled to be activated is caused by a trigger. The controller, in response to having obtained the indication, performs an action that affects at least one of: rendering of the first virtual world at the first XR device, rendering of the second virtual world at the second XR device.

Description

[0001] REAL-WORLD SENSORY STIMULI IN VIRTUAL WORLDS

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, a controller, a computer program, and a computer program product for controlling how activation of a real-world sensory stimuli affects rendering of virtual worlds.

[0004] BACKGROUND

[0005] In general terms, virtual reality (VR) is a simulated experience that can be similar to or completely different from the real world. Non-limiting examples of VR technologies can be found in entertainment applications (such as video games, and movies), in education applications (such as medical or military training, robot navigation, construction modelling, and airplane simulation) and business applications (such as virtual meetings or computer conferencing). VR can be combined with augmented reality technologies and mixed reality technologies, sometimes referred to as extended reality technologies. VR systems are examples of more general extended reality (XR) systems, that in addition to VR systems also include augmented reality (AR) systems, and mixed reality (MR) systems.

[0006] XR systems commonly use either headsets or multi-projected environments to generate realistic images, sounds and other sensations that simulate a user's physical presence in a virtual environment defining the simulated experience. A user using a piece of XR equipment is able to look around in the simulated experience, move around in it, and interact with virtual features or items. The effect is commonly created by headsets comprising a head-mounted display with a small screen to be placed in front of the eyes of the user, but can also be created through specially designed rooms with multiple large screens. VR systems typically incorporates auditory and visual feedback, but may also allow other types of sensory and force feedback through haptic technologies.

[0007] While visual and auditory stimuli have been at the forefront when it comes to XR development, further immersion of the user in the virtual space might be achieved by providing stimuli that affect also other senses. As an illustrative example, attempts have been made to use olfactory stimuli (i.e., scents, smells, fragrances, etc.) to enhance audio-visual experiences, although often with limited success. Still, users P107946W001

[0008] 2 who engage with XR systems that have an olfactory component congruent to the virtual environment report a higher level of immersion and satisfaction, even when the fragrances are light enough not to be consciously noticed. In this respect, one issue concerns how to provide olfactory stimuli, or other types of real-world sensory stimuli such as audio and / or visual stimuli, etc. to users without overloading or overwriting other senses. Taking olfactory stimuli as an example, the human olfactory system’s sensitivity to smells makes human beings susceptible to virtual smells lingering when they should be gone. Likewise, if a scent is perceived as incongruent with other types of real-world sensory stimuli immersion might even be lost.

[0009] Technology is available for fitting conventional XR headsets with olfactory devices. For example, an olfactory device can be attached as a peripheral device to some XR headsets. In one non-limiting example, such a peripheral device can be based on discs containing scent stored using micro-encapsulation techniques, possibly having scent guide pipes directing the scents to the user’s nose. Some technologies are based on using a liquid perfume-based cartridge system, where a block with a limited number of scents is inserted into a device mounted on an XR headset. Scents can then be released as the user engages with the virtual environment. Due to the cartridges requiring perfume vials for each scent, the number of scents is limited to only a few that can be utilized at the same time. Since the devices need to fit the cartridges along with the necessary mechanics, they tend to be significantly bulky as well, often being half an ordinary XR headset in size. This limits not only the different scents that can be released but also how many times any given scent can be released. A further consequence of the above is that each user is limited to the scents that can be released by the olfactory device attached to its own XR headset.

[0010] While these issues apply to XR devices with respect to handling of scents in particular, the same issues apply also to XR devices with respect to handling of other types of real-world sensory stimuli, such as auditory feedback, visual feedback, sensory and force feedback, etc.

[0011] SUMMARY

[0012] An object of embodiments herein is to address the above issues. P107946W001

[0013] 3

[0014] A particular object is to take advantage of real-world sensory stimuli that is available for multiple users.

[0015] A particular object is to enable communication between XR devices in the context of activation of real-world sensory stimuli.

[0016] A particular object is to enable coordinated activation of real-world sensory stimuli for multiple XR devices.

[0017] According to a first aspect there is presented a controller for controlling how activation of a real-world sensory stimuli affects rendering of virtual worlds. The controller is configured to communicate with at least a first XR device, a second XR device, and sensory stimuli releasing devices. The controller comprises processing circuitry. The processing circuitry is configured to cause the controller to obtain an indication that a real-world sensory stimuli is scheduled to be activated at a real- world location by one of the sensory stimuli releasing devices. The real-world location has a correspondence to a first virtual location in a first virtual world rendered at the first XR device and to a second virtual location in a second virtual world rendered at the second XR device. According to the indication, that the real-world sensory stimuli is scheduled to be activated is caused by a trigger. The processing circuitry is configured to cause the controller to, in response to having obtained the indication, perform an action that affects at least one of: rendering of the first virtual world at the first XR device, rendering of the second virtual world at the second XR device.

[0018] According to a second aspect there is presented a computer program for controlling how activation of a real-world sensory stimuli affects rendering of virtual worlds. The computer program comprises computer code which, when run on processing circuitry of a controller, causes the controller to perform actions. The controller is configured to communicate with at least a first XR device, a second XR device, and sensory stimuli releasing devices. One action comprises the controller to obtain an indication that a real-world sensory stimuli is scheduled to be activated at a real-world location by one of the sensory stimuli releasing devices. The real-world location has a correspondence to a first virtual location in a first virtual world rendered at the first XR device and to a second virtual location in a second virtual world rendered at the second XR device. According to the indication, that the real-world sensory stimuli is scheduled to be activated is caused by a trigger. One action comprises the controller P107946W001

[0019] 4 to, in response to having obtained the indication, perform an action that affects at least one of: rendering of the first virtual world at the first XR device, rendering of the second virtual world at the second XR device.

[0020] According to a third aspect there is presented a computer program product comprising a computer program according to the second aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0021] Advantageously, these aspects do not suffer from the above issues.

[0022] Advantageously, these aspects take advantage of real-world sensory stimuli that is available for multiple users when controlling how activation of a real-world sensory stimuli affects rendering of virtual worlds.

[0023] Advantageously, these aspects provide control with respect to which type of real- world sensory stimuli is activated at which location and at which point in time.

[0024] Advantageously, these aspects provide control with respect to how activation of a real-world sensory stimuli affects rendering of virtual worlds.

[0025] Advantageously, these aspects enable communication between XR devices in the context of activation of real-world sensory stimuli.

[0026] Advantageously, these aspects enable coordinated activation of real-world sensory stimuli for multiple XR devices.

[0027] Advantageously, by taking advantage of multiple available sensory stimuli releasing devices, these aspects enable real-world sensory stimuli to be activated with short delay.

[0028] Advantageously, by taking advantage of multiple available sensory stimuli releasing devices whose activation can be coordinated, these aspects enable a high variation of different types of real-world sensory stimuli to be activated. P107946W001

[0029] 5

[0030] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0031] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0032] BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0034] Fig. 1 schematically illustrates an XR device according to an embodiment;

[0035] Fig. 2 schematically illustrates an XR communication system according to an embodiment;

[0036] Fig. 3 is a flowchart of methods according to embodiments;

[0037] Figs. 4, 5, and 6 are signalling diagrams of methods according to embodiments;

[0038] Fig. 7 is a schematic diagram showing functional units of a controller according to an embodiment;

[0039] Fig. 8 is a schematic diagram showing functional modules of a controller according to an embodiment; and

[0040] Fig. 9 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.

[0041] DETAILED DESCRIPTION

[0042] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different P107946W001

[0043] 6 forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0044] Fig. i is a schematic diagram illustrating an XR device noa, nob according to an embodiment. The XR device noa, nob is intended to be used by a user (see, Fig. 2). There could be different types of XR devices noa, nob. Fig. 1 schematically illustrates an example where the XR devices noa, nob are provided as a head- mountable display taking the shape of a pair of glasses. The XR device noa, nob comprises a user interface 120a, 120b for rendering virtual worlds 120a, 120b (see, Fig. 2). In some examples, the user interface 120a, 120b may comprise two small high-resolution monitors adapted to provide separate images for each eye for stereoscopic graphics rendering a 3D virtual environment, a binaural audio system, positional and rotational real-time head tracking for six degrees of movement. The XR device noa, nob may be provided with motion controls with haptic feedback for physically interacting within the virtual world in an intuitive way. In particular, the XR device noa, nob further comprises a communication interface 114a, 114b for communicating with another XR device noa, nob and / or with a controller (see, Fig. 2). Although illustrated as an antenna, for example suitable for any type of radio communications (cellular was well as non-cellular), the communication interface 114a, 114b might be a wired communication interface, an infrared communication interface, a visible light communication interface, or some other kind of communication interface. In this respect, the XR device noa, nob might be configured for direct communication with each other or for communicating with each other via at least one other device, such as a controller 200 (see, Fig. 2), a mobile phone, personal computer, gaming machine, a server, a cloud computational system, or the like. The XR device noa, nob further comprises a processing unit 116a, 116b, for controlling operation of the XR device noa, nob. With intermediate reference to Fig. 2, although the controller 200 is illustrated as a separate device in Fig. 2, the controller 200 might equally be implemented partly or fully by the processing unit 116a, 116b in at least one XR device noa, nob. In the example of Fig. 1, the XR device P107946W001

[0045] 7 noa, nob further comprises a speaker and / or microphone n8a, n8b or other type of audio playing or recording device. Audio recorded at one XR device noa, nob can be transferred to be played out at another XR device noa, nob via the communication interface 114a, 114b. In the example of Fig. 1, the XR device noa, nob further comprises a scent releasing device, as an example of a sensory stimuli releasing device 140a, 140b, configured to release one or more scents. The release of the scent can be controlled by the processing unit 116a, 116b. However, the sensory stimuli releasing device 140a, 140b might also be external and separated from the XR device noa, nob (see, Fig. 2). Each XR device noa, nob might be configured to track movements made by its user, or equally, as made by the avatar of the user in the virtual space. The processing unit 116a, 116b of the XR device noa, nob is configured to render a virtual world 120a, 120b. In some aspects, the processing unit 116a, 116b of the XR device noa, nob might further be configured to render, or at least record, the position of the user, or, equally, of the avatar, in the virtual world 120a, 120b. This information might also, via the communication interface 114a. 114b, be provided to other XR devices noa, nob, either directly, or via a network to which all XR device noa, nob are operatively connected. The processing unit 116a, 116b further keeps track of the position of other users, or equally, of other avatars in the virtual space. In order to do so, information is communicated to the processing unit 116a, 116b via the communication interface 114a, 114b, for example from the network.

[0046] Fig. 2 is a schematic diagram illustrating an XR communication system 100 where embodiments presented herein can be applied. The XR communication system 100 comprises XR devices noa, nob, sensory stimuli releasing devices 140a, 140b, and a controller 200. Each XR device noa, nob is used by a respective user 1160a, 160b. As disclosed above, each of the XR devices noa, nob is configured to render a virtual world 120a, 120b for its user 160a, 160b. In the illustrated example, the virtual world 120a rendered at the XR device noa worn by user 160a comprises an open fire and the virtual world 120b rendered at the XR device nob worn by user 160b comprises a flower. A sensory stimuli releasing device 140a in the vicinity of user 160a might therefore be configured to release real-world sensory stimuli 150a in terms of a smoke scent whereas sensory stimuli releasing device 140b in the vicinity of user 160b might be configured to release real-world sensory stimuli 150b in terms of a floral scent. These sensory stimuli releasing devices 140a, 140b and real-world P107946W001

[0047] 8 sensory stimuli 150a, 150b are just examples and also other types of sensory stimuli releasing devices 140a, 140b and real-world sensory stimuli 150a, 150b might be provided. For example, the sensory stimuli releasing devices 140a, 140b might be a loudspeaker for playing out audio, a display screen for displaying visual content, a radiator for impacting environmental temperature, a fan for creating a wind, or any kind of haptic feedback device for creating vibrations or other types of haptic signals. Therefore, activation of the real-world sensory stimuli 150a, 150b involves any, or any combination of: release a scent, play out an audio clip, play out a video clip, affect temperature of a radiator, start a fan, provide a haptic signal. In some examples, some of the sensory stimuli releasing device 140a, 140b are configured for activation of two or more different real-world sensory stimuli 150a, 150b at the same time (e.g., to release two different scents). For example, a sensory stimuli releasing device 140a, 140b might be configured to release different scents from different valves, allowing multiple scents to be active in the system at once. In this way a first scent can be released in front of the user’s nose, whilst a second scent is released around the same user. In yet further examples, some of the sensory stimuli releasing device 140a, 140b are configured for activation of two or more different types of real-world sensory stimuli 150a, 150 at the same time (e.g., to release a scent whilst at the same time playing out an audio clip, etc.). In other examples, each sensory stimuli releasing device 140a, 140b is configured for activation of only a single type of real-world sensory stimuli 150a, 150b. In some examples, the sensory stimuli releasing devices 140a, 140b are dedicated for activating real-world sensory stimuli 150a, 150. In other examples the sensory stimuli releasing devices 140a, 140b are integrated with, part of, or collocated with, other types of devices that might, or might not, be worn by the users 160a, 160b. Examples of such devices that might be worn by the users 160a, 160b are smart watches, XR device hand controllers, mobile phones, etc. Further, the sensory stimuli releasing devices 140a, 140b might be configured to provide information back to the controller 200 regarding real-world sensory stimuli availability and activation of real-world sensory stimuli as performed by the sensory stimuli releasing devices 140a, 140b.

[0048] As schematically illustrated by arrows 130a, 130b, the controller 200 is configured to communicate with the XR devices 110a, nob. The controller 200 might for this purpose be operatively connected to, or be part of, a wireless, or wired, P107946W001

[0049] 9 communication system, where the controller 200 is configured to orchestrate, and act as a hub for, the communication between the XR devices 110a, nob. For this purpose, the functionality of the controller 200 might be provided in a modem, or an access node, such as a radio access network node. For notation purposes and without imposing any hierarchical relationship among the XR devices 110a, nob, XR device 110a is hereinafter denoted a first XR device 110a, whereas XR device nob is hereinafter denoted a second XR device nob. Further, as is appreciated by the skilled person, the XR communication system 100 might comprise a plurality of XR devices, each having its own user interface, and each being configured for communication with the other XR devices, either directly or via the controller 200. It is here emphasized that the schematic diagram in Fig, 2 only represents one illustrative example of an environment in which the herein disclosed embodiments could be applied.

[0050] Issues with present technologies for activating real-world sensory stimuli 150a, 150b in virtual worlds 120a, 120b have been mentioned above.

[0051] Further in this context, it could be advantageous to take advantage of real-world sensory stimuli that is available for multiple users. It could also be advantageous to enable communication between XR devices in the context of activation of real-world sensory stimuli. It could also be advantageous to enable coordinated activation of real-world sensory stimuli for multiple XR devices.

[0052] The embodiments disclosed herein therefore relate to techniques for controlling how activation of a real-world sensory stimuli 150a, 150b affects rendering of virtual worlds 120a, 120b. In order to obtain such techniques, there is provided a controller 200, a method performed by the controller 200, a computer program product comprising code, for example in the form of a computer program, that when run on a controller 200, causes the controller 200 to perform the method.

[0053] In some aspects, two or more XR devices 110a, nob in a shared physical space share and utilize data regarding real-world sensory stimuli availability through communication with the controller 200. Therefore, the controller 200 might be configured to facilitate an operative connection between the first XR device 110a and the second XR device nob (as well as between other XR devices). In turn, the controller 200 is in communication with a set of sensory stimuli releasing devices P107946W001

[0054] 10

[0055] 140a, 140b for controlling when and where real-world sensory stimuli should be activated (i.e., which one or more type of real-world sensory stimuli to be activated when and by which one or more sensory stimuli releasing devices 140a, 140b). Alternatively, the XR devices 110a, nob are in direct communication with the set of sensory stimuli releasing devices 140a, 140b. Communication between the XR devices 110a, nob and the controller 200 as well as between the controller 200 and the sensory stimuli releasing devices 140a, 140b and / or between the XR devices 110a, nob and the sensory stimuli releasing devices 140a, 140b might be facilitated over a wireless network, such as a wide area, or cellular, wireless network or a local area wireless network based on technologies such as WiFi, fifth or sixth generation telecommunication systems, etc. In this way, the XR devices 110a, nob and the sensory stimuli releasing devices 140a, 140b might share information with each other, either directly, or indirectly via the controller 200.

[0056] In some non-limiting examples, the XR devices 110a, nob might share the following information (or part thereof): positioning of the XR devices 110a, nob in real-world space or relative to other devices in the network, positioning of the users in the virtual worlds, positioning of virtual sensory stimuli triggers in the application virtual worlds, predicted future location(s) of the users in the real-world space and / or the virtual worlds.

[0057] In some non-limiting examples, the sensory stimuli releasing devices 140a, 140b might share the following information (or part thereof): positioning of the sensory stimuli releasing devices 140a, 140b in real-world space or relative to other devices in the network, real-world sensory stimuli availability, properties of available real-world sensory stimuli (such as location, strength, permanence, etc.), real-world sensory stimuli activation (past, present, and / or scheduled).

[0058] Utilizing the information published by the devices, any given XR device 110a, nob is able to get the positioning of other devices (both in terms of other XR devices and sensory stimuli releasing devices 140a, 140b) relative to itself. The positioning can be translated into positioning in the virtual environment, allowing XR applications to utilize the information within the virtual worlds. An example of this would be to shape the virtual worlds in such a way that a user is triggered to move closer towards, or farther away from, a given sensory stimuli releasing device in the real-world space, P107946W001

[0059] 11 based on activation of real-world sensory stimuli at this given sensory stimuli releasing device. Furthermore, the XR devices noa, nob are also able to send requests (either to the controller 200 or directly to the sensory stimuli releasing devices 140a, 140b) for a certain real-world sensory stimuli to be activated, or even to block a certain real-world sensory stimuli from being activated.

[0060] Fig. 3 is a flowchart illustrating embodiments of methods for controlling how activation of a real-world sensory stimuli 150a, 150b affects rendering of virtual worlds 120a, 120b. The methods are performed by the controller 200. The controller 200 is configured to communicate with at least a first XR device 110a, a second XR device nob, and sensory stimuli releasing devices 140a, 140b. The methods are advantageously provided as computer programs 920. As disclosed above, the controller 200 might be configured to facilitate an operative connection between the first XR device noa and the second XR device nob.

[0061] S102: The controller 200 obtains an indication that a real-world sensory stimuli 150a, 150b is scheduled to be activated at a real-world location by one of the sensory stimuli releasing devices 140a, 140b.

[0062] The real-world location has a correspondence to a first virtual location in a first virtual world 120a rendered at the first XR device noa and to a second virtual location in a second virtual world 120b rendered at the second XR device nob.

[0063] According to the indication, that the real-world sensory stimuli 150a, 150b is scheduled to be activated is caused by a trigger.

[0064] S114: The controller 2ooin response to having obtained the indication (and possibly after having performed some optional actions as will be disclosed below) perform an action that affects at least one of: rendering of the first virtual world 120a at the first XR device noa, rendering of the second virtual world 120b at the second XR device nob.

[0065] In this way, communication and collaboration between multiple XR devices noa, nob can be facilitated, thereby improving the XR experience for two or more users 160a, 160b sharing a (physical) space but not necessarily sharing the same virtual world (e.g., two users playing two different games at the same physical location). P107946W001

[0066] 12

[0067] Embodiments relating to further details of controlling how activation of a real-world sensory stimuli 150a, 150b affects rendering of virtual worlds 120a, 120b as performed by the controller 200 will now be disclosed with continued reference to Fig. 3-

[0068] At which real-world location the real-world sensory stimuli 150a, 150b is scheduled to be activated and by which of the sensory stimuli releasing devices 140a, 140b the real- world sensory stimuli 150a, 150b is scheduled to be activated might be based on real- world location information of the first XR device 110a and real-world location information of the sensory stimuli releasing devices 140a, 140b. Examples of such location information have been disclosed above.

[0069] There may be different examples of triggers causing the real-world sensory stimuli 150a, 150b to be scheduled to be activated. In some embodiments, the trigger is any of: an action taken in the first virtual world 120a by a user 160a of the first XR device 110a, a timer expiring. Hence, that the real-world sensory stimuli 150a, 150b is scheduled to be activated might not only be based on direct actions taken by the users 160a, 160b but also be based on other properties, such as timers, sound cues, visual information, or a combination of these.

[0070] As will be further disclosed below, the action performed by the controller 200 in S114 can either be an action to directly activate the real-world sensory stimuli 150a, 150b or an action that indirectly causes the real-world sensory stimuli 150a, 150b to be activated. An illustrating example of the former is that the user 160a opens a bottle of perfume in the first virtual world 120a which might trigger activation of a real-world sensory stimuli in terms of release of a certain scent. An illustrating example of the latter is that the user 160a enters a room full of flowers in the first virtual world 120a which, again, might trigger activation of a real-world sensory stimuli in terms if release of a certain scent.

[0071] Further, there might be different examples of actions performed by the controller 200 in S114 that affects at least one of: rendering of the first virtual world 120a at the first XR device 110a, rendering of the second virtual world 120b at the second XR device nob. In some examples, the action is based on a location of the second XR device nob relative the real-world location at which the real-world sensory stimuli 150a, 150b is scheduled to be activated. P107946W001

[0072] 13

[0073] In some aspects, the indication that the real-world sensory stimuli 150a, 150b is scheduled to be activated is obtained as a request from the first XR device 110a. The controller 200 might then notify the second XR device nob about this. Therefore, in some embodiments, the processing circuitry 210 is configured to cause the controller 200 to perform (optional) action S104.

[0074] S104: The controller 200 notifies the second XR device nob that the real-world sensory stimuli 150a, 150b is scheduled to be activated.

[0075] Further aspects of this will be disclosed below with reference to the signaling diagrams of Fig. 4, Fig. 5, and Fig. 6.

[0076] In some aspects, in response to having received a request from the first XR device 110a to activate the real-world sensory stimuli 150a, 150b, the controller 200 facilitates the activation of the real-world sensory stimuli 150a, 150b from at least one of the sensory stimuli releasing devices 140a, 140b

[0077] Therefore, in some embodiments, the processing circuitry 210 is configured to cause the controller 200 to perform (optional) action S106.

[0078] S106: The controller 200 request one of the sensory stimuli releasing devices 140a, 140b to activate the real-world sensory stimuli 150a, 150b.

[0079] Further aspects of this will be disclosed below with reference to the signaling diagrams of Fig. 4 and Fig. 6.

[0080] In some aspects, the second XR device nob, in response to having received the notification issued by the controller 200 in S104, requests that the activation of the real-world sensory stimuli 150a, 150b is blocked. Therefore, in some embodiments, the processing circuitry 210 is configured to cause the controller 200 to perform (optional) action S108.

[0081] S108: The controller 200 obtains a response from the second XR device nob requesting blocking activation of the real-world sensory stimuli 150a, 150b. The action taken in S114 could then be based on the response.

[0082] Further aspects of this will be disclosed below with reference to the signaling diagrams of Fig. 5 and Fig. 6. P107946W001

[0083] 14

[0084] In some aspects, the request from the first XR device noa of the activation of the real-world sensory stimuli 150a, 150b is rejected, or denied. Therefore, in some embodiments, the processing circuitry 210 is configured to cause the controller 200 to perform (optional) action S110.

[0085] S110: The controller 200 notifies the first XR device 110a that the activation of the real-world sensory stimuli 150a, 150b has been blocked.

[0086] Further aspects of this will be disclosed below with reference to the signaling diagram of Fig. 5.

[0087] In case the activation of the real-world sensory stimuli 150a, 150b has been blocked this implies that any request received from the first XR device 110a to activate the real-world sensory stimuli 150a, 150b has been rejected or denied. In case the content of the first virtual world 120a (as rendered at the first XR device 110a for the first user 160a) is dependent on the real-world sensory stimuli 150a, 150b, this might imply that the virtual content of the first virtual world 120a needs to be changed. Therefore, in some embodiments, the action taken by the controller in S114 involves changing virtual content of the first virtual world 120a.

[0088] However, in other aspects, the response from the second XR device nob requesting blocking activation of the real-world sensory stimuli 150a, 150b is denied, or rejected. This implies that the request from the first XR device noa of the activation of the real-world sensory stimuli 150a, 150b is accepted. Therefore, in some embodiments, the processing circuitry 210 is configured to cause the controller 200 to perform (optional) action S112.

[0089] S112: The controller 200 notifies the second XR device nob that the real-world sensory stimuli 150a, 150b has been activated.

[0090] Further aspects of this will be disclosed below with reference to the signaling diagrams of Fig. 4 and Fig. 6.

[0091] In case the content of the second virtual world 120b (as rendered at the second XR device nob for the second user 160b) is dependent on the real-world sensory stimuli 150a, 150b, this might imply that the virtual content of the second virtual world 120b P107946W001

[0092] 15 needs to be changed. Therefore, in some embodiments, the action taken by the controller in S114 involves changing virtual content of the second virtual world 120b.

[0093] There might be different examples of virtual content of the virtual worlds 120a, 120b. In some non-limiting examples, the virtual content is, or at least is represented by, a map per each of the virtual worlds 120a, 120b. In other non-limiting examples, the virtual content is, or at least is represented by, a storyline per each of the virtual worlds 120a, 120b.

[0094] In further detail, in case a certain real-world sensory stimuli 150a, 150b has been scheduled to be activated, the XR devices 110a, nob (or the controller 200 on behalf of the XR devices 110a, nob) might take an action that depends on that the certain real-world sensory stimuli 150a, 150b will be activated at a certain location at a certain time. In some embodiments, in case the virtual content is a map of the second virtual world 120b, the action taken by the controller 200 in S114 might involve changing the map to either guide the user 160b of the second XR device nob towards the real-world location or away from the real-world location (i.e., the location where the real-world sensory stimuli 150a, 150b is scheduled to be activated). This could take the form of one (or both) of the virtual worlds 120a, 120b being changed to accommodate a real-world sensory stimuli 150a, 150b that has a high likelihood of being activated. Consequences of this have already been disclosed with reference to actions S104, S110, and S112.

[0095] Further, in some embodiments, in case the virtual content is a storyline of the second virtual world 120b, the action taken by the controller 200 in S114 might involve changing the storyline to either include or exclude the real-world sensory stimuli 150a, 150b. Consequences of this have already been disclosed with reference to actions S104, S110, and S112.

[0096] Different types of real-world sensory stimuli 150a, 150b, and how they might be activated, have been described above. Therefore, in some non-limiting examples, activation of the real-world sensory stimuli 150a, 150b involves any, or any combination of: release a scent, play out an audio clip, play out a video clip, affect temperature of a radiator, start a fan, provide a haptic signal, etc. P107946W001

[0097] 16

[0098] Further, in some examples if real-world sensory stimuli 150a, 150b that has been activated is found to be irrelevant (for example that the trigger to activate the real- world sensory stimuli 150a, 150b has been withdrawn, expired, or actively removed), the controller 200 might perform an action not only to deactivate the sensory stimuli releasing devices 140a, 140b from which the real-world sensory stimuli 150a, 150b was released, played out, etc. (depending on the type of real-world sensory stimuli 150a, 150b) but also to perform some mitigating action to remove any traces of the real-world sensory stimuli 150a, 150b, if needed. In case the real-world sensory stimuli 150a, 150b is a scent that has been released, the controller 200 might instruct one of the sensory stimuli releasing devices 140a, 140b to release a neutralizing scent and / or sensory stimuli releasing devices 140a, 140b in terms of a fan to be started to blow a stream of air to disperse the released scent.

[0099] Three different embodiments will be disclosed next with reference to the signaling diagrams of Fig. 4, Fig. 5, and Fig. 6. In all these embodiments it is assumed that a first user 160a is wearing a first XR device 110a and a second user 160b is wearing a second XR device nob. It is further assumed that the first user 160a and the second user 160b are located within the same real-world location, e.g., in the same room. It is further assumed that the first XR device 110a is rendering a first virtual world 120a and that the second XT device nob is rendering a second virtual world 120b, different from the first virtual world 120a. In all these embodiments, the real-world sensory stimuli is a scent and thus, activation of the real-world sensory stimuli implies that a scent is released from one or more of the sensory stimuli releasing devices 140a, 140b.

[0100] Fig. 4 represents a scenario where the first XR device 110a requests some real-world sensory stimuli to be activated and that the XR second device nob is notified about this.

[0101] 5201. Message “Update physical (L_A_1) and virtual location data”: The first XR device 110a provides information to the controller 200 about the physical location L_A_1 of the first user 160a as well as the location of the first user 160a in the first virtual world 120a.

[0102] 5202. Message “Update physical (L_B_1) and virtual location data”: The second XR device nob provides information to the controller 200 about the physical location P107946W001

[0103] 17

[0104] L_B_i of the second user 160b as well as the location of the second user 160b in the second virtual world 120b.

[0105] 5203. Message “Request / notify about possible scent S_A release at location L_A_2 in X seconds”: The first XR device 110a provides either a request or a notification to the controller 200 about the release of a scent at a physical location L_A_2 in X seconds. The controller 200 forwards a notification of this to the second XR device nob. The second XR device nob thus gets the notification about the scent S_A scheduled to be released at location L_A_2 in X seconds. The second XR device nob does not object to the release. One reason for this could be that the location L_A_2 does not overlap with a location (say, L_B_2) where the second user 160b is predicted to be located in X seconds. A further reason for this could be that the scent S_A to be released could be incorporated into the second virtual world 120b.

[0106] 5204. Message “Release scent S_A”: The controller 200 instructs the sensory stimuli releasing device at location L_A_2 to release the scent S_A.

[0107] 5205. Message “Confirm release scent S_A at location L_A_2”: The controller 200 notifies the first XR device 110a and the second XR device nob that the scent S_A has been released at location L_A_2.

[0108] Fig. 5 represents a scenario where the first XR device 110a requests some real-world sensory stimuli to be activated and that the second XR device nob is notified about this, and further where the second XR device nob requests the activation to be blocked.

[0109] 5301. Message “Update physical (L_A_1) and virtual location data”: The first XR device 110a provides information to the controller 200 about the physical location L_A_1 of the first user 160a as well as the location of the first user 160a in the first virtual world 120a.

[0110] 5302. Message “Update physical (L_B_1) and virtual location data”: The second XR device nob provides information to the controller 200 about the physical location L_B_i of the second user 160b as well as the location of the second user 160b in the second virtual world 120b. P107946W001

[0111] 18

[0112] 5303. Message “Request / notify about possible scent S_A release at location L_A_2 at time X seconds”: The first XR device 110a provides either a request or a notification to the controller 200 about the release of a scent at a physical location L_A_2 at time X. The controller 200 forwards a notification of this to the second XR device nob. The second XR device nob thus gets the notification about the scent S_A scheduled to be released at location L_A_2 at time X.

[0113] 5304. Message “Request blocking of scent S_A release at location L_A_2 at time X”: The second XR device nob requests that scent S_A is not released at location L_A_2 at time X. One reason for this could be that the location L_A_2 overlaps with a location (say, L_B_2) where the second user 160b is predicted to be located at time X. A further reason for this could be that the scent S_A to be released could not be incorporated into the second virtual world 120b.

[0114] 5305. Message “Confirm blocking of scent S_A release at location L_A_2 at time X”: The controller 200 informs the first XR device 110a as well as the second XR device nob that scent S_A will not be released at location L_A_2 at time X.

[0115] Fig. 6 represents a scenario where the first XR device 110a requests some real-world sensory stimuli to be activated and that the second XR device nob is notified about this, and further where the second XR device nob requests the activation to be blocked, and yet further where the first XR device 110a rejects the request for the activation to be blocked.

[0116] 5401. Message “Update physical (L_A_1) and virtual location data”: The first XR device 110a provides information to the controller 200 about the physical location L_A_1 of the first user 160a as well as the location of the first user 160a in the first virtual world 120a.

[0117] 5402. Message “Update physical (L_B_1) and virtual location data”: The second XR device nob provides information to the controller 200 about the physical location L_B_i of the second user 160b as well as the location of the second user 160b in the second virtual world 120b.

[0118] 5403. Message “Request / notify about possible scent S_A release at location L_A_2 at time X seconds”: The first XR device 110a provides either a request or a notification P107946W001

[0119] 19 to the controller 200 about the release of a scent at a physical location L_A_2 at time X. The controller 200 forwards a notification of this to the second XR device nob. The second XR device nob thus gets the notification about the scent S_A scheduled to be released at location L_A_2 at time X.

[0120] 5404. Message “Request blocking of scent S_A release at location L_A_2 at time X”: The second XR device nob requests that scent S_A is not released at location L_A_2 at time X. One reason for this could be that the location L_A_2 overlaps with a location (say, L_B_2) where the second user 160b is predicted to be located at time X. A further reason for this could be that the scent S_A to be released does not match the present content of the second virtual world 120b as rendered to the second user 160b.

[0121] 5405. Message “Reject blocking of scent S_A release at location L_A_2 at time X”: The controller 200 informs the second XR device nob that the request to block release of scent S_A at location L_A_2 at time X has been rejected.

[0122] 5406. Message “Handle rejection of blocking of scent S_A release at location L_A_2 at time X”: The second XR device nob handles the fact that scent S_A indeed will be released at location L_A_2 at time X. One way for the second XR device nob to handle this is to modify the second virtual world 120b. For example, the second XR device nob might modify the second virtual world 120b such that the second user 160b will not be located at a location (say, L_B_2) that overlaps with the location L_A_2 at time X. For example, the second XR device nob might modify the second virtual world 120b such that the scent S_A to be released indeed is incorporated in the second virtual world 120b.

[0123] 5407. Message “Release scent S_A”: The controller 200 instructs the sensory stimuli releasing device at location L_A_2 to release the scent S_A at time X.

[0124] 5408. Message “Confirm release scent S_A at location L_A_2”: The controller 200 notifies the first XR device 110a and the second XR device nob that the scent S_A has been released at location L_A_2.

[0125] Fig. 7 schematically illustrates, in terms of a number of functional units, the components of a controller 200 according to an embodiment. Processing circuitry P107946W001

[0126] 20

[0127] 210 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 910 (as in Fig. 9), e.g. in the form of a storage medium 230. The processing circuitry 210 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0128] Particularly, the processing circuitry 210 is configured to cause the controller 200 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 230 may store the set of operations, and the processing circuitry 210 maybe configured to retrieve the set of operations from the storage medium 230 to cause the controller 200 to perform the set of operations. The set of operations maybe provided as a set of executable instructions.

[0129] Thus the processing circuitry 210 is thereby arranged to execute methods as herein disclosed. The storage medium 230 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The controller 200 may further comprise a communications (comm.) interface 220 at least configured for communications with other entities, functions, nodes, and devices, as illustrated in Figs. 1 to 6. As such the communications interface 220 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 210 controls the general operation of the controller 200 e.g. by sending data and control signals to the communications interface 220 and the storage medium 230, by receiving data and reports from the communications interface 220, and by retrieving data and instructions from the storage medium 230. Other components, as well as the related functionality, of the controller 200 are omitted in order not to obscure the concepts presented herein.

[0130] Fig. 8 schematically illustrates, in terms of a number of functional modules, the components of a controller 200 according to an embodiment. The controller 200 of Fig. 8 comprises a number of functional modules; an obtain module 210a configured to perform step S102, and an action module 210g configured to perform step S114. The controller 200 of Fig. 8 may further comprise a number of optional functional modules, such as any of a notify module 210b configured to perform step S104, a P107946W001

[0131] 21 request module 210c configured to perform step Sio6, an obtain module 2iod configured to perform step Sio8, a notify module 2ioe configured to perform step Sno, and a notify module 2iof configured to perform step S112.

[0132] In general terms, each functional module 2ioa:2iog may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium 230 which when run on the processing circuitry makes the controller 200 perform the corresponding steps mentioned above in conjunction with Fig 8. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules 2ioa:2iog maybe implemented by the processing circuitry 210, possibly in cooperation with the communications interface 220 and / or the storage medium 230. The processing circuitry 210 may thus be configured to from the storage medium 230 fetch instructions as provided by a functional module 210a: 210g and to execute these instructions, thereby performing any steps as disclosed herein.

[0133] The controller 200 maybe provided as a standalone device or as a part of at least one further device. For example, the controller 200 maybe provided in one of the XR devices 110a, nob. For example, the controller 200 maybe provided in a server device, such as a central server e.g., residing in a cloud computational environment. In particularly, in some embodiments, the controller 200 is part of, integrated with, or collocated with, either the first XR device 110a or the second XR device nob, or is part of, integrated with, or collocated with, a server device. A first portion of the instructions performed by the controller 200 may be executed in a first device, and a second portion of the of the instructions performed by the controller 200 may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the controller 200 maybe executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by a controller 200 residing in a cloud computational environment. Therefore, although a single processing circuitry 210 is illustrated in Fig. 7 the processing circuitry 210 maybe distributed among a plurality P107946W001

[0134] 22 of devices, or nodes. The same applies to the functional modules 2ioa:2iog of Fig. 8 and the computer program 920 of Fig. 9.

[0135] Fig. 9 shows one example of a computer program product 910 comprising computer readable storage medium 930. On this computer readable storage medium 930, a computer program 920 can be stored, which computer program 920 can cause the processing circuitry 210 and thereto operatively coupled entities and devices, such as the communications interface 220 and the storage medium 230, to execute methods according to embodiments described herein. The computer program 920 and / or computer program product 910 may thus provide means for performing any steps as herein disclosed.

[0136] In the example of Fig. 9, the computer program product 910 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 910 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 920 is here schematically shown as a track on the depicted optical disk, the computer program 920 can be stored in any way which is suitable for the computer program product 910.

[0137] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

P107946W00123CLAIMS1. A controller (200) for controlling how activation of a real-world sensory stimuli (150a, 150b) affects rendering of virtual worlds (120a, 120b), wherein the controller (200) is configured to communicate with at least a first XR device (110a), a second XR device (110b), and sensory stimuli releasing devices (140a, 140b), wherein the controller (200) comprises processing circuitry (210), and wherein the processing circuitry (210) is configured to cause the controller (200) to: obtain an indication that a real-world sensory stimuli (150a, 150b) is scheduled to be activated at a real-world location by one of the sensory stimuli releasing devices (140a, 140b), wherein the real-world location has a correspondence to a first virtual location in a first virtual world (120a) rendered at the first XR device (110a) and to a second virtual location in a second virtual world (120b) rendered at the second XR device (110b), and wherein, according to the indication, that the real-world sensory stimuli (150a, 150b) is scheduled to be activated is caused by a trigger; and in response thereto: perform an action that affects at least one of: rendering of the first virtual world (120a) at the first XR device (110a), rendering of the second virtual world (120b) at the second XR device (nob).

2. The controller (200) according to claim 1, wherein the trigger is any of: an action taken in the first virtual world (120a) by a user (160a) of the first XR device (110a), a timer expiring.

3. The controller (200) according to claim 2, wherein the action is based on a location of the second XR device (nob) relative said real-world location.

4. The controller (200) according to any preceding claim, wherein the controller (200) is configured to facilitate an operative connection between the first XR device (110a) and the second XR device (nob).P107946W001245. The controller (200) according to any preceding claim, wherein the indication that the real-world sensory stimuli (150a, 150b) is scheduled to be activated is obtained as a request from the first XR device (110a), and wherein the processing circuitry (210) is configured to cause the controller (200) to: notify the second XR device (nob) that the real-world sensory stimuli (150a, 150b) is scheduled to be activated.

6. The controller (200) according to claim 5, wherein the processing circuitry (210) is configured to cause the controller (200) to: request said one of the sensory stimuli releasing devices (140a, 140b) to activate the real-world sensory stimuli (150a, 150b).

7. The controller (200) according to claim 2 and claim 5, wherein the processing circuitry (210) is configured to cause the controller (200) to: obtain a response from the second XR device (nob) requesting blocking activation of the real-world sensory stimuli (150a, 150b), wherein the action is based on the response.

8. The controller (200) according to claim 7, wherein the processing circuitry (210) is configured to cause the controller (200) to: notify the first XR device (110a) that the activation of the real-world sensory stimuli (150a, 150b) has been blocked.

9. The controller (200) according to claim 2 and claim 8, wherein the action involves changing virtual content of the first virtual world (120a).

10. The controller (200) according to claims 6 and 7, wherein the processing circuitry (210) is configured to cause the controller (200) to: notify the second XR device (nob) that the real-world sensory stimuli (150a, 150b) has been activated.

11. The controller (200) according to claims 2, 6 and 7, or claims 2 and 10, wherein the action involves changing virtual content of the second virtual world (120b).P107946W0012512. The controller (200) according to claims 2, 6 and 7, or claims 2 and 10, wherein the virtual content is a map of the second virtual world (120b), and wherein the action involves changing the map to either guide a user of the second XR device (nob) towards said real-world location or away from said real-world location.

13. The controller (200) according to claims 2, 6 and 7, or claims 2 and 10, wherein the virtual content is a storyline of the second virtual world (120b), and wherein the action involves changing the storyline to either include or exclude the real-world sensory stimuli (150a, 150b).

14. The controller (200) according to any preceding claim, wherein at which real- world location the real-world sensory stimuli (150a, 150b) is scheduled to be activated and by which of the sensory stimuli releasing devices (140a, 140b) the real- world sensory stimuli (150a, 150b) is scheduled to be activated is based on real-world location information of the first XR device (110a) and real-world location information of the sensory stimuli releasing devices (140a, 140b).

15. The controller (200) according to any preceding claim, wherein activation of the real-world sensory stimuli (150a, 150b) involves any, or any combination of: release a scent, play out an audio clip, play out a video clip, affect temperature of a radiator, start a fan, provide a haptic signal.

16. The controller (200) according to any preceding claim, wherein the controller (200) is part of, integrated with, or collocated with, either the first XR device (110a) or the second XR device (110b), or is part of, integrated with, or collocated with, a server device.

17. A computer program (920) for controlling how activation of a real-world sensory stimuli (150a, 150b) affects rendering of virtual worlds (120a, 120b), the computer program comprising computer code which, when run on processing circuitry (210) of a controller (200), wherein the controller (200) is configured to communicate with at least a first XR device (110a), a second XR device (110b), and sensory stimuli releasing devices (140a, 140b), causes the controller (200) to:P107946W00126 obtain an indication that a real-world sensory stimuli (150a, 150b) is scheduled to be activated at a real-world location by one of the sensory stimuli releasing devices (140a, 140b), wherein the real-world location has a correspondence to a first virtual location in a first virtual world (120a) rendered at the first XR device (110a) and to a second virtual location in a second virtual world (120b) rendered at the second XR device (110b), and wherein, according to the indication, that the real -world sensory stimuli (150a, 150b) is scheduled to be activated is caused by a trigger; and in response thereto: perform an action that affects at least one of: rendering of the first virtual world (120a) at the first XR device (110a), rendering of the second virtual world (120b) at the second XR device (nob).

18. A computer program product (910) comprising a computer program (920) according to claim 17, and a computer readable storage medium (930) on which the computer program is stored.

Images