Controlling VR / AR headsets

Abstract

A system is provided that includes a virtual reality (VR) and / or augmented reality (AR) enabled headset and a sensor device for imaging a real environment and determining a direction in which a user is facing. The headset is configured to display a virtual scene and provide a corresponding virtual sound to the user in a virtual mode. The headset is also configured to display a real environment and provide a corresponding real sound to the user in a local mode. In response to detecting a real person, depending on their location and facing direction, the headset switches from the virtual mode to the local mode.

Description

[Technical Field]

[0001] The present invention relates to the field of virtual reality / augmented reality / mixed reality enabled headsets, and in particular to a user interacting with a real environment while wearing one of said headsets. [Background technology]

[0002] Current headsets can be categorized into one of three types: dedicated virtual reality (VR) headsets, which require the user to remove the headset to see the outside world; pass-through headsets, which have a built-in camera that allows access to the outside visual world; and see-through headsets, which are, for example, semi-transparent and allow outside light to pass through to the user's eyes at all times.

[0003] An example of a pass-through headset is shown in Figure 1. Headset 100 has a housing 102 with two displays (not shown) and a strap 106 for holding headset 100 on a user's head. The headset also has sensors 104, at least one of which is a camera for capturing images of the user's surrounding environment.

[0004] This type of pass-through headset has a pass-through mode that is currently used to prevent users from colliding with objects in a room: for example, if the user gets too close to a wall in the room, the pass-through mode will automatically turn on.

[0005] Additionally, both pass-through and see-through headsets allow for the mixing of locally captured or locally visible (i.e., directly visible outside the headset) images with remotely captured images (i.e., virtual scenes). Summary of the Invention [Problem to be solved by the invention]

[0006] In telecommunication settings with multiple people on one or both sides, there may be a need to mix local and remote data. The challenge is to achieve a natural communication experience without requiring complex hardware setup on either device. In particular, there is a need to improve the interaction between users wearing VR / AR-enabled headsets and real people without the need for complex hardware setup or physically removing the headset.

[0007] US2022 / 214743A1 discloses increasing the transparency of virtual content in a VR / AR headset when a user leans or turns towards a second user. [Means for solving the problem]

[0008] The invention is defined by the claims.

[0009] According to an embodiment of one aspect of the present invention, there is provided a system comprising a virtual reality (VR) and / or augmented reality (AR) enabled headset and a sensor device for imaging a real environment and determining a facing direction in which a user is facing, the headset comprising: displaying a virtual scene and providing a corresponding virtual sound to the user in a virtual mode; presenting a real environment in a local mode and providing the user with corresponding real audio; In response to the detection of a real person, the device is configured to switch from the virtual mode to the local mode depending on the position of the real person and the facing direction in which the user is facing.

[0010] A virtual scene is a remote scene. It can be a real-world scene captured by a real camera but located far away from the user (and therefore virtual from the user's perspective), or a non-real-world scene / avatar created in software. The term "virtual scene" should be understood accordingly. When a user wears a VR / AR headset, their vision and hearing are often not fully aware of the surrounding real environment. Thus, if a real person is detected (in the real environment), it is proposed to present the real environment in local mode. The real environment can be presented when a real person is detected to be in or near the user's field of view, i.e., when the user is facing the real person. This allows the user to interact with the real person despite wearing an AR / VR headset. The facing direction corresponds to the user's head orientation, i.e., the direction the user is looking (normal to the face).

[0011] In a first example, the headset may be augmented reality (AR) capable, in which case the lenses of the AR headset may be configured to display a virtual scene in a virtual mode and become transparent in a local mode, allowing the user to see the real environment (e.g., as if wearing regular glasses).

[0012] In a second example, the headset is virtual reality (VR) capable, in which case the screen of the VR headset can be configured to display a virtual scene in a virtual mode, and to display to the user the real environment as imaged by the sensor device in a local mode.

[0013] For example, the virtual scene may contain another user with their own headset, but at a different spatial location than the user. Thus, a user can use virtual reality to interact with other users at remote locations in the system, or in local mode with local, real-world people. This interaction depends on the direction the user is facing (i.e., towards other users at remote locations in the virtual space, or towards real people in the real space).

[0014] For example, the headset allows for 3D, 360 degree functionality, allowing the user to look around the actual user, who may be in different spatial positions, including height.

[0015] The headset can be configured to switch from the virtual mode to the local mode depending on the angle between a first direction in which the real person is detected and a direction the user is facing, for example, when the angle between the direction the user is facing and the direction of the real person as seen by the user falls below a first threshold, the headset switches from a conventional virtual mode to a local mode displaying the real environment.

[0016] The headset can be further configured to display a portion of the virtual scene and a portion of the real environment to the user in a transition mode, and switching from the virtual mode to the local mode includes switching from the virtual mode to the transition mode in response to an angle between a first direction and a direction the user is facing being between a first threshold angle and a second larger threshold angle, and switching from the transition mode to the local mode in response to the angle between the first direction and the direction the user is facing being less than the first threshold angle.

[0017] Transition mode allows the user to see both the virtual scene and the real environment while transitioning from virtual mode to local mode. For example, this allows the user to see part of a virtual scene while being aware of the presence of a real person nearby.

[0018] At the same time, the transition mode does not require a large amount of computing resources because the image displayed on the headset is simply divided into a virtual scene and a real environment, meaning that the virtual world and the real environment become side by side in the transition mode.

[0019] Of course, the boundary between the virtual scene and the real environment can be blurred.

[0020] In the transition mode, the ratio between the portion of the virtual scene displayed to the user and the portion of the real environment displayed to the user may depend on the angle between the first direction and the direction the user is facing.

[0021] For example, in transition mode, the ratio of virtual to real portions can change as the user turns toward the real person, such that more of the real environment is displayed as the user moves closer to the real person (i.e., as the angle between the first direction and the direction the user is facing decreases).

[0022] The headset can further be configured to identify a common reference object that appears in both the virtual scene and the real environment in a reference direction from the user, stitch the virtual scene and the real environment around the common reference object as captured by the sensor device to generate a transition image, and display the transition image in a transition mode.

[0023] This makes the transition between virtual and local modes more natural. Furthermore, the presence of common reference objects in the virtual scene provides the user with information about the real environment while in virtual mode.

[0024] The virtual scene may include a virtual object in a second direction relative to the user, and the first threshold angle may depend on an angle between the second direction and an orientation of the user.

[0025] For example, a user can have a conference call with a virtual object, a virtual person, displayed in a virtual scene and a real person, and thus, depending on the user's orientation, the user can interact with both the virtual person and the real person.

[0026] For example, the first threshold angle may be equal to the angle between the second direction of the virtual object and the user's orientation, such that if the user's orientation is closer to a real person than to a virtual object, the headset switches from virtual mode to local mode.

[0027] The headset can be configured to switch from the virtual mode to the local mode only if the angle between the second direction and the user's orientation is greater than a third threshold angle.

[0028] This gives display priority to the virtual mode when both a real person and a virtual object (e.g., a virtual person in a conference call) are relatively close together. For example, the third threshold angle can be equal to or similar to the angular dimension of the virtual object.

[0029] The headset may further be configured to determine color characteristics of the real environment imaged by the sensor device and adapt the virtual scene based on the color characteristics of the real environment.

[0030] This allows the headset to adapt the color characteristics of the virtual scene to the lighting characteristics of the real environment, making switching between virtual and local modes more natural for the user.

[0031] A similar approach may be used for audio: the headset may be further configured to determine the acoustic properties of the real environment and adapt the virtual audio based on the acoustic properties of the real environment.

[0032] Of course, a VR headset can also adapt the color characteristics of the real environment displayed in the VR headset to match the color characteristics of the virtual scene.

[0033] The present invention also provides a method for controlling a virtual reality (VR) and / or augmented reality (AR) enabled headset using a sensor device for imaging a real environment and determining a facing direction in which a user is facing, the method comprising: In a virtual mode, displaying a virtual scene on the headset and providing virtual audio to the user; displaying a real environment in a local mode and providing corresponding real sounds to a user; In response to the detection of a real person, switching from the virtual mode to the local mode depending on the position of the real person and the facing direction in which the user is facing. The switch from virtual mode to local mode may depend on the angle between the first direction in which the real person is detected and the orientation of the user.

[0034] The method may further include displaying a portion of the virtual scene and a portion of the real environment to the user in a transition mode, wherein switching from the virtual mode to the local mode includes switching from the virtual mode to the transition mode in response to an angle between the first direction and a direction the user is facing being between a first threshold angle and a second, larger threshold angle, and switching from the transition mode to the local mode in response to the angle between the first direction and the direction the user is facing being less than the first threshold angle.

[0035] In the transition mode, the ratio between the portion of the virtual scene displayed to the user and the portion of the real environment displayed to the user may depend on the angle between the first direction and the direction the user is facing.

[0036] The method may further include identifying a common reference object that appears in both the virtual scene and the real environment in a reference direction from the user, stitching the virtual scene and the real environment, as captured by the sensor device, around the common reference object to generate a transition image, and displaying the transition image in a transition mode.

[0037] The virtual scene can include a virtual object in a second direction relative to the user, and the method can include switching from the virtual mode to the local mode only if an angle between the second direction and a facing direction of the user is greater than a third threshold angle.

[0038] The invention also provides a computer program carrier comprising computer program code which, when executed on a processing system, causes the processing system to perform all of the steps of the above-described method.

[0039] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0040] For a better understanding of the present invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1] 1 illustrates an example of a pass-through headset. [Figure 2] A diagram showing a user interacting with a virtual person and a real person. [Figure 3] 3 shows the user of FIG. 2 after turning towards a real person. [Figure 4] FIG. 4 illustrates the images displayed in the headset to the user of FIGS. 2 and 3. [Figure 5] 10A and 10B show transition images using a table as a common reference object in transition mode. [Figure 6]A diagram showing a user interacting with a virtual character and a real character in a situation where the virtual character and the real character overlap. DETAILED DESCRIPTION OF THE INVENTION

[0041] The present invention will now be described with reference to the drawings.

[0042] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0043] The present invention provides a system including a virtual reality (VR) and / or augmented reality (AR) enabled headset and a sensor device for imaging a real environment and determining a user's facing direction. The headset is configured to display a virtual scene and provide a corresponding virtual sound to the user in a virtual mode. The headset is also configured to display a real environment and provide a corresponding real sound to the user in a local mode. In response to detecting a real person, depending on their position and facing direction, the headset switches from the virtual mode to the local mode.

[0044] Natural communication between a user wearing a headset and a real person in the same room is a non-trivial problem, as illustrated in Figures 2 and 3.

[0045] FIG. 2 illustrates a user interacting with a virtual person and a real person. The user 202 wears a VR headset 100 to virtually communicate with the virtual person 204. In this case, the user 202 is physically located in a different location from the virtual person 204 (e.g., during a virtual meeting). In this manner, the headset 100 allows the user 202 to communicate with the virtual person 204 as if the virtual person 204 were in the same room as the user 202. The user 202 can see and hear the virtual person 204 in a virtual scene displayed on the headset 100 as long as they are facing a particular direction. When using the headset 100, captured video of the virtual person 204 is displayed to the user 202 as stereoscopic video, potentially allowing for head motion parallax. This last option depends on the camera capture system capturing the virtual person 204.

[0046] Similarly, head movement of the user 202 can be used to render more immersive and correct audio emanating from the virtual person 204. For example, this can be achieved by using the user's head orientation (e.g., as measured by the headset 100) to select a head-related transfer function for binaural rendering.

[0047] However, the headset 100 completely traps the user 202 in the virtual world. Their vision and hearing are not aware of the real world around them. The user 202 must remove the headset 100 to return to the real world. In particular, removing the headset 100 is not a particularly practical solution if the user 202 wants to communicate with a real person 208 who is in the same physical room as the user 202.

[0048] Therefore, it is proposed to fill the viewport of the user's 202 VR headset 100 with a mix of content from the virtual scene and content from the real environment, where this mix depends on the user's 202 geometric configuration with respect to the real person 208. Furthermore, this mix can depend on the user's 202 geometric configuration with respect to the virtual person 204 (or any virtual object in the virtual scene). As a result, the user 202 can more easily communicate with the real person 208 and the remote virtual person 204 while wearing the headset.

[0049] In particular, the geometric configuration is based on the orientation of the user 202 (i.e., the direction the user is facing). For example, the headset 100 can automatically switch to its local mode (displaying the real environment / world and providing local audio corresponding to the user's 202 surroundings) if the user's gaze / facing direction is more oriented toward the real person 208 than toward the virtual person 204. In particular, the headset 100 can switch between the virtual mode (displaying a virtual scene and providing corresponding virtual audio) and the local mode based on the angle between the facing direction 206 and the direction 210 of the real person 208 relative to the user (i.e., the so-called first direction 210). For example, in FIG. 2, the user 202 is facing the virtual person 204, and the angle is approximately 90 degrees.

[0050] Figure 3 shows the user of Figure 2 after turning towards the real person. Thus, the angle 302 between the facing direction 206 and the first direction 210 of the real person 208 relative to the user 202 is smaller than in Figure 2. For example, the headset 100 can switch to the local mode when the angle 302 is smaller than the angle between the facing direction 206 and the second direction 304 of the virtual person 204 relative to the user 202. The headset 100 can also switch to the local mode when the angle 302 is smaller than a predetermined threshold (e.g., 45 degrees). Generally, the switch to the local mode occurs at a first angle threshold.

[0051] Other conditions can also be added for the headset to switch to local mode, such as the real person 208 facing the user 202 (or looking in a direction closer to the user).

[0052] In local mode, audio from the real world is provided to the user 202 (e.g., from the microphone of the headset 100), while in remote / virtual mode, audio from a virtual scene is provided to the user 202 (e.g., speech from a virtual person 204). Local and virtual audio can be provided to the user through headphones or transducers attached to the headset 100 worn by the user 202.

[0053] 2 and 3 are shown in two dimensions (2D). However, it will be understood that the concepts discussed herein also apply to three-dimensional (3D) cases. In such cases, two angles are considered. For example, a vertical angle (e.g., if a real person is positioned higher relative to the user) can be considered as well as a horizontal angle (e.g., angle 302). In the 3D case, both the facing direction 206, the first direction 210, and the second direction 304 are provided in 3D.

[0054] Figure 4 shows images displayed in the headset to the user of Figures 2 and 3. Image 402 shows a virtual scene 404, image 406 shows an image split between the virtual scene 404 and the real world 408, and image 410 shows the real world 408. That is, image 402 is an example of an image displayed in the headset in virtual mode, and image 410 is an example of an image displayed in the headset in local mode.

[0055] A transition mode can also be provided, consisting of a mode used by the headset to smoothly transition between virtual and local modes and alert the user 202 of Figures 2 and 3 that a real person 208 is nearby without completely removing the user 202 from the virtual scene 404. Image 406 is an example of an image displayed on the headset in transition mode. In transition mode, the image 406 displayed by the headset is split between a portion of the virtual scene 204 and a portion of the real world 208. This does not require significant computational resources.

[0056] The headset may initially switch from the virtual mode to the transition mode when the angle between the facing direction and the first direction is between a first threshold angle and a second, larger threshold angle. For example, the headset may switch to the transition mode between 90 degrees and 45 degrees, and switch to the local mode when the angle is less than 45 degrees. Of course, the specific angles of the first and second threshold angles may depend on the particular use case and may vary depending on the orientation of the virtual person relative to the user.

[0057] A virtual reality headset can have a large field of view, e.g., 120 degrees. Therefore, if user 202 in FIGS. 2 and 3 simply turns his or her head slightly from virtual person 204 toward real person 208, he or she should already be able to see real person 208. To achieve this, as user 202 turns from virtual person 204 toward real person 208, local and remote visual data can be combined into a single viewport image (i.e., image 406) for user 202. The viewport on the headset is increasingly filled with data captured, for example, by a camera mounted on the headset, as user 202 turns toward real person 208.

[0058] In image 406, the virtual scene 404 and real world 408 are split horizontally within the viewport (i.e., there are vertical transitions between regions at different horizontal locations), resulting in an abrupt transition. To give the user the impression that the virtual person 204 and real person 208 actually exist in one space, the abrupt transition can be blurred or otherwise blended when using a VR headset. This can be achieved by creating a viewport image by horizontally arranging images of the virtual scene and real world with a vertical transition that depends on the user's gaze direction, and by an algorithm that low-pass filters or blends the viewport image in a constant-width band around the vertical transition.

[0059] The second filtering step effectively creates a gradual color transition from the virtual scene to the real world (e.g., via the local camera image from the headset). Because both the remote video capture and the local headset camera that make up the virtual scene are capable of producing stereo video, blending can be done separately for the left and right eyes.

[0060] Blending can also be performed in 3D space using depth information. Both the remote and local capture systems can capture or calculate one or more depth maps of the virtual scene and / or the real world. These depth maps can be used to stitch the local and remote image data together based on the images and depth maps. For example, if both the remote and local scenes have a table with a person sitting at it, the table can be selected as a common 3D reference plane, and the seam can be positioned to be at least above the table.

[0061] Figure 5 shows a transition image using a table as a common reference object in transition mode. During transition mode, a remote table 508a gradually blends into a local table 508b. In this case, both the local and remote images appear stereoscopic. Therefore, the seam / transition 506 may also appear stereoscopic. Stereoscopic correctness can be enforced using a depth map. This results in a smooth transition between a remote image 502 showing the virtual scene and a local image 504 showing the real world.

[0062] A known technique for stereoscopically correct stitching, called 3D stitching, is typically used to create 180-degree or 360-degree video using multiple input cameras. In this example, the seam covers a conference table. Such a table is likely to be visible both remotely (i.e., remote table 508a) and locally (i.e., local table 508b), and is likely to have a similar shape and color. In this case, both the shape and color are blended together. This shape is then used to re-render the left- and right-eye images in the seam area. If the remote image 502 and the local image 504 do not match spatially and / or scale, a depth transition (blending) can first be calculated at the "seam" region to transition between different scales and heights of the table. Based on the blended depth map, a new stereo image can be calculated for the blend region.

[0063] 6 illustrates a user interacting with a virtual and real person in an overlapping situation. The virtual person 204 and the real person 208 should be separated in the horizontal field of view of the user 202, as they are not allowed to cover the same area of ​​the viewport. For example, valid values ​​for the angular separation are 90 degrees, 60 degrees, or 45 degrees.

[0064] 6, however, priority may be given to one of the virtual person 204 and the real person 208. For example, a third threshold angle may be defined such that the local mode is not used when the angle between the user's facing direction 206 and the second direction is smaller than the third threshold angle.

[0065] 6, a field of view (FOV) 602 corresponding to the virtual person 204 is shown. A third threshold angle can be defined as half of the FOV 602 so that switching to local mode is permitted when the user 202 is not looking directly at the virtual person 204. Of course, the third threshold angle can be greater or less than half of the FOV 602.

[0066] The lighting conditions of the virtual scene and the local scene may be significantly different, making the transition between the remote and local images more noticeable. A global color space transformation can be applied to the incoming remote video using the color statistics of the locally captured image. Similarly, a color space transformation can be applied to the locally captured image using the color statistics of the incoming remote video.

[0067] During the transition mode, a "hear-through" mode is enabled or enhanced, and audio cues from the real world are mixed in with increasing intensity as soon as the user turns toward the real person. This means that as the user's head rotates from (approximately) facing the virtual person to facing the real person 208, the relative strength of the real-world audio signal increases in the synthesized audio signal. To enable full control of the local and remote audio, the user preferably wears closed-back headphones with one or more external microphones attached to the headset or headphones. Binaural audio can be used to render the remote user's voice. For binaural rendering of a real person, at least two microphones are required, preferably attached outside the headset and near the user's left and right ears. When the user is substantially facing the virtual person, appropriate head-related transfer functions (HRTFs) are selected for the left and right ears corresponding to the measured angle (between the facing direction and the first direction) depending on the user's head orientation relative to the virtual person.

[0068] There are various ways to determine facing direction, for example this can be achieved using a head tracker or using the camera in the headset.

[0069] Additionally, the position of the virtual person can also be taken into account: for example, if the virtual person moves to the left and the user does not move, the angle will be changed, resulting in a binaural experience that corresponds to the same effect as in the real world.

[0070] Similar to a video experience, crossfading can be applied between the remote and local audio signals to soften harsh audio transitions. Preferably, the crossfading ensures that when the user is facing a virtual character, there is some audio bleed-through from the real character, which can draw the user's attention. Conversely, when the user is facing a real character, there is preferably audio bleed-through from the virtual character.

[0071] It may also be desirable to apply gain control to either the remote audio, the local audio, or both to achieve seamless transitions in audio signal levels.

[0072] The room acoustics in a virtual person's location can be significantly different from those of the user or real person. Depending on the use case, it may be desirable to employ local or remote acoustics (e.g., reverberant components) in the rendering of both the virtual and real person, or to enable seamless switching between different room acoustics.

[0073] For example, if a user is immersed in a game scenario and interacts with a virtual character that is part of the game scenario, it may be desirable to maintain the audio of the remote environment even when the user turns to face the real character, allowing the user to easily return to the remote environment without significant loss of immersion.

[0074] In the case of a conference call with local and remote participants, it may be desirable to employ the room acoustics of the local environment when rendering both the local and remote users.

[0075] In scenarios involving both local and remote audio, for example when artistic expression (such as a musical performance) is involved, it may be desirable to provide seamless switching between the two environments.

[0076] Switching between remote and local modes can also be applied when there are multiple remote and local users. For real people, the visual and auditory cues will match those in the real world. However, for virtual / remote people, their positions (and relative positions) can be chosen appropriately to fit a realistic or desired layout. For example, virtual and real people should not overlap, and it would be undesirable to interleave virtual and real people in a location.

[0077] In some cases, interleaving people in a (local or remote) gathering may be undesirable due to the possibility that people in the gathering may talk to each other. Scaling the video / attenuating the audio signal can be used to create distance to accommodate more people. Audio can also be rendered with specific HRTF distance parameters, which is somewhat more realistic than simply attenuating the signal.

[0078] The above examples have been described based on the presence of a virtual person. However, since switching to the local mode (or switching to the transition mode) can be performed depending only on the angle between the direction the user is facing and the first direction (the direction between the user and the real person), it is possible to switch to the local mode without specifying the direction of the virtual person relative to the user.

[0079] Of course, it will be appreciated that any virtual object (including a virtual person) may be used in the methods described herein.

[0080] Those skilled in the art can easily develop a processor to perform any of the methods described herein. Accordingly, each step in the flowchart may represent a respective operation performed by a processor, and may be performed by a respective module of the processor.

[0081] As described above, the system utilizes a processor to process data. The processor may be implemented in various ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors that are programmed with software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0082] Examples of circuitry that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0083] In various embodiments, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, EEPROM, etc. The storage media may be encoded with one or more programs that, when executed by the one or more processors and / or controllers, perform the necessary functions. The various storage media may be mounted within the processor or controller, or may be transportable such that one or more programs stored on the storage media are readable by the processor.

[0084] The computer program carrier may comprise a relatively permanent storage device (such as a hard drive, solid state drive, etc.) and / or may comprise a transitory carrier (such as a bitstream).

[0085] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0086] The functions implemented by a processor may be implemented by a single processor or by multiple separate processing units, which may be considered to constitute a “processor.” Such processing units may be remote from each other and may communicate with each other via wired or wireless means.

[0087] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0088] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0089] When the term "adapted for" is used in the claims or description, it is meant to be equivalent to the term "configured to." When the term "apparatus" is used in the claims or description, it is intended to be equivalent to the term "system," and vice versa.

[0090] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A system comprising a virtual reality (VR) and / or augmented reality (AR) enabled headset and a sensor device for imaging a real environment and determining a facing direction of a user, the system comprising: the headset, in a virtual mode, displays a virtual scene and provides a corresponding virtual sound to the user; presenting the real-world environment and providing corresponding real-world audio to the user in a local mode; displaying a portion of the virtual scene and a portion of the real environment to the user in a transition mode; configured to switch from the virtual mode to the local mode in response to a real person being detected, depending on a position of the real person and a facing direction in which the user is facing, and further depending on an angle between a first direction in which the real person is detected and the facing direction of the user; Switching from the virtual mode to the local mode switching from the virtual mode to the transition mode in response to the angle between the first direction and the facing direction of the user being between a first threshold angle and a second threshold angle greater than the first threshold angle; switching from the transition mode to the local mode in response to the angle between the first direction and the facing direction of the user being smaller than the first threshold angle; Including, the system.

2. 2. The system of claim 1, wherein in the transition mode, the ratio of the portion of the virtual scene displayed to the user to the portion of the real environment displayed to the user depends on the angle between the first angle and the facing direction of the user.

3. The headset, identifying a common reference object that appears in both the virtual scene and the real environment at a reference angle from the user; stitching the virtual scene and the real environment, as captured by the sensor device, around the common reference image to generate a transition image; The system of claim 1 or 2, configured to display the transition image in the transition mode.

4. 4. The system of claim 1, wherein the virtual scene includes a virtual object in a second direction relative to the user, and the headset is configured to switch from the virtual mode to the local mode only if an angle between the second direction and the facing direction of the user is greater than a third threshold angle.

5. 4. The system of claim 1, wherein the virtual scene includes a virtual object in a second direction relative to the user, and the first threshold angle depends on an angle between the second direction and the facing direction of the user.

6. 6. The system of claim 5, wherein the headset is configured to switch from the virtual mode to the local mode only if the angle between the second direction and the facing direction of the user is greater than a third threshold angle.

7. 7. The system of claim 1, wherein the headset is configured to determine color characteristics of the real environment imaged by the sensor device and to adapt the virtual scene based on the color characteristics of the real environment.

8. 1. A method for controlling a virtual reality (VR) and / or augmented reality (AR) enabled headset by using a sensor device for imaging a real environment and determining a facing direction in which a user is facing, comprising: displaying a virtual scene and providing a corresponding virtual sound to the user on the headset in a virtual mode; presenting the real-world environment and providing corresponding real-world audio to the user in a local mode; displaying a portion of the virtual scene and a portion of the real environment to the user in a transition mode; in response to a real person being detected, switching from the virtual mode to the local mode depending on a position of the real person and a facing direction in which the user is facing, and further depending on an angle between a first direction in which the real person is detected and the facing direction of the user; switching from the virtual mode to the local mode, switching from the virtual mode to the transition mode in response to the angle between the first direction and the facing direction of the user being between a first threshold angle and a second threshold angle greater than the first threshold angle; switching from the transition mode to the local mode in response to the angle between the first direction and the facing direction of the user being smaller than the first threshold angle; A method comprising:

9. 9. The method of claim 8, wherein in the transition mode, the ratio of the portion of the virtual scene displayed to the user to the portion of the real environment displayed to the user depends on the angle between the first angle and the facing direction of the user.

10. identifying a common reference object that appears in both the virtual scene and the real environment at a reference angle from the user; stitching the virtual scene and the real environment, as captured by the sensor device, around the common reference image to generate a transition image; displaying the transition image in the transition mode; 10. The method of claim 8 or 9, comprising:

11. 11. The method of claim 8, wherein the virtual scene includes a virtual object in a second direction relative to the user, and wherein the method switches from the virtual mode to the local mode only if an angle between the second direction and the facing direction of the user is greater than a third threshold angle.

12. A computer program product which, when executed by a processing system, causes said processing system to perform the method of any one of claims 8 to 11.

Images