Controlling VR / AR headphones

Abstract

A system includes a headset with virtual reality (VR) and / or augmented reality (AR) capabilities and a sensor device for imaging a real environment and determining a facing direction in which a user is facing. The headphone is configured to display a virtual scene to a user and provide corresponding virtual audio in a virtual mode. The headphone is also configured to show the real environment and provide corresponding real audio to the user in the local mode. In response to detecting a real person and depending on their position and orientation, the headphone switches from a virtual mode to a local mode.

Description

Technical Field

[0001] The present invention relates to the field of head-mounted headsets with virtual / augmented / mixed reality capabilities. In particular, the present invention relates to a user interacting with the real environment while wearing one of the head-mounted headsets. Background Art

[0002] Current head-mounted headsets can be classified into one of three types: a virtual reality (VR) only headset, where the user needs to remove the headset to view the outside world; a pass-through headset with an integrated camera that allows access to the external visual world; and a see-through headset, which is, for example, semi-transparent and always allows light from the outside to reach the user's eyes.

[0003] An illustration of a pass-through headset is shown in Figure 1 The headset 100 has: a housing 102 having two displays (not shown); and a strap 106 to hold the headset 100 on the user's head. The headset also has sensors 104, where at least one of the sensors is a camera for imaging the environment around the user.

[0004] These types of pass-through headsets have a current pass-through mode for preventing the user from colliding with objects in the room. For example, once the user gets too close to the wall of the room, the pass-through mode is automatically turned on.

[0005] In addition, both pass-through headsets and see-through headsets allow mixing of locally captured or locally visible (i.e., directly outside the headset) images with images captured at a remote location (i.e., a virtual scene).

[0006] When there are multiple people on one or both sides, there may be a need to mix local and remote data in a telecommunications setting. The challenge then is to enable a natural communication experience without a complex hardware setup at either end. In particular, there is a need to improve the interaction of a user wearing a VR / AR capable headset with a real person without a complex hardware setup or without physically removing the headset.

[0007] US2022 / 214743A1 discloses increasing the transparency of virtual content in a VR / AR headset when the user tilts or turns towards a second user. Summary of the Invention

[0008] The present invention is defined by the claims.

[0009] According to an example of one aspect of the present invention, a system is provided that includes a headset having virtual reality (VR) and / or augmented reality (AR) capabilities and a sensor device for imaging a real environment and determining a facing direction in which the user is facing, wherein the headset is configured to:

[0010] display a virtual scene to the user and provide corresponding virtual audio in a virtual mode;

[0011] show the real environment to the user and provide corresponding real audio in a local mode; and

[0012] switch from the virtual mode to the local mode in response to detecting a real person and based on the position of the real person and the facing direction in which the user is facing.

[0013] The virtual scene is a remote scene. It can be a real-world scene captured using a real camera but at a location far from the user (and thus virtual from the user's viewpoint), or it can be a non-real-world scene / avatar created in software. Thus, the term "virtual scene" should be understood. When users are wearing a VR / AR headset, their vision and hearing are often not fully aware of the real environment around them. Thus, it is proposed that when a real person is detected (in the real environment), the real environment is shown in the local mode. The real environment can be shown when the real person is detected to be within or near the user's field of view, i.e., when the user is facing the real person. This allows the user to interact with the real person despite wearing an AR / VR headset. The facing direction corresponds to the orientation of the user's head, i.e., the direction in which they are looking (relative to the normal direction of the face plane).

[0014] In a first example, the headset has augmented reality (AR) capabilities. In this case, the lens of the AR headset can be configured to display a virtual scene in the virtual mode and then become transparent in the local mode to allow the user to see the real environment (e.g., as if they were wearing ordinary glasses).

[0015] In a second example, the headset has virtual reality (VR) capabilities. In this case, the screen of the VR headset can be configured to display a virtual scene to the user in the virtual mode and display the real environment imaged by the sensor device to the user in the local mode.

[0016] The virtual scene includes, for example, another user who has their own headset but is located at a different spatial position from the user. Thus, the user can interact with other remote users of the system using virtual reality and can interact with local real people in the local mode. The interaction depends on the direction the user is facing (i.e., towards other remote users in the virtual space or towards local real people in the real space).

[0017] The headset allows, for example, 3D, 360-degree functionality, whereby the user can look around the real user, which can be at different spatial positions including elevation angles.

[0018] The headset can be configured to switch from the virtual mode to the local mode based on the angle between the first direction of detecting a real person and the facing direction of the user. For example, when the angle between the user's facing direction and the direction between the real person and the user is below a first threshold, the headset switches from the normal virtual mode to the local mode that shows the real environment.

[0019] The headset can also be configured to display a part of the virtual scene and a part of the real environment to the user in a transition mode, wherein switching from the virtual mode to the local mode includes: switching from the virtual mode to the transition mode in response to the angle between the first direction and the user's facing direction being between the first threshold angle and a larger second threshold angle, and switching from the transition mode to the local mode in response to the angle between the first direction and the user's facing direction being less than the first threshold angle.

[0020] The transition mode enables the user to see parts of both the virtual scene and the real environment during the transition from the virtual mode to the local mode. For example, this can signal to the user that a real person is nearby while still being able to see parts of the virtual scene.

[0021] At the same time, the transition mode does not require a large amount of computing resources because the images displayed by the headset during the transition mode are simply split between the virtual scene and the real environment. In other words, the virtual world and the real environment are placed side by side in the transition mode.

[0022] Of course, the boundary between the virtual scene and the real environment may be blurred.

[0023] In the transition mode, the ratio between the part of the virtual scene shown to the user and the part of the real environment shown to the user can depend on the angle between the first direction and the user's facing direction.

[0024] For example, in the transition mode, the ratio of the virtual part to the real part can change as the user turns towards the real person, such that as the user gets closer to the real person (i.e., the angle between the first direction and the user's facing direction gets smaller), more of the real environment is shown.

[0025] The headset can also be configured to identify a common reference object present in both the virtual scene and the real environment in a reference direction from the user, stitch the virtual scene and the real environment (as captured by the sensor device) around the common reference object to generate a transition image, and display the transition image in a transition mode.

[0026] This provides a more natural transition between the virtual mode and the local mode. Additionally, the presence of the common reference object in the virtual scene can provide information about the real environment to the user when in the virtual mode.

[0027] The virtual scene can include virtual objects in a second direction relative to the user, and the first threshold angle can depend on the angle between the second direction and the user's facing direction.

[0028] For example, the user can have a phone conference with a virtual object in the form of a virtual person and a real person that appear in the virtual scene. Thus, the user can interact with both the virtual person and the real person according to the user's facing direction.

[0029] For example, the first threshold angle can be equal to the angle between the second direction of the virtual object and the user's facing direction. Thus, when the user's facing direction is closer to the real person than to the virtual object, the headset switches from the virtual mode to the local mode.

[0030] The headset can be configured to switch from the virtual mode to the local mode only if the angle between the second direction and the user's facing direction is greater than a third threshold angle.

[0031] When both the real person and the virtual object (e.g., the virtual person in the phone conference) are relatively close, this provides display priority for the virtual mode. For example, the third threshold angle can be equal to or similar to the angular size of the virtual object.

[0032] The headset can also be configured to determine the color characteristics of the real environment as imaged by the sensor device, and adapt the virtual scene based on the color characteristics of the real environment.

[0033] This enables the headset to adapt the color characteristics of the virtual scene to match the light characteristics of the real environment. Thus, the switch between the virtual mode and the local mode will feel more natural to the user.

[0034] A similar method can be used with audio. The headset can also be configured to determine the audio characteristics of the real environment, and adapt the virtual audio based on the audio characteristics of the real environment.

[0035] Of course, by using a VR headset, the color characteristics of the real environment, such as those displayed on the VR headset, can be adapted based on the color characteristics of the virtual scene.

[0036] The present invention also provides a method for controlling a headset having virtual reality VR and / or augmented reality AR capabilities by using a sensor device for imaging a real environment and determining a facing direction in which the user is facing, the method comprising:

[0037] displaying a virtual scene to the user on the headset in a virtual mode and providing corresponding virtual audio to the user;

[0038] showing the real environment to the user and providing corresponding real audio in a local mode; and

[0039] switching from the virtual mode to the local mode in response to detecting a real person and based on the position of the real person and the facing direction in which the user is facing.

[0040] Switching from the virtual mode to the local mode may depend on the angle between the first direction in which the real person is detected and the facing direction of the user.

[0041] The method may further include displaying a part of the virtual scene and a part of the real environment to the user in a transition mode. Switching from the virtual mode to the local mode may include: switching from the virtual mode to the transition mode in response to the angle between the first direction and the facing direction of the user being between a first threshold angle and a larger second threshold angle, and switching from the transition mode to the local mode in response to the angle between the first direction and the facing direction of the user being less than the first threshold angle.

[0042] The ratio between the part of the virtual scene displayed to the user in the transition mode and the part of the real environment displayed to the user may depend on the angle between the first direction and the facing direction of the user.

[0043] The method may further include: identifying a common reference object that appears in both the virtual scene and the real environment in the reference direction of the user; stitching the virtual scene and the real environment (as captured by the sensor device) around the common reference object to generate a transition image; and displaying the transition image in the transition mode.

[0044] The virtual scene may include a virtual object in a second direction relative to the user. The method may include: switching from the virtual mode to the local mode only if the angle between the second direction and the facing direction of the user is greater than a third threshold angle.

[0045] The present invention also provides a computer program carrier comprising computer program code which, when run on a processing system, causes the processing system to perform all steps of the above method.

[0046] These and other aspects of the invention will be apparent and elucidated with reference to the (one or more) embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] For a better understanding of the present invention, and to more clearly show how the present invention may be implemented, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0048] Figure 1 An example of a through-headset is shown;

[0049] Figure 2 A user interacting with a virtual person and a real person is shown;

[0050] Figure 3 Shows after turning to a real person Figure 2 of the user;

[0051] Figure 4 An image shown on the headset and presented to Figure 2 and Figure 3 the user is shown;

[0052] Figure 5 A transition image having a table as a common reference object in a transition mode is shown; and

[0053] Figure 6 A user interacting with a virtual person and a real person is illustrated, wherein the virtual person and the real person overlap. DETAILED DESCRIPTION

[0054] The present invention will be described with reference to the accompanying drawings.

[0055] It should be understood that the detailed description and specific examples are intended for purposes of illustration only when indicating exemplary embodiments of devices, systems and methods and are not intended to limit the scope of the present invention. These and other features, aspects and advantages of the devices, systems and methods of the present invention will be better understood from the following description, claims and drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar components.

[0056] The present invention provides a system, the system comprising a headset having virtual reality VR and / or augmented reality AR capabilities and a sensor device for imaging a real environment and determining a facing direction the user is facing. The headset is configured to display a virtual scene to the user in a virtual mode and provide corresponding virtual audio. The headset is also configured to show the real environment to the user in a local mode and provide corresponding real audio. In response to detecting real people and based on their positions and facing directions, the headset switches from the virtual mode to the local mode.

[0057] Natural communication between a user wearing a headset and real people in the same room is not an insignificant matter. This problem is illustrated in Figure 2 and Figure 3 in.

[0058] Figure 2 Illustrated is user 202 interacting with virtual person 204 and real person 208. User 202 is wearing VR headset 100 to communicate virtually with virtual person 204. In this case, user 202 is physically located in a different place from virtual person 204 (e.g., during a virtual meeting). Thus, headset 100 allows user 202 to communicate with virtual person 204 as if virtual person 204 were located in the same room as user 202. As long as user 202 is facing a certain direction, he will see and hear virtual person 204 in the virtual scene displayed on headset 100. When using headset 100, the captured video of virtual person 204 can be presented to user 202 as a stereoscopic video, and even potentially allow head motion parallax. This last option depends on the camera capture system that captures virtual person 204.

[0059] Similarly, the head movement of user 202 can also be used to draw correct and more immersive sounds for the voice originating from virtual person 204. For example, this can be achieved by using the head orientation of the user (e.g., measured by headset 100) to select a head-related transfer function for binaural rendering.

[0060] However, due to headset 100, user 202 is completely locked inside the virtual world. Their visual and auditory senses are unaware of the real world around them. User 202 will have to remove headset 100 to return to the real world. In particular, when user 202 wants to communicate with real person 208 in the same physical room as user 202, removing headset 100 is not a particularly practical solution.

[0061] Accordingly, a mixture of content from a virtual scene and from the real environment is proposed to fill the viewport of the VR headset 100 of the user 202, where the mixture depends on the geometric configuration of the user 202 relative to the real person 208. Additionally, the mixture can depend on the geometric configuration of the user 202 relative to the virtual person 204 (or any virtual object in the virtual scene). Thus, the user 202 can more easily communicate with the real person 208 and the remote virtual person 204 while wearing the headset.

[0062] In particular, the geometric configuration is based on the orientation of the user 202 (i.e., the direction the user is facing). For example, if the user's viewing / facing direction is more towards the real person 208 than towards the virtual person 204, the headset 100 can automatically switch to the local mode on the headset 100 (showing the real environment / world and providing corresponding local audio around the user 202). In particular, the headset 100 can switch between the virtual mode (showing the virtual scene and providing corresponding virtual audio) and the local mode based on the angle between the facing direction 206 and the direction of the real person 208 relative to the user's direction 210 (i.e., the so-called first direction 210). For example, in Figure 2 the user 202 is facing the virtual person 204, and the angle is approximately 90 degrees.

[0063] Figure 3 Shows the user 202 after turning towards the real person 208. Thus, the angle 302 between the facing direction 206 and the first direction 210 of the real person 208 relative to the user 202 is less than Figure 2 the angle in Figure 2 For example, when the angle 302 is less than the angle between the facing direction 206 and the second direction 304 of the virtual person 204 relative to the user 202, the headset 100 can switch to the local mode. When the angle 302 is less than a predetermined threshold (e.g., 45 degrees), the headset 100 can also switch to the local mode. Generally, the switch to the local mode occurs at a first angle threshold.

[0064] Other conditions can also be added for the headset to switch to the local mode. For example, the condition can be that the real person 208 is facing the user 202 (or looking towards a direction close to the user).

[0065] In the local mode, audio from the real world is provided to the user 202 (e.g., from a microphone on the headset 100). At the same time, in the remote / virtual mode, audio from the virtual scene is provided to the user 202 (e.g., the voice of the virtual person 204). The local audio and the virtual audio can be provided to the user via transducers or headphones installed on the headset 100 worn by the user 202.

[0066] Figure 2 and Figure 3 has been illustrated in two dimensions (2D). However, it should be understood that the concepts discussed herein are also applicable to three-dimensional (3D) situations. In these cases, two angles can be considered. For example, a vertical angle (e.g., when the virtual person is elevated relative to the user) and a horizontal angle (e.g., angle 302) can be considered. In the 3D case, the facing direction 206, the first direction 210, and the second direction 304 are also provided in 3D.

[0067] Figure 4 illustrates images 402, 406, and 410 shown to Figure 2 and Figure 3 user 202 on the head-mounted transceiver 100. Image 402 shows a virtual scene 404, image 406 shows an image split between the virtual scene 404 and an image of the real world 408, and image 410 shows the real world 408. In other words, image 402 is an example of an image displayed on the head-mounted transceiver in virtual mode, and image 410 is an example of an image shown on the head-mounted transceiver in local mode.

[0068] A transition mode can also be provided, including a mode used by the head-mounted transceiver to smooth the transition between virtual mode and local mode, and to alert Figure 2 and Figure 3 user 202 that the real person 208 is nearby without completely removing the user 202 from the virtual scene 404. Image 406 is an example of an image displayed on the head-mounted transceiver in transition mode. In transition mode, the image 406 displayed by the head-mounted transceiver is split between a portion of the virtual scene 204 and a portion of the real world 208. This does not require a large amount of computing resources.

[0069] When the angle between the facing direction and the first direction is between a first threshold angle and a larger second threshold angle, the head-mounted transceiver can first switch from virtual mode to transition mode. For example, the head-mounted transceiver can switch to a transition mode between 90 degrees and 45 degrees, and then switch to local mode when the angle is less than 45 degrees. Of course, the specific angles for the first threshold angle and the second threshold angle can depend on the specific use case and can vary according to the direction of the virtual person relative to the user.

[0070] The virtual reality head-mounted transceiver can have a large field of view of, for example, 120 degrees. Thus, Figure 2 and Figure 3The small head rotation of user 202 away from the virtual human 204 and towards the real human 208 in [the context] should have made the real human 208 visible. To achieve this, when user 202 turns away from the virtual human 204 and towards the real human 208, local and remote visual data can be combined in a single viewport image for user 202 (i.e., image 406). As user 202 turns towards the real human 208, the viewport on the headset can be increasingly filled with data captured by, for example, a camera mounted on the headset.

[0071] In image 406, the virtual scene 404 and the real world 408 are horizontally split in the viewport (i.e., there is a vertical transition between regions at different horizontal positions), which results in a sudden transition. To give the user the impression that the virtual human 204 and the real human 208 actually exist in a single space, when using a VR headset, the hard transition can be blurred or otherwise blended. The algorithm can achieve this by creating a viewport picture that horizontally positions the images of the virtual scene and the real world using a vertical transition that depends on the user's viewing direction and low-pass filtering or blending the viewport image in a fixed-width band around the vertical transition.

[0072] The second filtering step effectively creates a gradual color change from the virtual scene to the real world (e.g., via the local camera image from the headset). Since both the remote video capture including the virtual scene and the local headset camera can produce stereoscopic video, the blending can be performed separately for the left-eye image and the right-eye image.

[0073] Depth information can also be used to perform blending in 3D space. Both the remote and local capture systems are capable of capturing or computing one or more depth maps for the virtual scene and / or the real world. Using these depth maps, the stitching between the local and remote image data can be based on the images and the depth maps. For example, if both the remote scene and the local scene contain a table where a person is located, the table can be selected as a common 3D reference plane, and the seam for the stitching can be positioned such that it at least falls on the table.

[0074] Figure 5 A transition image 500 with a table as a common reference object in the transition mode is shown. During the transition mode, the remote table 508a is gradually blended into the local table 508b. In this case, both the local and remote images are stereoscopic. Therefore, the seam / transition 506 can also be stereoscopic. Depth maps can be used to enforce stereoscopic correctness. This provides a smooth transition between the remote image 502 showing the virtual scene and the local image 504 showing the real world.

[0075] Known techniques for stereoscopic correct stitching (often referred to as 3D stitching) are typically used to produce 180-degree or 360-degree videos using multiple input cameras. In this example, the stitching covers a conference table. Such a table may be remotely visible (i.e., remote table 508a) and locally visible (i.e., local table 508b), and may have similar geometries and colors. In such a case, both the geometry and the color are blended. Then the geometry is used to redraw the left-eye and right-eye images for the stitched region.

[0076] If the remote image 502 and the local image 504 do not match in space and / or scale, a depth transition (blend) can be calculated first in the "stitched" region to transition between different scales and heights of the table. Based on the blended depth map, new stereoscopic images can be calculated for the blended region.

[0077] Figure 6 Illustrated is a user 202 interacting with a virtual person 204 and a real person 208, where the virtual person 204 and the real person 208 overlap. The virtual person 204 and the real person 208 may not be allowed to cover the same area of the viewport and should therefore be separated in the horizontal field of view of the user 202. For example, useful values for the angular separation can be 90, 60 degrees, or 45 degrees.

[0078] However, in cases such as Figure 6 as shown, priority can be given to one of the virtual person 204 or the real person 208. For example, a third threshold angle can be defined such that if the angle between the user's facing direction 206 and a second direction is less than the third threshold angle, the local mode is not used.

[0079] In Figure 6 is shown the field of view (FOV) 602 corresponding to the virtual person 204. The third threshold angle can be defined as half of the FOV 602 such that when the user 202 is not looking directly at the virtual person 204, a switch to the local mode is allowed. Of course, the third threshold angle can be greater than or less than half of the FOV 602.

[0080] The light conditions of the virtual scene and the local scene can be significantly different, making the transition between the remote image and the local image more obvious. The global color space transformation can be applied to the incoming remote video using the color statistics of the locally captured images. Similarly, the color space transformation can be applied to the locally captured images using the color statistics of the incoming remote video.

[0081] During the transition mode, the "overhearing" mode can be activated or enhanced so that once the user turns to face a real person, audio cues from the real world are mixed with an increased intensity. This means that as the user's head rotates from (roughly) facing the virtual person towards facing the real person 208, the relative intensity of the real-world audio signal increases in the combined sound signal. To achieve full control over local and remote sounds, the user preferably wears closed-back headphones, where one or more external microphones are mounted on the headset or earphone. Binaural audio can be used to map the voice of the remote user. Note that for binaural mapping of a real person, at least two microphones are required, preferably mounted externally on the headset near the user's left and right ears. When the user is basically facing the virtual person, appropriate head-related transfer functions (HRTFs) for the left and right ears corresponding to the measured angle (between the facing direction and the first direction) are selected according to the user's head orientation relative to the virtual person.

[0082] There are various ways to determine the facing direction. For example, this can be achieved by using a head tracker or a camera on the headset.

[0083] Additionally, the position of the virtual person can be considered. For example, when the virtual person moves to the left and the user does not move, the angle can be modified to create a binaural experience corresponding to the same effect in the real world.

[0084] Similar to the video experience, hard audio transitions can be blurred by applying cross fading between the audio signals from the remote and local locations. Preferably, the cross fading causes some audio leakage from the real person when the user is facing the virtual person, enabling them to attract the user's attention. Vice versa, when the user is facing the real person, some audio leakage from the virtual person is preferably present.

[0085] It is also desirable to apply gain control to either or both of the remote and local audio to provide a seamless transition of the audio signal levels.

[0086] The room acoustics at the location of the virtual person can be substantially different from the room acoustics of the user and the real person. Depending on the usage scenario, it may be desirable to incorporate the acoustics (e.g., reverberation component) of the local or remote environment in the rendering of both the virtual and the real person, or to allow seamless switching between different room acoustics.

[0087] For example, in a situation where the user is immersed in a game scene and interacting with a virtual person that is part of that game scene, it may be desirable to maintain the acoustics of the remote environment when the user turns to face the real person. In this case, the user can easily switch back to the remote environment without substantially losing the immersive experience.

[0088] In the case of a conference call with local and remote participants, it may be desirable to adopt the room acoustics of the local environment when rendering both the local and remote users.

[0089] In scenarios where both local and remote acoustics are relevant, for example, when art expression (e.g., music performance) is involved, it may be desirable to provide a seamless switch between the two environments.

[0090] In the case of multiple remote and / or local users, the switch between the remote and local modes can also be applied. For real people, the visual and audio cues can match those in the real world. However, for virtual / remote people, their positions (and relative positions) can be appropriately selected to fit a realistic or desired layout. For example, virtual people should not coincide with real people. It may also not be desirable to interleave virtual people from one location with real people, and vice versa.

[0091] In some cases, it may not be desirable to interleave people forming a group (local or remote) because the people in the group may talk to each other. Video / attenuation scaling of the audio signal can be used to create distance in order to accommodate more people. Specific HRTF distance parameters can also be used to render the audio, which is slightly more realistic than the direct attenuation of the signal.

[0092] The above examples have been described based on the presence of virtual people. However, it is possible to switch to the local mode without having to identify the direction of the virtual people relative to the user, because it is possible to switch to the local mode (or switch to the transition mode) only based on the angle between the facing direction of the user and the first direction (between the user and the real person).

[0093] Of course, it should be understood that any virtual object (including virtual people) can be used in the methods described herein.

[0094] A person skilled in the art will be able to easily develop a processor for performing any of the methods described herein. Accordingly, each step of the flowchart can represent a different action performed by the processor and can be executed by the corresponding module of the processor.

[0095] As discussed above, the system utilizes a processor to perform data processing. The processor can be implemented in many ways using software and / or hardware to perform the various required functions. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuitry for performing other functions.

[0096] Examples of circuitry that can be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0097] In various embodiments, a processor may be associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform the required functions when run on one or more processors and / or controllers. The various storage media may be fixed within the processor or controller or may be transportable, such that the one or more programs stored thereon may be loaded into the processor.

[0098] A computer program carrier may include relatively permanent storage devices (e.g., hard disk drives, solid state drives, etc.) and / or may include temporary carriers (e.g., bitstreams, etc.).

[0099] By studying the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

[0100] Functions implemented by a processor may be implemented by a single processor or by multiple individual processing units that may be considered together to constitute a "processor". In some cases, such processing units may be remote from each other and communicate with each other in a wired or wireless manner.

[0101] Although specific measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used advantageously.

[0102] A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid state medium provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0103] If the term "adapted to" is used in a claim or the specification, it should be noted that the term "adapted to" is intended to be equivalent to the term "configured to". If the term "means" is used in a claim or the specification, it should be noted that the term "means" is intended to be equivalent to the term "system", and vice versa.

[0104] Any reference signs in the claims shall not be construed as limiting the scope.

Claims

1. A system comprising a head-mounted transceiver (100) having virtual reality VR and / or augmented reality AR capabilities and a sensor device (104) for imaging a real environment and determining a facing direction in which a user (202) is facing, wherein, the head-mounted transceiver is configured to: display a virtual scene (404) to the user and provide corresponding virtual audio in a virtual mode; show the real environment (408) to the user and provide corresponding real audio in a local mode; display a part of the virtual scene and a part of the real environment to the user in a transition mode; and in response to detecting a real person (208), and based on the position of the real person and the facing direction (206) in which the user is facing, and further based on an angle (302) between a first direction in which the real person is detected and the facing direction of the user, switch from the virtual mode to the local mode, wherein, switching from the virtual mode to the local mode includes: switching from the virtual mode to the transition mode in response to the angle between the first direction and the facing direction of the user being between a first threshold angle and a larger second threshold angle; and switching from the transition mode to the local mode in response to the angle between the first direction and the facing direction of the user being less than the first threshold angle.

2. The system according to claim 1, wherein, in the transition mode, a ratio between the part of the virtual scene displayed to the user and the part of the real environment displayed to the user depends on the angle between the first direction and the facing direction of the user.

3. The system according to claim 1 or 2, wherein, the head-mounted transceiver is further configured to: identify a common reference object (508) that appears in both the virtual scene and the real environment in a reference direction starting from the user; stitch the virtual scene and the real environment captured by the sensor device around the common reference object to generate a transition image (406, 500); and display the transition image in the transition mode.

4. The system according to any one of claims 1 to 3, wherein, the virtual scene includes a virtual object (204) in a second direction (304) relative to the user, and wherein the head-mounted transceiver is configured to switch from the virtual mode to the local mode only if an angle between the second direction and the facing direction of the user is greater than a third threshold angle.

5. The system according to any one of claims 1 to 3, wherein, the virtual scene includes a virtual object (204) in a second direction (304) relative to the user, and wherein the first threshold angle depends on an angle between the second direction and the facing direction of the user.

6. The system according to claim 5, wherein, The headset is configured to switch from the virtual mode to the local mode only if the angle between the second direction and the facing direction of the user is greater than a third threshold angle.

7. The system according to any one of claims 1 to 6, wherein, the headset is further configured to: determine color characteristics of the real environment imaged by the sensor device; and adapt the virtual scene based on the color characteristics of the real environment.

8. A method for controlling a headset (100) having virtual reality VR and / or augmented reality AR capabilities by using a sensor device (104), the sensor device for imaging a real environment and determining a facing direction in which a user (202) is facing, the method comprises: displaying a virtual scene (404) on the headset to the user and providing corresponding virtual audio in a virtual mode; showing the real environment (408) to the user and providing corresponding real audio in a local mode; displaying a part of the virtual scene and a part of the real environment to the user in a transition mode; and in response to detecting a real person (208), and based on the position of the real person and the facing direction (206) in which the user is facing, and further based on the angle (302) between a first direction in which the real person is detected and the facing direction of the user, switching from the virtual mode to the local mode, wherein switching from the virtual mode to the local mode comprises: switching from the virtual mode to the transition mode in response to the angle between the first direction and the facing direction of the user being between a first threshold angle and a larger second threshold angle; and switching from the transition mode to the local mode in response to the angle between the first direction and the facing direction of the user being less than the first threshold angle.

9. The method according to claim 8, wherein, in the transition mode, the ratio between the part of the virtual scene displayed to the user and the part of the real environment displayed to the user depends on the angle between the first direction and the facing direction of the user.

10. The method according to claim 8 or 9, further comprises: identifying a common reference object (508) that appears in both the virtual scene and the real environment in a reference direction starting from the user; stitching the virtual scene and the real environment captured by the sensor device around the common reference object to generate a transition image (406, 500); and displaying the transition image in the transition mode.

11. The method according to any one of claims 8 to 10, wherein, The virtual scene includes a virtual object (204) in a second direction (304) relative to the user, and wherein the method includes: switching from the virtual mode to the local mode only if an angle between the second direction and the facing direction of the user is greater than a third threshold angle.

12. A computer program carrier comprising computer program code which, when run on a processing system, causes the processing system to perform all steps of the method according to any one of claims 8 to 11.

Images