Integration of a Computing Device and Extended Reality

Abstract

Techniques are provided for integrating mobile devices with extended reality experiences. Extended reality technologies can include virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. In some examples, a synthetic (or virtual) representation of a device (e.g., a mobile device such as a mobile phone, or other type of device) may be generated and displayed along with VR content being displayed by a VR device (e.g., a head-mounted display (HMD)). In another example, content from the device (e.g., visual content being displayed by the device and / or audio content being played) may be output along with the VR content being displayed by the VR device. In another example, one or more images captured by a camera of the device and / or audio obtained by a microphone of the device may be obtained from the device by a virtual reality device and output by the virtual reality device.

Description

Technical Field

[0001]

[0001] This disclosure generally relates to techniques and systems for providing an integration of a computing device and an extended reality experience.

Background Art

[0002]

[0002] Extended reality technology can be used to present virtual content to a user to provide an extended reality experience and / or to combine a real environment from the physical world with a virtual environment. The term extended reality can encompass virtual reality, augmented reality, mixed reality, etc. Each of these forms of extended reality enables a user to experience or interact with immersive virtual environments or content. For example, an extended reality experience can enable a user to interact with a real or physical environment enhanced or augmented with virtual content. Extended reality technology can be implemented to improve the user experience in a wide range of contexts such as entertainment, healthcare, retail, education, social media, etc.

Summary of the Invention

[0003]

[0003] In some examples, systems, methods, and computer-readable media for integrating a computing device and an extended reality experience are described. For example, the techniques described herein can enable a user enjoying an extended reality experience to seamlessly use a computing device (e.g., a mobile device such as a mobile phone, a wearable device, or other types of devices) or one or more functionalities of the computing device without interrupting the extended reality experience.

[0004]

[0004] In one exemplary example, a synthetic (or virtual) representation of a user's computing device (e.g., a mobile device such as a mobile phone, a wearable device, or other types of devices) is generated and can be displayed together with virtual reality content being displayed by a virtual reality device (e.g., a head-mounted display (HMD)). In another example, content from the computing device (e.g., content being displayed by the device, messages received by the computing device, notifications generated by the computing device, and / or other content) can be displayed together with virtual reality content being displayed by the virtual reality device. In another example, audio content from the computing device (e.g., voice messages, video messages, browser content, calls, streamed or locally stored music, and / or other audio content) can be played by the virtual reality device (e.g., through one or more speakers of the virtual reality device). In some cases, the audio content can be played by the virtual reality device while visual content from the computing device is being displayed. In such examples, the user can view the computing device and / or content from the computing device together with the virtual reality content, enabling the user to interact with the computing device (e.g., control one or more applications of the mobile phone) without removing the virtual reality device (e.g., while still wearing the HMD). The virtual reality content can continue to be displayed by the virtual reality device when the synthetic representation of the computing device is being displayed, or can be paused while the synthetic representation of the computing device is being displayed by the virtual reality device.

[0005]

[0005] In another exemplary example, one or more images captured by a camera of a computing device (e.g., a mobile device such as a mobile phone, a wearable device, or other types of devices) can be obtained from the device by a virtual reality device. The virtual reality device can display one or more images together with virtual reality content being viewed through the virtual reality device (e.g., an HMD). In such an example, the user can view the real world (captured by one or more images) without removing the virtual reality device (e.g., while still wearing the HMD). In some cases, audio from a microphone of the computing device can be provided to the virtual reality device. For example, providing audio from the microphone of the computing device can enable the user to perceive what is going on in the real world without the need to remove the virtual reality device.

[0006]

[0006] According to one exemplary example, a method for determining one or more sizes of one or more objects is provided. The method includes displaying virtual reality content by a first device, obtaining a synthetic representation of a second device by the first device, displaying the synthetic representation of the second device together with the virtual reality content by the first device, receiving an input by the first device that requests a change in functionality of the second device, displaying a change in the synthetic representation of the second device based on the input received by the first device, and wherein the change in the synthetic representation of the second device represents a change in functionality of the second device.

[0007]

[0007] In another example, an apparatus for determining one or more sizes of one or more objects is provided. The apparatus includes a memory configured to store one or more images, and one or more processors implemented in circuitry and coupled to the memory. The one or more processors are configured to cause virtual reality content to be displayed on a display, obtain a synthetic representation of a device, cause the synthetic representation of the device to be displayed on the display together with the virtual reality content, receive an input requesting a change in a function of the device, cause a change in the synthetic representation of the device to be displayed on the display based on the received input, and cause the change in the synthetic representation of the device to represent a change in the function of the device, and can perform these operations.

[0008] In another example, a non-transitory computer-readable medium storing instructions is provided. The instructions, when executed by one or more processors, cause the one or more processors to cause virtual reality content to be displayed by a first device, obtain a synthetic representation of a second device by the first device, cause the synthetic representation of the second device to be displayed together with the virtual reality content, receive an input requesting a change in a function of the second device by the first device, cause a change in the synthetic representation of the second device to be displayed by the first device based on the input received by the first device, and cause the change in the synthetic representation of the second device to represent a change in the function of the second device.

[0008]

[0009] In another example, an apparatus for determining one or more sizes of one or more objects is provided. The apparatus includes means for displaying virtual reality content by a first device, means for obtaining a composite representation of a second device by the first device, means for displaying the composite representation of the second device together with the virtual reality content by the first device, means for receiving an input for requesting a change in the function of the second device by the first device, means for displaying a change in the composite representation of the second device based on the input received by the first device, and wherein the change in the composite representation of the second device represents a change in the function of the second device.

[0009]

[0010] In some aspects, the composite representation of the second device includes a composite representation of the display of the second device, wherein the content displayed by the second device is displayed in the composite representation of the display of the second device.

[0010]

[0011] In some aspects, the method, apparatus, and computer-readable medium described above further include receiving, by the first device, an indication of one or more inputs processed by the second device, displaying an additional change in the composite representation of the second device based on the indication of the one or more inputs processed by the second device, and wherein the additional change in the composite representation of the second device represents an additional change in the function of the second device.

[0011]

[0012] In some aspects, the composite representation of the second device is overlaid on the virtual reality content. In some aspects, the first device includes a virtual reality head-mounted display. In some aspects, the second device includes a mobile device.

[0012]

[0013] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise obtaining, by a first device, audio content from a second device, and playing, by the first device, the audio content from the second device.

[0013]

[0014] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise obtaining, by a first device, audio content captured by a microphone of a second device, and playing, by the first device, the audio content.

[0014]

[0015] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise obtaining, by a first device, one or more images captured by a camera of a second device, and displaying, by the first device, the one or more images together with virtual reality content. In some aspects, the one or more images are part of a video captured by the camera of the second device. In some aspects, the one or more images are displayed together with virtual reality content as part of a composite representation of the second device. In some aspects, the one or more images are displayed within a composite representation of the display of the second device.

[0015]

[0016] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise obtaining, by a first device, audio content captured by a microphone of a second device, and playing, by the first device, the audio content while displaying the one or more images together with virtual reality content.

[0016]

[0017] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise receiving a trigger and, based on receiving the trigger, displaying a composite representation of a second device along with virtual reality content. In some aspects, the trigger is based on information received from the second device.

[0017]

[0018] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise receiving a removal trigger and, based on receiving the trigger, removing, by a first device, a composite representation of a second device from a display.

[0018]

[0019] The summary of the present invention does not identify the main or essential features of the claimed subject matter, nor is it used independently to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0019]

[0020] The above will become more apparent when considered in conjunction with other features and embodiments, with reference to the following specification, claims, and accompanying drawings.

[0020]

[0021] Exemplary embodiments of the present application will be described in detail below with reference to the following figures.

Brief Description of the Drawings

[0021]

Figure 1

[0022] A diagram showing an example of virtual content displayed by a virtual reality device according to some examples.

Figure 2

[0023] A diagram showing an example of a composite representation of a device displayed along with virtual content, where the composite representation of the device and the virtual content is displayed by a virtual reality device according to some examples.

Figure 3

[0024] Diagram showing an example of the content of a device and virtual content displayed by a virtual reality device together with the virtual content, according to some examples.

Figure 4

[0025] Diagram showing an example of an image captured by a device and displayed together with virtual content, where the image and virtual content are displayed by a virtual reality device, according to some examples.

Figure 5

[0026] Diagram showing an example of switching between the display of virtual content and the display of the real world by a virtual reality device, according to some examples.

Figure 6

[0027] Diagram showing an example of the components of a virtual reality device, according to some examples.

Figure 7

[0028] Diagram showing an example of different display layers, according to some examples.

Figure 8

[0029] Diagram showing another example of the components of a virtual reality device, according to some examples.

Figure 9

[0030] Flowchart showing an example of a process for displaying virtual content, according to some examples.

Figure 10

[0031] Flowchart showing another example of a process for displaying virtual content, according to some examples.

Figure 11

[0032] Flowchart showing another example of a process for displaying virtual content, according to some examples.

Figure 12

[0033] Diagram of an exemplary computing device architecture of an exemplary computing device capable of implementing various techniques described herein. [Modes for Carrying Out the Invention]

[0022]

[0034] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a complete understanding of the embodiments of the present application. However, it will be apparent that the various embodiments may be practiced without these specific details. The figures and the description are not restrictive.

[0023]

[0035] The following description provides only exemplary embodiments and does not limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the exemplary embodiments provides those skilled in the art with an explanation that enables the implementation of the exemplary embodiments. It should be understood that various changes can be made in the functions and configurations of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0024]

[0036] As described previously, extended reality (XR) technology can provide virtual content to a user and / or combine a real or physical environment with a virtual environment (consisting of virtual content) to provide an extended reality experience to the user. The extended reality experience can include virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or other immersive content. The examples described herein use VR for illustrative purposes. However, those skilled in the art will understand that the techniques described herein can be applied to other extended reality experiences such as AR and MR. For example, in some cases, the techniques described herein can be used for AR devices and / or MR devices. An MR device can use video pass-through (e.g., shown in FIG. 5 and described below with respect to FIG. 5) to render the physical reality along with the virtual content. However, due to the poor quality of typical video pass-through devices, when a user holds a phone up to an MR device (e.g., an MR headset or head-mounted display), the user can see the outline of the user's phone, but may not be able to view / read the content due to the resolution, distortion, flickering, and / or latency associated with the video pass-through. Using the techniques described below, the content can be easily browsed by the user on an MR device display (e.g., the display of an MR headset or head-mounted display).

[0025]

[0037] VR represents a three-dimensional, computer-generated environment that can be interacted with in a seemingly real or physical way. As the user experiencing the VR environment moves in the real world, the images rendered in the virtual environment also change, giving the user the perception that they are moving within the virtual environment. For example, the user can turn left or right, look up or down, and / or move forward or backward, thus changing the user's viewpoint within the virtual environment. The VR content presented to the user can change accordingly as the user moves, so that the user's experience is as seamless as in the real world. VR content can, in some cases, include VR videos, which can be captured and rendered with extremely high quality and potentially provide a truly immersive virtual reality experience. VR applications include gaming, training, education, video (e.g., especially sports events, movies, shows), online shopping, remote operation (e.g., especially remote control of robots, unmanned aerial vehicles, remote surgery), and / or other applications.

[0026]

[0038] Various devices can be used to render XR content. For example, VR content can be rendered and displayed using VR devices such as head-mounted displays (HMDs) or other VR headsets, which completely cover the user's eyes during the VR experience. FIG. 1 is a diagram showing an example of virtual content displayed by a VR headset 102 worn by a user 100. As shown, the user is fully immersed in the VR environment (shown by VR frame 110).

[0027]

[0039] Since VR devices may completely cover a user's eyes and in some cases the user's ears, the user may not be able to interact with other devices (such as a mobile phone) and / or the real world while immersed in a virtual reality environment. For example, in some cases, the user's phone or other computing device may be used as a controller for the VR headset (or other VR device), but the user has no ability to interact with the phone's applications and / or other functions while immersed in the VR experience. Additionally, when using a VR headset, the user may feel vulnerable and may feel out of touch with what is happening in the real-world environment around the user. In some cases, the audio provided by the VR headset may prevent the user from hearing sounds from the outside world. It would be beneficial to provide an easy way for the user to visually and / or audibly check what is happening in the real world without removing the VR headset and without pausing or stopping the virtual reality experience when engaging in the VR experience. The VR headset can be removed so that the user can check the physical surroundings and / or interact with another device (in which case the VR content may need to be paused or stopped), but removing the headset every time the user wishes to view their phone (or other computing device) or check the environment can be cumbersome and may lead to an inadequate VR user experience.

[0028]

[0040] A system, method, and computer-readable medium for integrating a computing device and an extended reality experience are described herein. As described above, an example using VR as an example of an XR experience is provided herein. However, those skilled in the art will understand that the techniques described herein can be applied to other types of XR experiences.

[0029]

[0041] According to the techniques described herein, a VR device can include a plurality of different operating modes. The VR device can include a VR headset such as a head-mounted display (HMD). The VR device can enable a user to toggle between multiple modes without pausing or stopping the VR experience being displayed by the VR device. For example, various modes that a user can toggle between include a normal VR mode (shown in FIG. 1 and described above), a VR mode in which a synthetic representation of an actual computing device and / or content from the computing device is overlaid on top of the VR content (shown in FIGS. 2 and 3 and described below), a VR mode in which a peek at the outside world is overlaid on top of the VR content (shown in FIG. 4 and described below), and / or a complete switch between browsing VR content and browsing the real-world environment through the VR device (shown in FIG. 5 and described below). The computing device can include a mobile device (e.g., a mobile phone, a tablet computer, or other mobile device), a wearable device (e.g., a smartwatch, a heart rate monitor, or other wearable device), an Internet of Things (IoT) device, a remote controller (e.g., a television remote, or other remote controller), or other types of devices.

[0030]

[0042] The user can toggle between different modes using one or more inputs. The inputs for switching to one or more of the modes can include voice input detected using the audio input device (e.g., microphone) of the VR device and / or the computing device, touch input on the touch interface of the VR device and / or the computing device, gesture input detected using one or more sensors of the VR device and / or the computing device, selection of a physical button of the VR device or the computing device, selection of a physical button on a VR controller device that can control the VR device, selection of an icon or virtual button displayed by the VR device or the computing device (e.g., the user pressing an icon, the user gazing at an icon detected using gaze detection, or other types of selection), any combination thereof, and / or any other suitable input provided by the user, such as other inputs. An example of a touch input that can cause a mode change can include pressing a finger on the touch screen of the computing device for a threshold amount of time, such as for 1 second, 2 seconds, or more seconds, and / or for a certain amount of pressure (measured, for example, by an accelerometer or other component capable of measuring pressure or force). Another example of a touch input can include a stroke gesture, which can include the user drawing a circle around the touch screen in one stroke and / or any other stroke gesture. Another example of a touch input can include a strong tap gesture on the touch screen, which can include a finger tap on the touch screen with a pressure or force above a force threshold (measured, for example, by an accelerometer).

[0031]

[0043] In some examples, to trigger one or more of the modes (e.g., a VR mode in which a synthetic representation of an actual computing device and / or content from the computing device is overlaid on the VR content, a VR mode in which a peek outside is overlaid on the VR content, etc.), a movement (e.g., acceleration, velocity, etc.) of the computing device that exceeds a threshold amount can be detected and used. In some examples, the VR device can detect that the user has placed a phone in front of the VR device (e.g., using an outward-facing camera of the VR device, using a light sensor, using motion detection or other suitable techniques), in which case the VR device can switch to one or more of the modes (e.g., a VR mode in which a synthetic representation of an actual computing device and / or content from the computing device is overlaid on the VR content, a VR mode in which a peek outside is overlaid on the VR content, etc.).

[0032]

[0044] In some examples, a VR device can detect user input using a six degrees of freedom (6DOF) input interface. An XR device, such as a VR headset, can always or periodically track its location in the physical world to implement VR functionality. Such tracking is sometimes referred to as inside-out 6DOF tracking, which is inside-out because the device can track itself without any external beacons or transmitters, and 6DOF because the device can track its position with respect to three rotational vectors (pitch, yaw, roll) and three translational vectors (up / down, left / right, forward / backward). One or more sensors, such as one or more inertial measurement units (IMUs), one or more global positioning system (GPS) devices, one or more accelerometers, one or more gyroscopes, any combination thereof, and / or other sensors can be used to detect user input and / or perform tracking. In some cases, tracking can be achieved using Visual Inertial Odometry (VIO), which is a process in which visual data (from one or more camera sensors) is fused with inertial data (from gyroscopes and accelerometers) to measure the distance the device has moved in the physical world. VIO can be used simultaneously to determine (locate) the position of the device in the world and map the scene (in the world) related to the device.

[0033]

[0045] FIG. 2 is a diagram showing an example of the VR mode of the VR headset 102 that enables the VR headset 102 to display a synthetic representation 214 of an actual mobile device 204 (physical device) overlaid on a frame 210 of VR content. As shown in FIG. 2, the synthetic representation 214 of the mobile device 204 includes a synthetic representation 216 of the display (or screen) of the mobile device 204. The content displayed by the actual mobile device 204 in the real world is displayed as virtual content in the synthetic representation 216 of the display of the mobile device 204. In this VR mode, the user 100 can view the representation of the mobile device 204 display presented by the display of the VR headset 102. In some examples, audio content from the mobile device 204 can be played by the VR headset 102 while the synthetic representation 214 of the mobile device 204 is being displayed. For example, when the user selects a voice messaging application on the mobile device 204, a voice message can be played. In other examples, a video (including the visual and audio parts of the video), a video message, audio from a web browser, a call, streaming music from a network location (e.g., from an Internet-based provider), locally stored music (e.g., previously downloaded music), and / or other audio content can be played. The VR content can continue to be displayed by the VR headset 102 when the synthetic representation of the mobile device 204 is being displayed and / or when the audio content is being played, or can be paused while the synthetic representation of the mobile device 204 is being displayed and / or when the audio content is being played by the VR headset 102.

[0034]

[0046] The user can interact with the actual mobile device 204 (without removing the VR headset 102 from the user's head), view whatever is displayed on the mobile device 204 (and in some cases listen to the audio output by the mobile device 204), provided that it can be seen inside the VR headset 102. In some cases, the user can use one or more input interfaces of the mobile device 204 (such as a touch screen, push buttons, voice input, touch hover feedback, etc.) to interact with the actual mobile device 204, and changes to the functions of the mobile device 204 are displayed on the composite representation 216 of the display of the mobile device 204. In some cases, the user can interact with the mobile device 204 by providing input to the VR headset 102, such as gesture input, gaze input, voice input, VR remote controller, any combination thereof, and / or other suitable inputs that may be provided to the VR headset 102. In one exemplary example, the user 100 can provide gesture input to the VR headset 102 that includes moving the index finger downward, which, when detected by the VR headset 102, can scroll the content displayed on the composite representation 216 of the display of the mobile device 204 downward. In another exemplary example, the user 100 can provide gesture input to the VR headset 102 that includes moving the index finger in a depth direction away from the VR headset 102 (as if the user is selecting an icon), which, when detected by the VR headset 102, can cause an item of the content displayed on the composite representation 216 of the display of the mobile device 204 to be selected. In another exemplary example, the user can use the VR remote controller as a pointer to interact with the mobile device 204.

[0035]

[0047] Inputs (such as gestures, eye gaze, voice, etc.) that cause changes to the content displayed on the composite representation 216 of the display of the mobile device 204 can also change the content displayed by the actual mobile device 204. In one exemplary example, the selection of a messaging icon using a gesture detected by the VR headset 102 can cause the messaging application to be displayed on the composite representation 216 of the display of the mobile device 204 and on the display of the actual mobile device 204.

[0036]

[0048] In some examples, the user can interact with the actual mobile device 204 using touch hover feedback. For example, touch hover feedback can include the user holding a finger or thumb at a distance from the physical display of the mobile device 204 (such as 0.5 centimeters (cm), 0.75 cm, 1 cm, 1.25 cm, or other suitable distance), and a display element corresponding to the item covered by the finger or thumb (such as an icon, an item in a list, or other display element in the composite representation 216 of the display of the mobile device 204) can indicate a focused state in the composite representation 216 of the display of the mobile device 204 (for example, the display element can be highlighted, made larger, and / or otherwise emphasized). The selection of the display element can then be performed by the user by selecting the highlighted display element (for example, using gesture input, eye gaze input, touch input, voice input, and / or other input as described above). Touch hover feedback enables more effective interaction with the composite representation 214 of the mobile device 204 by allowing the user to correctly position the finger before making a selection. This can be particularly useful in the example shown in FIG. 2 where the user cannot directly view the mobile device 204 display to target a selection.

[0037]

[0049] In some implementations, an action notification 215 can be displayed on the frame 210. The action notification 215 indicates to the user that an action has occurred on the mobile device 204. In some examples, the action notification 215 can include a notification that a message has been received by the mobile device 204 (e.g., a new text message has been received, a new email has been received, an incoming phone call notification to the mobile device 204, a social media post has been notified by the mobile device 204, an emergency alert has been received by the mobile device 204, and / or other messages). In some examples, the action notification 215 can include the actual message received. In some cases, the user can select the action notification 215 (if not already displayed) to cause the synthetic representation 214 of the mobile device 204 to be displayed.

[0038]

[0050] The synthetic representation 214 of the mobile device 204 can be generated by the VR headset 102 using various techniques. In one exemplary example, the user of the VR headset 102 can provide an input indicating the type of the mobile device 204 being used (e.g., Samsung (registered trademark) Galaxy S10 (registered trademark), etc., the type, model, and version of the mobile device 204), and the VR headset 102 can generate or obtain a virtual representation having the same characteristics and dimensions as the mobile device of that type. In another example, the VR headset 102 can capture one or more images of the mobile device 204, and the VR headset 102 can use the one or more images to generate a virtual representation of the mobile device 204.

[0039]

[0051] The composite representation 214 of the mobile device 204 can be displayed at various locations in the VR coordinate system. For example, in some cases, the composite representation 214 can be displayed to appear at the actual real-world location of the phone. In one exemplary example, when the user holds the mobile phone out in front with the user's right hand, the composite representation 214 of the mobile device 204 can be displayed at the lower right corner of the frame 210. In other examples, the composite representation 214 can be displayed on the edge of the frame 210 or at another location in the frame 210 such that the composite representation 214 does not overlap with the VR content and / or the most important region within the frame 210 (e.g., the center of the frame 210 corresponding to the VR user's field of view). In some examples, the location at which the composite representation 214 of the mobile device 204 is displayed can be user-selectable. For example, the user can indicate that the composite representation 214 should be displayed at the upper left corner of the frame displayed by the VR headset 102.

[0040]

[0052] In some cases, the content being displayed and / or the audio content being output by the actual mobile device 204 can be sent or streamed to the VR headset 102, and the VR headset 102 can display the content in the composite representation 216 of the display of the mobile device 204. In some cases, the content being displayed by the mobile device 204 can be mirrored on the display of the VR headset 102 (e.g., using screen mirroring such as Miracast using the peer-to-peer WiFi Direct® standard, or AirPlay®).

[0041]

[0053] FIG. 6 is a diagram showing an example of components of an XR device, such as a VR device (e.g., of VR headset 102 or other VR device), an MR device, and / or an AR device. The components operate to enable content displayed and / or audio content output by an actual mobile device 204 (e.g., mobile device 204) to be displayed by the XR device. The XR device can include one or more applications 632. The applications can register for runtime services (e.g., rendering and / or other runtime services) provided by a runtime engine 634. The runtime engine 634 can handle rendering for each display layer or graphic layer that can be displayed by the XR device. The runtime engine 634 can also process, for each layer, a head tracking pose and input events (e.g., controller, hand gesture, voice, and / or other input events). In some cases, the runtime engine 634 can be part of a portion of one or more applications 632, part of an operating system (OS) 638, or the runtime engine 634 can be combined with a compositor engine 636 when coordinating XR activities that can include composition.

[0042]

[0054] The compositor engine 636 can weave together a plurality of graphic layers (e.g., in particular, windows, tiles) for viewing within XR (e.g., a VR session). The compositor engine 636 can also perform process XR-related processes, in particular, timewarp, spacewarp, etc. In some cases, the compositor engine 636 can be part of the runtime engine 634 or part of the OS 638, as described above. The OS 638 can be any suitable OS, such as Android™ or other suitable OS.

[0043]

[0055] Figure 7 is a diagram showing an example of different graphic layers that can be handled by the runtime engine 634. For example, layer A shown in Figure 7 corresponds to the content displayed by a mobile device (e.g., mobile device 204), and layer B corresponds to the XR content displayed by an XR device (e.g., VR headset 102). VR content is used as an example of XR content. However, those skilled in the art will understand that XR content can include MR content, AR content, and / or other content. The OS 638 (e.g., Android or other OS) can allocate the mobile device user interface (UI) to one display (e.g., the primary display), and the VR view can be allocated to another display (e.g., the secondary display). The runtime engine 634 can render the primary display allocated to the mobile device content as a layer (e.g., layer A in Figure 7), which can be viewed as a video stream on the secondary VR display rendered as another layer (e.g., layer B in Figure 7). The compositor engine 636 can weave together each layer so that the mobile device view layer (layer A) is visible in the foreground and the VR view layer (layer B) operates in the background behind layer A.

[0044]

[0056] The runtime engine 634 can handle input events (e.g., controller, hand gestures, voice, etc.) from the layered VR session. For example, an input event that selects an item on the mobile device layer (layer A) can cause the runtime engine 634 to map the input event to a touch event, which is then passed to and returned from the mobile device for processing. In another example, an input event that selects an item on the VR layer (layer B) can cause the runtime engine 634 to pass the input event to the VR device for processing.

[0045]

[0057] FIG. 8 is a diagram showing another example of components of a virtual reality device. As shown, an XR application 832 controls an XR application user interface (UI) 833 of the XR application 832. A mobile device 804 controls a mobile device UI 835 of the mobile device 804. User input provided by a user to the mobile device 804 can be provided by the mobile device 804 to the mobile device UI 835. Similarly, user input provided by a user to the XR device can be provided by the XR device to the XR application UI 833. The user can interact with the mobile device UI 835 and / or the XR application UI 833 using input devices such as, in particular, a controller, a pointer, a line of sight, a hand tracking input, a voice, etc. An XR runtime engine 834 can process the user input and handle the rendering for each graphic layer that can be displayed in an XR scene user interface 844 by one or more XR device displays 846 of the XR device. The XR scene user interface 844 includes all the content displayed by the XR device, including the content of the XR application UI 833 and / or the UI of other XR applications on the XR device.

[0046]

[0058] In some cases, a mobile device (e.g., a mobile phone) can have multiple displays, which can support UI interactions (e.g., smartphone UI interactions) and displays simultaneously with XR application interactions and displays. The content displayed by the mobile device 804 (e.g., the mobile device UI 835) on the mobile device display 84 can be entirely represented in the XR scene user interface 844 (e.g., all content displayed by the mobile device can be displayed together with the XR content), or can be represented in component form (e.g., a single application UI or a UI fragment of a single application UI can be displayed together with the XR content).

[0047]

[0059] The mobile device UI 835 can be visible on the mobile device display 842, on one or more XR device displays 846, on both the mobile device display 842 and one or more XR device displays 846, or may not be visible on any display. In some cases, the XR runtime engine 834 can reside inside or outside the XR application 832. In some cases, the XR runtime engine 834 can reside inside or outside the OS, as described above. In some cases, the XR runtime engine 834 can co-reside on the mobile device and on the XR device. In some cases, the XR runtime engine 834 can reside on a connected device, such as an integrated VR headset that incorporates the mobile device. The connection between the XR device and the connected device can be wired or wireless (e.g., using WiFi®, Bluetooth®, Zigbee®, or other wireless connections).

[0048]

[0060] Using the mode shown in FIG. 2, user 100 can view the mobile device 204 presented on the display of the VR headset 102 (and in some cases listen to audio content from the mobile device 204) while still viewing VR content, and can interact with all of the functions of the mobile device 204 as if the user were interacting directly with the mobile device 204. In some cases, the VR content can continue to play while the user 100 is interacting with the mobile device 204. In some cases, the VR headset 102 can provide an option to pause the VR content while the user 100 is interacting with the mobile device 204. In any case, the user can interact with the mobile device 204 without having to remove the VR headset 102.

[0049]

[0061] FIG. 3 is a diagram showing an example of the VR mode of the VR headset 102 that enables the VR headset 102 to display content from the mobile device 204 overlaid on the frame 310 of the VR content. The content being displayed by the actual mobile device 204 in the real world is displayed as virtual content 317 on the frame 310 of the virtual content. In the VR mode, the user 100 can see what is being displayed by the mobile device 204, but can see it without displaying the composite representation of the mobile device 204. The virtual content 317 can be displayed at various locations in the VR coordinate system. For example, in some cases, the virtual content 317 can be displayed to appear at the actual real-world location of the phone. In other examples, the virtual content 317 can be displayed on the edge of the frame 310 (as shown in FIG. 3, for example), or otherwise positioned within the frame 310 so that the virtual content 317 does not collide with the VR content and / or the most important area (e.g., the center of the frame 310 corresponding to the field of view of the VR user). In some examples, the location where the virtual content 317 is displayed can be user-selectable.

[0050]

[0062] In some examples, audio content from the mobile device 204 can be played by the VR headset 102 while the content from the mobile device 204 is being displayed. The VR content can continue to be displayed by the VR headset 102 when the content from the mobile device 204 is being displayed and / or when the audio content is being played, or can be paused while the content from the mobile device 204 is being displayed and / or when the audio content is being played by the VR headset 102.

[0051]

[0063] Similar to what has been described with respect to the mode shown in FIG. 2, the user can interact with the actual mobile device 204 (without removing the VR headset 102) and view whatever is displayed by the phone, but inside the VR headset 102. The user can interact with the actual mobile device 204 using the (one or more) input interfaces of the mobile device 204 and / or can interact with the mobile device 204 by providing input to the VR headset 102, similar to what has been described above with respect to FIG. 2.

[0052]

[0064] The composite representation 214 of the mobile device 204 can be generated by the VR headset 102 based on one or more images of the mobile device 204 captured by the VR headset 102, based on input provided by the user indicating the type of mobile device 204 being used, and / or using any other suitable technique. The content being displayed by the actual mobile device 204 can be sent or streamed to the VR headset 102, mirrored on the display of the VR headset 102 (e.g., using screen mirroring), or provided to the VR headset 102 using any other suitable technique.

[0053]

[0065] Similar to what was described with respect to FIG. 2, an action notification 315 can be displayed on the frame 310. The action notification 315 can indicate that an action has occurred on the mobile device 204, such as a message or call being received. In some examples, the action notification 315 can include the actual message received. The user can select the action notification 315 (if not already displayed) to cause the virtual content 317 from the mobile device 204 to be displayed. For example, in response to the selection of the action notification 315, a messaging application from the mobile device 204 can be displayed on the VR device 102.

[0054]

[0066] Using the mode shown in FIG. 3, the user 100 can view the content from the mobile device 204 presented on the display of the VR headset 102 while still viewing VR content, and can interact with all of the functions of the mobile device 204 as if the user were interacting directly with the mobile device 204.

[0055]

[0067] FIG. 4 is a diagram showing an example of a VR mode of the VR headset 102 that enables providing a peek outside the world by displaying one or more images (including image 418) captured by the mobile device 204 as an overlay overlaid on the frame 410 of the VR content. The one or more images can be captured by one or more cameras of the mobile device 204 and can include one or more still images or a video including a plurality of consecutive images. In some cases, the camera of the mobile device 204 used to capture the one or more images can be a rear-facing camera (on the back surface of the mobile device 204) and / or another camera of the mobile device 204 (e.g., a front-facing camera, a front-facing and a rear-facing camera, a plurality of front-facing and / or rear-facing cameras, etc.). The mobile device 204 can send the one or more images to the VR headset 102, and the VR headset 102 can then display the one or more images on the display of the VR headset 102. In some cases, audio from the microphone of the mobile device 204 can be provided to the VR headset 102, and the VR headset 102 can output the audio using one or more speakers of the VR headset 102. Providing audio from the microphone of the mobile device 204 can provide the user with more information to enable the user to perceive what is going on in the real world without having to remove the headset.

[0056]

[0068] The example shown in FIG. 4 includes an image 418 that is displayed as part of the composite representation 414 of the mobile device 204. In some implementations, one or more images captured by the mobile device 204 (including the image 418) are displayed on virtual content, in which case the composite representation 414 of the mobile device 204 is not displayed by the VR headset 102. In some implementations, one or more images can be fully displayed on the display of the VR headset 102 (e.g., in full screen mode), in which case the VR content is not displayed by the VR headset 102 while the one or more images are being displayed. In some implementations, the user 100 can control how much of the VR headset 102 display is used to display one or more images.

[0057]

[0069] Using the mode shown in FIG. 4, the user 100 can still view a view of the real-world environment from the camera of the mobile device 204 presented on the display of the VR headset 102 while browsing VR content. As described above, the user can also be provided with audio from the microphone of the mobile device 204, which can enable the user to hear any sounds occurring in the real-world environment. In some cases, the VR headset 102 can provide the option to pause the VR content while the user 100 is browsing one or more images from the mobile device 204. In any case, the user can view a view of the real-world environment (and in some cases, hear what is going on in the real-world environment) without having to remove the VR headset 102.

[0058]

[0070] FIG. 5 is a diagram showing an example of the VR mode of the VR headset 102 that enables the VR headset 102 to switch between the display of virtual VR content and the display of the real world. For example, a frame (including frame 510) of VR content can be displayed by the VR headset 102, which includes a scene of a virtual environment based on the user's pose. In response to user input, the VR headset 102 can switch to the real world display mode, and when entering the real world display mode, it can display a view (including frame 520) of the real world environment based on the user's pose.

[0059]

[0071] In some cases, an external camera that is part of the VR headset 102 can be used to capture an image of the real world environment. The captured image can be displayed by the display of the VR headset 102 when the user switches to the real world display mode. The captured image can include one or more still images, or a video including a plurality of consecutive images. The (one or more) still images, or the video, can be displayed until an input is provided to instruct the VR headset 102 for the user to switch back to the full VR mode in which the VR content is displayed without one or more still images.

[0060]

[0072] Using the mode shown in FIG. 5, the user 100 can toggle between a view of the VR world and a view of the real world environment where the user is located without having to remove the VR headset 102 to interact with the real world environment. For example, the user 100 can be provided with a full view of the room in which the user 100 is sitting so that the user 100 can interact with objects in the room while still wearing the VR headset 102.

[0061]

[0073] Next, an example of a process implemented using the techniques described herein is described. FIG. 9 is a flowchart showing an example of a process 900 for displaying virtual content. At block 902, process 900 includes displaying virtual reality content by a first device. The first device can include a VR headset (e.g., an HMD or other type of VR headset), such as the virtual reality (VR) headset 102 shown in FIGS. 1-5.

[0062]

[0074] At block 904, process 900 includes generating a synthetic representation of a second device by the first device. The second device can include a mobile device, such as the mobile device 204 shown in FIGS. 1-4. At block 906, process 900 includes displaying the synthetic representation of the second device together with the virtual content by the first device. In some examples, the synthetic representation of the second device is overlaid on the virtual content. For example, using FIG. 2 as an illustrative example, the synthetic representation 214 of the mobile device 204 is displayed together with the virtual content frame 210.

[0063]

[0075] The synthetic representation of the second device (e.g., the synthetic representation 214 shown in FIG. 2) can include a synthetic representation of the display of the second device. The content displayed by the second device is displayed in the synthetic representation of the display of the second device. For example, as shown in FIG. 2, the synthetic representation 214 of the mobile device 204 also includes a synthetic representation 216 of the display of the mobile device 204, and the content displayed by the mobile device 204 is shown in the synthetic representation 216 of the display of the mobile device 204.

[0064]

[0076] In some cases, process 900 can include displaying a change in the composite representation of a second device based on an indication of one or more inputs processed by the second device. A change in the composite representation of the second device represents a change in the functionality of the second device. For example, a user can interact with a second device (e.g., a mobile device) to cause the second device to perform one or more functions (e.g., specifically, send a text message using a messaging application, post a message to a social networking account using a social networking application, send an email). A change in the content displayed by the second device based on a change in the functionality of the second device can also be displayed as a change in the composite representation of the second device that represents the change in the functionality of the second device.

[0065]

[0077] In some cases, process 900 can include displaying a change to the composite representation of a second device based on an input received by a first device. The change to the composite representation of the second device represents a change to the functionality of the second device. For example, a user can interact with a first device (e.g., a VR headset) or the second device to cause the second device to perform one or more functions (e.g., specifically, send a text message using a messaging application, post a message to a social networking account using a social networking application, send an email). The user can interact with the second device to cause the second device to perform one or more functions using any suitable input such as those described above (e.g., by providing an input to the VR headset using one or more input interfaces of a mobile device, using touch hover feedback or other suitable input). A change to the content displayed by the second device based on a change to the functionality of the second device can also be displayed as a change to the composite representation of the second device that represents the change to the functionality of the second device.

[0066]

[0078] In some examples, process 900 includes detecting a trigger and displaying a composite representation of a second device (along with content displayed by the second device) in response to the trigger. In some cases, the trigger is based on information received from the second device. In one exemplary example, the information received from the second device includes a notification that a message has been received by the second device (e.g., a new text message or email has been received), an input received by the second device (e.g., user input provided to the second device), and / or other notifications. In some cases, the trigger is based on the second device being moved beyond a threshold amount. For example, an accelerometer or other sensor of the second device can provide information indicating the amount of movement of the second device. In some cases, the trigger is based on detection of a user's line of sight directed to the second device, detected by the first device (e.g., using 6DOF tracking). In some cases, the trigger is based on an amount of time. For example, the composite representation of the second device (along with the content displayed by the second device) can be displayed by the first device with virtual content at threshold time intervals (e.g., every 5 minutes, every 10 minutes, or other suitable time period) when no other triggers have been received. In some cases, the trigger is based on selection of an icon to be displayed with the virtual content. For example, the user can select an icon using a gesture, line of sight, voice command, and / or other input. In some examples, the trigger is based on voice input provided by the user. For example, the user can say out loud "show me my mobile device" or other phrase, which can be received as input by the first device and interpreted as a trigger to display the composite representation of the second device. In some examples, the trigger is based on the user picking up the mobile device and / or unlocking the mobile device (e.g., via fingerprint input, by pressing a hard button, via face recognition or face identification (ID), or using another mechanism for unlocking the device).

[0067]

[0079] The composite representation of the second device can be removed from the display by the first device based on various deletion triggers. For example, the user can provide a touch input (e.g., a swipe input on the user interface), a voice input, a gaze input, a gesture input, and / or other input to the first device or the second device, which can be interpreted by the first device as a deletion trigger to trigger the first device to remove the composite representation of the second device from what is being displayed. In another example, the deletion trigger can be based on the user moving the second device to a particular point or using a particular movement (e.g., by lowering the device as detected by an accelerometer or other sensor of the second device), causing the first device to remove the composite representation of the second device from the display. In another example, the composite representation of the second device can be removed based on the amount of time since the last interaction with the second device. For example, if a threshold amount of time has passed since an input was received from the second device or since the user interacted with the composite representation of the second device, the composite representation can be removed from the display. As a result of removing the composite representation of the second device from the display, the virtual content is displayed without the composite representation of the second device.

[0068]

[0080] FIG. 10 is a flowchart showing another example of a process 1000 for displaying virtual content. At block 1002, process 1000 includes displaying virtual reality content by a first device. The first device can include a VR headset (e.g., an HMD or other type of VR headset), such as the virtual reality (VR) headset 102 shown in FIGS. 1-5.

[0069]

[0081] In block 1004, process 1000 includes obtaining, by a first device, content displayed by a second device. The second device can include a mobile device, such as mobile device 204 shown in FIGS. 1 - 4. In block 1006, process 1000 includes displaying, by the first device, the content together with virtual content. In some cases, the content includes information from at least one application executed by the second device, such as an email application, a messaging application, a social networking application, a business application, a map application, and / or other suitable applications. In some examples, the content is overlaid on the virtual content. For example, using FIG. 3 as an illustrative example, the content displayed by mobile device 204 is displayed as virtual content 317 on frame 310 of the virtual content. In some examples, a composite representation of the second device is overlaid on the virtual content, and the content can be displayed together with the composite representation of the second device (as shown, for example, in FIG. 2).

[0070]

[0082] In some cases, process 1000 can include displaying, by the first device, a change to the content of the second device based on an indication of one or more inputs processed by the second device. The change to the content represents a change to the functionality of the second device. For example, a user can interact with the second device (such as a mobile device) to cause the second device to perform one or more functions (such as, in particular, sending a text message using a messaging application, posting a message to a social networking account using a social networking application, sending an email). A change to the content displayed by the second device based on a change to the functionality of the second device can also be displayed as a change to the content that represents the change to the functionality of the second device.

[0071]

[0083] In some examples, process 1000 can include displaying a change to the content of a second device based on an input received by a first device. The change to the content of the second device represents a change to the functionality of the second device. For example, a user can interact with a first device (e.g., a VR headset) to cause the second device to perform one or more functions (e.g., specifically, send a text message using a messaging application, post a message to a social networking account using a social networking application, send an email). A change to the content displayed by the second device based on the change to the functionality of the second device can also be displayed as a change to the content of the second device that represents the change to the functionality of the second device.

[0072]

[0084] In some examples, process 1000 includes detecting a trigger and displaying content along with virtual content in response to the trigger. In some cases, the trigger is based on information received from a second device. In one exemplary example, the information received from the second device includes a notification that a message has been received by the second device (e.g., a new text message or email has been received), an input received by the second device (e.g., user input provided to the second device), and / or other notifications. In some cases, the trigger is based on the second device being moved beyond a threshold amount. For example, an accelerometer or other sensor of the second device can provide information indicating the amount of movement of the second device. In some cases, the trigger is based on detection of a user's line of sight directed at the second device, detected by the first device. In some cases, the trigger is based on an amount of time. For example, content from the second device can be displayed by the first device along with virtual content at threshold time intervals (e.g., every 5 minutes, every 10 minutes, or other suitable time periods) if no other triggers have been received. In some cases, the trigger is based on selection of an icon to be displayed along with the virtual content. For example, a user can select an icon using a gesture, line of sight, voice command, and / or other input. In some examples, the trigger is based on voice input provided by the user. For example, a user can say out loud "show me my mobile device" or other phrase, which can be received as an input by the first device and interpreted as a trigger to display the content displayed by the second device. In some examples, the trigger is based on the user lifting the mobile device and / or unlocking the mobile device (e.g., via fingerprint input, by pressing a hard button, via face recognition or face identification (ID), or using another mechanism for unlocking the device).

[0073]

[0085] The content displayed by the second device can be deleted from the display by the first device based on various deletion triggers. For example, the user can provide touch input (e.g., a swipe input on the user interface), voice input, eye gaze input, gesture input, and / or other input to the first device or the second device, which can be interpreted by the first device as a deletion trigger to trigger the first device to delete the content displayed by the second device from what is displayed on the first device. In another example, the deletion trigger can be based on the user moving the second device to a particular point or using a particular movement (e.g., by lowering the device as detected by an accelerometer or other sensor of the second device). In another example, the content displayed by the second device can be deleted based on the amount of time since the last interaction with the second device. For example, if a threshold amount of time has passed since input was received from the second device or since the user interacted with the content from the second device displayed on the first device, the content displayed by the second device can be deleted from the display on the first device. As a result of deleting the content displayed by the second device from the display, the virtual content is displayed without the content displayed by the second device.

[0074]

[0086] Figure 11 is a flowchart showing another example of a process 1100 for displaying virtual content. At block 1102, process 1100 includes displaying virtual reality content by a first device. The first device can include a VR headset (e.g., an HMD or other type of VR headset), such as the virtual reality (VR) headset 102 shown in FIGS. 1-5.

[0075]

[0087] In block 1104, process 1100 includes obtaining, by a first device, one or more images captured by a camera of a second device. The second device can include a mobile device, such as mobile device 204 shown in FIGS. 1-4. In block 1106, process 1100 includes displaying, by the first device, one or more images together with virtual content. In some examples, the one or more images are overlaid on the virtual content. For example, using FIG. 4 as an illustrative example, an image 418 captured by mobile device 204 is shown together with virtual content frame 410. In some examples, the one or more images include video captured by the camera of the second device. In some examples, the one or more images include at least one still image captured by the camera of the second device.

[0076]

[0088] In some implementations, process 1100 includes using the first device to display a composite representation of the second device together with virtual content. The one or more images can be displayed together with the virtual content as part of the composite representation of the second device. For example, the one or more images can be displayed within the composite representation of the display that represents the actual display of the second device. In one example using FIG. 4 for illustrative purposes, a composite representation 414 of mobile device 204 is shown together with an image 418 that is displayed in the composite representation of the display that represents the actual display of mobile device 204.

[0077]

[0089] In some examples, process 900, 1000, and / or 1100 may be performed by a computing device or apparatus, such as a computing device having the computing device architecture 1200 shown in FIG. 12. In one exemplary example, the computing device (e.g., performing process 900, 1000, and / or 1100) can include an extended reality display device, such as a head-mounted display (HMD) or other type of XR headset.

[0078]

[0090] In some cases, a computing device or apparatus may include an input device, an output device, one or more processors, one or more microprocessors, one or more microcomputers, and / or one or more other components configured to perform the steps of processes 900, 1000, and / or 1100. The components of the computing device (e.g., one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in circuitry. For example, the components can include an electronic circuit or other electronic hardware that can include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits) for performing the various operations described herein, and / or can be implemented using them, and / or can include computer software, firmware, or any combination thereof, and / or can be implemented using them. The computing device may further include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or one or more other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0079]

[0091] Processes 900, 1000, and / or 1100 are shown as logic flow diagrams, and their operations represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents computer-executable instructions stored in one or more computer-readable storage media that, when executed by one or more processors, perform the recited operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not to be construed as limiting, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0080]

[0092] Further, processes 900, 1000, and / or 1100 may be implemented under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that runs on one or more processors, by hardware, or by a combination thereof. As described above, the code may be stored in a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0081]

[0093] FIG. 12 shows an exemplary computing device architecture 1200 of an exemplary computing device that can implement various techniques described herein. Components of the computing device architecture 1200 are shown that are in electrical communication with each other using a connection 1205 such as a bus. The exemplary computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and a computing device connection 1205 that couples various computing device components, including a computing device memory 1215 such as a read only memory (ROM) 1220 and a random access memory (RAM) 1225, to the processor 1210.

[0082]

[0094] Computing device architecture 1200 can include a cache of high-speed memory that is directly connected to, very close to, or integrated as part of processor 1210. Computing device architecture 1200 can copy data from memory 1215 and / or storage device 1230 to cache 1212 for rapid access by processor 1210. In this way, the cache can provide a performance improvement that avoids processor 1210 delays while waiting for data. These modules and other modules can control processor 1210 or be configured to control processor 1210 to perform various actions. Other computing device memory 1215 may also be available for use. Memory 1215 can include multiple different types of memory with different performance characteristics. Processor 1210 can include any general-purpose processor, as well as hardware or software services such as service 1 1232, service 2 1234, and service 3 1236 stored in storage device 1230 and configured to control processor 1210, and dedicated processors in which software instructions are incorporated into the processor design. Processor 1210 can be a self-contained system that includes multiple cores or processors, a bus, a memory controller, a cache, etc. The multi-core processor can be symmetric or asymmetric.

[0083]

[0095] To enable user interaction with the computing device architecture 1200, the input device 1245 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 1235 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 1200. The communication interface 1240 can generally control and manage user input and computing device output. There are no restrictions on operating on any particular hardware configuration, and thus the basic features here can be easily substituted with them as improved hardware configurations or firmware configurations are developed.

[0084]

[0096] The storage device 1230 is a non-volatile memory and can be a hard disk or other types of computer-readable media that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 1225, a read-only memory (ROM) 1220, and their hybrids. The storage device 1230 can include services 1232, 1234, 1236 for controlling the processor 1210. Other hardware modules or software modules are contemplated. The storage device 1230 can be connected to the computing device connection 1205. In one aspect, a hardware module that performs a specific function can include software components stored in a computer-readable medium with respect to the necessary hardware components, such as the processor 1210, the connection 1205, the output device 1235, etc., to perform that function.

[0085]

[0097] The techniques of the present disclosure are not necessarily limited to wireless applications or settings. The techniques can be applied to video coding that supports any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0086]

[0098] As used herein, the term "computer-readable medium" includes, without limitation, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying one or more instructions and / or data. A computer-readable medium can include non-transitory media on which data can be stored, which does not include carrier waves and / or transient electronic signals propagating wirelessly or via a wired connection. Examples of non-transitory media include, without limitation, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memories or memory devices. A computer-readable medium may store thereon code and / or machine-executable instructions that can represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0087]

[0099] In some embodiments, a computer-readable storage device, medium, and memory can include a cable signal or wireless signal that includes a bitstream, etc. However, when stated, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0088]

[0100] To provide a complete understanding of the embodiments and examples provided herein, specific details are given in the above description. However, one of ordinary skill in the art will understand that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks including devices, device components, steps or routines in methods implemented in software, or functional blocks comprising combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0089]

[0101] Individual embodiments may be described above as a process or method shown as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. A flowchart may describe operations as a sequential process, but many of the operations may be performed in parallel or simultaneously. Further, the order of the operations may be rearranged. When the operations are completed, the process ends, but it may have additional steps not included in the figure. The process may correspond to a method, function, procedure, subroutine, subprogram, etc. When the process corresponds to a function, its end may correspond to the return of the function to the calling function or main function.

[0090]

[0102] The processes and methods according to the examples described above can be implemented using computer-executable instructions that are stored or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. A portion of the computer resources used can be accessible via a network. The computer-executable instructions can be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store the instructions, the information used, and / or the information created during the methods according to the examples described include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0091]

[0103] Devices implementing the processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) for performing the required tasks can be stored in a computer-readable or machine-readable medium. One or more processors can perform the required tasks. General examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, stand-alone devices, etc. The functions described herein can also be embodied in peripheral devices or add-in cards. Such functions can also, as a further example, be implemented on a circuit board between different chips or different processes running on a single device.

[0092]

[0104] Commands, media for transmitting such commands, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functions described in this disclosure.

[0093]

[0105] In the above description, the aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application are described in detail herein, except as limited by the prior art, the inventive concept may, in some cases, be variously implemented and adopted, and it should be understood that the appended claims are to be construed to include such variations. The various features and aspects of the application examples described above may be used individually or together. Further, embodiments may be utilized in any number of environments and application examples other than those described herein and in the environment and application examples described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the drawings should be considered as illustrative rather than restrictive. For purposes of explanation, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0094]

[0106] Those skilled in the art will understand that the symbols or terms "less than" ("<") and "greater than" (">") used herein may be replaced, without departing from the scope of this specification, by the symbols "less than or equal to" ("≦") and "greater than or equal to" ("≧"), respectively.

[0095]

[0107] When an element is described as being "configured to" perform an operation, such a configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor, or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0096]

[0108] The phrase "coupled to" refers to any element physically connected, either directly or indirectly, to another element, and / or any element communicating, either directly or indirectly, with another element (e.g., connected to another element via a wired or wireless connection and / or other suitable communication interface).

[0097]

[0109] The claim language or other language reciting "at least one of" a set, and / or "one or more" of a set, indicates that one member of the set or (in any combination) a plurality of members of the set satisfy the claim. For example, the claim language reciting "at least one of A and B" means A, B, or A and B. In another example, the claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set, and / or "one or more" of a set, does not limit the set to the items listed in the set. For example, the claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0098]

[0110] With respect to the embodiments disclosed in this specification, the various illustrative logical blocks, modules, circuits, and algorithm steps described may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those of ordinary skill in the art may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present application.

[0099]

[0111] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, including integrated circuit devices having multiple applications including applications in general purpose computers, wireless communication device handsets, or wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. When implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code that includes instructions to perform one or more of the methods described above when executed. The computer-readable data storage medium may form part of a computer program product that may include packaging material. The computer-readable medium may comprise a memory or data storage medium such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, magnetic or optical data storage media. The techniques may also be realized at least in part by a computer-readable communication medium that conveys or transmits program code in the form of instructions or data structures as a propagated signal or radio wave, and that can be accessed, read, and / or executed by a computer.

[0100]

[0112] The program code can be executed by a processor that can include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Such a processor can be configured to implement any of the techniques described in this disclosure. The general-purpose processor can be a microprocessor, but alternatively, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein can refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementation of the techniques described herein. Further, in some aspects, the functions described herein can be provided within dedicated software modules or hardware modules configured for encoding and decoding, or can be incorporated into a composite video encoder / decoder (codec). The invention described in the claims of the present application at the time of filing is appended below. [C1] displaying virtual reality content by a first device; obtaining a composite representation of a second device by the first device; displaying the composite representation of the second device together with the virtual reality content by the first device; receiving an input for requesting a change in the function of the second device by the first device; displaying a change in the composite representation of the second device based on the input received by the first device, and the change in the composite representation of the second device representing the change in the function of the second device. A method comprising: [C2] The method according to C1, wherein the composite representation of the second device includes a composite representation of a display of the second device, and content displayed by the second device is displayed in the composite representation of the display of the second device. [C3] receiving an instruction of one or more inputs processed by the second device by the first device; displaying an additional change in the composite representation of the second device based on the instruction of the one or more inputs processed by the second device, and the additional change in the composite representation of the second device representing an additional change in the function of the second device. The method according to C1, further comprising: [C4] The method according to C1, wherein the composite representation of the second device is overlaid on the virtual reality content. [C5] The method according to C1, wherein the first device includes a virtual reality head-mounted display. [C6] The method according to C1, wherein the second device includes a mobile device. [C7] obtaining audio content from the second device by the first device; The method according to C1, further comprising playing the audio content from the second device by the first device. [C8] obtaining audio content captured by a microphone of the second device by the first device; The method according to C1, further comprising playing the audio content by the first device. [C9] acquiring, by the first device, one or more images captured by a camera of the second device; The method according to C1, further comprising displaying, by the first device, the one or more images together with the virtual reality content. [C10] The method according to C9, wherein the one or more images are part of a video captured by the camera of the second device. [C11] The method according to C9, wherein the one or more images are displayed together with the virtual reality content as part of a synthetic representation of the second device. [C12] The method according to C9, wherein the one or more images are displayed within a synthetic representation of a display of the second device. [C13] acquiring, by the first device, audio content captured by a microphone of the second device; The method according to C9, further comprising playing the audio content while displaying, by the first device, the one or more images together with the virtual reality content. [C14] receiving a trigger; The method according to C1, further comprising displaying, based on receiving the trigger, the synthetic representation of the second device together with the virtual reality content. [C15] The method according to C14, wherein the trigger is based on information received from the second device. [C16] receiving a deletion trigger; The method according to C1, further comprising deleting, by the first device, the synthetic representation of the second device from the display based on receiving the trigger. [C17] a memory configured to store content for display; an apparatus comprising one or more processors implemented in a circuit and coupled to the memory, the one or more processors being causing virtual reality content to be displayed on a display; acquiring a synthetic representation of a device; causing the synthetic representation of the device to be displayed on the display together with the virtual reality content; receiving an input requesting a change in a function of the device; An apparatus configured to cause a change in the synthetic representation of the device to be displayed on the display based on the received input, and the change in the synthetic representation of the device represents the change in the function of the device. [C18] The apparatus according to C17, wherein the synthetic representation of the device includes a synthetic representation of a display of the device, and content displayed by the device is displayed in the synthetic representation of the display of the device. [C19] The one or more processors are configured to receive an instruction for one or more inputs processed by the device, configured to cause an additional change in the synthetic representation of the device to be displayed on the display based on the instruction for the one or more inputs processed by the device, and the additional change in the synthetic representation of the device represents an additional change in the function of the device, the apparatus according to C17. [C20] The apparatus according to C17, wherein the synthetic representation of the device is overlaid on the virtual reality content. [C21] The apparatus according to C17, wherein the apparatus includes a virtual reality head-mounted display, and the virtual reality head-mounted display includes the display. [C22] The apparatus according to C17, wherein the device includes a mobile device. [C23] The one or more processors are configured to obtain audio content from the device, and configured to play the audio content from the device, the apparatus according to C17. [C24] The one or more processors are configured to obtain audio content captured by a microphone of the device, and configured to play the audio content, the apparatus according to C17. [C25] The one or more processors are configured to obtain one or more images captured by a camera of the device, and configured to display the one or more images together with the virtual reality content, the apparatus according to C17. [C26] The apparatus according to C25, wherein the one or more images are part of a video captured by the camera of the device. [C27] The apparatus according to C25, wherein the one or more images are displayed together with the virtual reality content as part of the synthetic representation of the device. [C28] The apparatus according to C25, wherein the one or more images are displayed within a composite representation of a display of the device. [C29] The one or more processors obtain audio content captured by a microphone of the device, and play the audio content while the one or more images are displayed together with the virtual reality content, the apparatus according to C25. [C30] The one or more processors receive a trigger, and display the composite representation of the device together with the virtual reality content based on receiving the trigger, the apparatus according to C17. [C31] The apparatus according to C30, wherein the trigger is based on information received from the device. [C32] The one or more processors receive a deletion trigger, and delete the composite representation of the device from display by the display based on receiving the trigger, the apparatus according to C17. [C33] A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a first device to display virtual reality content by the first device, obtain a composite representation of a second device by the first device, display the composite representation of the second device together with the virtual reality content, receive an input requesting a change in a function of the second device by the first device, and display, by the first device, a change in the composite representation of the second device based on the input received by the first device, wherein the change in the composite representation of the second device represents the change in the function of the second device.

Claims

1. displaying virtual reality content by the first device; obtaining a composite representation of the second device by the first device; displaying the composite representation of the second device together with the virtual reality content by the first device; detecting, by the first device, an input requesting a change in the function of the second device; displaying, by the first device, a change in the composite representation of the second device based on the input detected by the first device, and the change in the composite representation of the second device representing the change in the function of the second device requested in the input detected by the first device. A method comprising:

2. The method according to claim 1, wherein the composite representation of the second device is overlaid on the virtual reality content.

3. obtaining, by the first device, audio content from the second device; further comprising playing, by the first device, the audio content from the second device, or the method according to claim 1, obtaining, by the first device, audio content captured by a microphone of the second device; further comprising playing, by the first device, the audio content. The method according to claim 1.

4. obtaining, by the first device, one or more images captured by a camera of the second device; further comprising displaying, by the first device, the one or more images together with the virtual reality content. The method according to claim 1.

5. the one or more images are part of a video captured by the camera of the second device, or the one or more images are displayed together with the virtual reality content as part of the composite representation of the second device, the one or more images are displayed within a composite representation of a display of the second device. The method according to claim 4.

6. obtaining, by the first device, audio content captured by a microphone of the second device; The method according to claim 4, further comprising, by the first device, playing the audio content while displaying the one or more images together with the virtual reality content.

7. Receiving a trigger; The method according to claim 1, further comprising, based on receiving the trigger, displaying the composite representation of the second device together with the virtual reality content.

8. The method according to claim 7, wherein the trigger is based on information received from the second device.

9. Receiving a deletion trigger; The method according to claim 1, further comprising, based on receiving the trigger, deleting, by the first device, the composite representation of the second device from the display.

10. A memory configured to store content for display; An apparatus comprising one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to display virtual reality content on a display of a first device; acquire a composite representation of a second device; display the composite representation of the second device on the display of the first device together with the virtual reality content; receive an input detected by the first device requesting a change in a function of the second device; and based on the received input, display a change in the composite representation of the second device on the display, wherein the change in the composite representation of the device represents the change in the function of the second device requested in the input received by the first device.

11. The method according to claim 1, wherein the composite representation of the second device includes a composite representation of a display of the second device, wherein content displayed by the second device is displayed in the composite representation of the display of the second device.

12. The apparatus according to claim 10, wherein the composite representation of the second device includes a composite representation of a display of the second device, wherein content displayed by the second device is displayed in the composite representation of the display of the second device.

13. The one or more processors, an apparatus according to claim 10, configured to perform the method according to any one of claims 2 to 9.

14. The method according to claim 1, wherein the first device includes a virtual reality head-mounted display, and the virtual reality head-mounted display includes a display, and / or the second device includes a mobile device.

15. The apparatus according to claim 10, wherein the first device includes a virtual reality head-mounted display, and the virtual reality head-mounted display includes the display, and / or the second device includes a mobile device.

16. A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a first device to perform the method according to any one of claims 1 to 9, 11, and 14.

Images