Dynamic mixed reality content in virtual reality

Abstract

In one embodiment, a method includes capturing, using one or more cameras of a mobile computing device, one or more images of a first user wearing a VR display device in a real-world environment. The mobile computing device sends a pose of the mobile computing device relative to the VR display device to a VR system. The mobile computing device receives a VR rendering of a VR environment from the VR system. The VR rendering is in accordance with a perspective of the mobile computing device relative to the VR display device. The method includes segmenting the first user from the one or more images and generating, in real-time, an MR rendering of the first user in the VR environment in response to capturing the one or more images. The MR rendering of the first user is based on a composition of the segmented first user from the one or more images and the VR rendering.

Description

Technical Field

[0001] This disclosure generally relates to file management within database and network environments, and more particularly to mixed reality (MR) rendering. Background Technology

[0002] Existing methods of sharing virtual reality (VR) experiences merely allow other users to view a two-dimensional (2D) rendering of the VR experience from the first user's point of view (POV). This method of transferring the VR experience of the first user (wearing a VR headset) to a second user's 2D display may be limited to allowing the second user to view what the first user is seeing through their headset. In other words, the second user may be restricted to watching a live stream of the VR environment within the first user's field of view (i.e., a 2D rendering of the VR experience based on the first user's viewpoint). Some experiences may also allow the second user to change their POV and explore the VR experience independently.

[0003] There are also methods for converting VR experiences into mixed reality (MR) experiences, in which real-world images of a first user are captured via a camera and inserted into the rendering of the VR experience. For example, the camera can be set to view the first user from a static position. The system can then extract images of the user from the camera and insert these images into the rendering of the VR experience based on a third-person perspective (i.e., the perspective from the camera's position). In this way, the system can provide a second user with a static view of the first user in the VR environment (e.g., an "over-the-shoulder" view from above and behind the first user). However, the second user may not be able to move independently of the first user to view the first user from other POVs, or to see the parts of the VR environment the first user is immersed in. Furthermore, creating an MR rendering of the first user in the VR environment may require the second user to have specialized high-end equipment, a green screen, a production camera, and a personal computer (PC) running specialized software. Summary of the Invention

[0004] In certain embodiments, users of mobile computing devices (e.g., smartphones) may be able to view mixed reality (MR) renderings of a first user immersed in a virtual reality (VR) environment. However, users of mobile computing devices may be limited in the amount of MR rendering content they can view about the first user and how they can interact with the VR environment. Therefore, a technical challenge may include providing users of mobile computing devices with more than just a static viewing experience when the first user is immersed in the VR environment. One solution proposed by the embodiments disclosed herein to address the technical challenge of providing interactive experiences for users of mobile computing devices is, for example, allowing users of mobile computing devices to interact with MR and VR content independently of what the user immersed in the VR environment is doing, and allowing users of mobile computing devices to dynamically interact with the MR renderings and the VR environment. By way of example and not limitation, the mobile computing device may render MR images of the first user in the VR environment and allow the user of the mobile computing device to move the mobile computing device to other poses (positions, orientations), and the VR system may dynamically update the MR renderings in real time without additional setup and repositioning processes. Although this disclosure describes a method for generating dynamic MR content in a VR environment in a specific manner, this disclosure contemplates generating dynamic MR content in a VR environment in any suitable manner.

[0005] In a particular embodiment, one or more computer systems may use one or more cameras of a mobile computing device to capture one or more images of a first user wearing a VR display device in a real-world environment. The one or more computer systems may send the pose of the mobile computing device to a VR system of the VR display device. The one or more computer systems may receive VR rendering of the VR environment from the VR system at the mobile computing device. The VR rendering may be based on the viewpoint of the mobile computing device relative to the VR display device. The one or more computer systems may segment the first user from the one or more images. In response to capturing one or more images, the one or more computer systems may generate a real-time MR rendering of the first user in the VR environment at the mobile computing device. The MR rendering of the first user may be based on a composite of the segmented one or more images of the first user and the VR rendering.

[0006] According to a first aspect of this disclosure, a method is provided, comprising: using a mobile computing device to: acquire one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment using one or more cameras of the mobile computing device; send a pose of the mobile computing device relative to the VR display device to a VR system of the VR display device; receive a VR rendering of a VR environment from the VR system at the mobile computing device, wherein the VR rendering is based on the viewpoint of the mobile computing device relative to the VR display device; segment the first user from the one or more images; and in response to acquiring the one or more images, generate a mixed reality (MR) rendering of the first user in the VR environment in real time at the mobile computing device, wherein the MR rendering of the first user is based on a composite of the segmented one or more images of the first user and the VR rendering.

[0007] In some embodiments, the method may further include receiving at a mobile computing device one or more anchor points in a real-world environment identified from one or more images from a VR system.

[0008] In some embodiments, the method may further include: determining the pose of the mobile computing device relative to the VR display device based on one or more of the one or more anchor points.

[0009] In some embodiments, the method may further include: detecting that the mobile computing device has moved; determining an updated pose of the mobile computing device relative to a VR display device based on one or more of the one or more anchor points; and, in response to detecting that the mobile computing device has moved, generating an updated MR rendering of the first user in the VR environment at the mobile computing device in real time, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device relative to the VR display device.

[0010] In some embodiments, the pose of the mobile computing device relative to the VR display device can be determined based on triangulation between: one or more of the one or more anchor points; one or more cameras of the mobile computing device; and the VR display device.

[0011] In some embodiments, the one or more anchor points may be fixed objects in a real-world environment.

[0012] In some embodiments, the method may further include switching between: VR rendering of a VR environment based on the perspective of a first user, and MR rendering of the first user in a VR environment based on the perspective of the mobile computing device relative to a VR display device.

[0013] In some embodiments, the method may further include: sending, in real time, a generated MR rendering of a first user in a VR environment from a mobile computing device to an online social network for live display to one or more additional users.

[0014] In some embodiments, segmenting a first user from one or more images may include: using a segmentation algorithm to identify the body of the first user within each of the one or more images.

[0015] In some embodiments, the method may further include: receiving at a mobile computing device an indication from a VR system that a first user has activated a specific VR application on the VR system; and in response to the received indication, presenting at the mobile computing device a notification that the first user wearing a VR display device has launched the specific VR application.

[0016] In some embodiments, the mobile computing device can be used to interact with a VR environment through MR rendering.

[0017] In some embodiments, interacting with a VR environment may include one or more of the following: modifying a VR object in the VR environment, creating a VR object in the VR environment, modifying an MR object in the VR environment, creating an MR object in the VR environment, modifying one or more properties of the VR environment, changing the VR environment from a first VR environment to a second VR environment, changing a VR application, recording MR rendering, or streaming MR rendering.

[0018] In some embodiments, interacting with the VR environment may include one or more of the following: creating a VR avatar corresponding to a second user of a mobile computing device; or interacting with a VR avatar of a third user.

[0019] According to another aspect of this disclosure, one or more computer-readable non-transitory storage media are provided, comprising software that, when executed by a mobile computing device, is operable to: acquire one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment using one or more cameras of the mobile computing device; send a pose of the mobile computing device relative to the VR display device to a VR system of the VR display device; receive a VR rendering of a VR environment from the VR system at the mobile computing device, wherein the VR rendering is from a viewpoint of the mobile computing device relative to the VR display device; segment the first user from the one or more images; and, in response to acquiring the one or more images, generate a mixed reality (MR) rendering of the first user in the VR environment in real time at the mobile computing device, wherein the MR rendering of the first user is based on a composite of the segmented one or more images of the first user and the VR rendering.

[0020] In some embodiments, the software, when executed, is also operable to: receive at a mobile computing device one or more anchor points in a real-world environment identified from one or more images by a VR system.

[0021] In some embodiments, the software, when executed, is also capable of determining the pose of the mobile computing device relative to the VR display device based on one or more of the one or more anchor points.

[0022] In some embodiments, when executed, the software is also operable to: detect that the mobile computing device has moved; determine an updated pose of the mobile computing device relative to the VR display device based on one or more of the one or more anchor points; and, in response to detecting that the mobile computing device has moved, generate an updated MR rendering of the first user in the VR environment at the mobile computing device in real time, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device relative to the VR display device.

[0023] In some embodiments, the pose of the mobile computing device relative to the VR display device can be determined based on triangulation between: one or more of the one or more anchor points; one or more cameras of the mobile computing device; and the VR display device.

[0024] In some embodiments, the one or more anchor points may be fixed objects in a real-world environment.

[0025] According to another aspect of this disclosure, a mobile computing device is provided, comprising: one or more processors; and non-transitory memory coupled to the processors, the non-transitory memory including instructions executable by the processors, the processors being operable, when executing the instructions, to: acquire one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment using one or more cameras of the mobile computing device; send a pose of the mobile computing device relative to the VR display device to a VR system of the VR display device; receive a VR rendering of a VR environment from the VR system at the mobile computing device, wherein the VR rendering is based on the viewpoint of the mobile computing device relative to the VR display device; segment the first user from the one or more images; and, in response to acquiring the one or more images, generate a mixed reality (MR) rendering of the first user in the VR environment in real time at the mobile computing device, wherein the MR rendering of the first user is based on a composite of the segmented one or more images of the first user and the VR rendering.

[0026] Generating dynamic MR content in a VR environment presents several technical challenges. One challenge includes allowing dynamic movement of a mobile computing device while maintaining the image of a user immersed in the VR environment within the frame of the image on the mobile computing device. A solution to this challenge proposed in the embodiments disclosed herein could be segmenting the user's outline or body and using a segmentation algorithm to overlay or superimpose the MR rendering of the user immersed in the VR experience onto the VR environment. Another challenge includes interacting with the generated MR and VR content without requiring specialized equipment. A solution to this challenge proposed in the embodiments disclosed herein could be leveraging the rendering and spatial mapping capabilities of a VR system to allow users of mobile computing devices to interact with the generated MR and VR content through the mobile computing device.

[0027] Certain embodiments disclosed herein may provide one or more technical advantages. These advantages may include rendering MR images of a first user in a VR environment without the use of a green screen or other specialized equipment. Another advantage may include allowing a user of a mobile computing device to move the device to other poses (positions, orientations), and after an initial positioning process (e.g., an automatic calibration process), the VR system dynamically updates the MR rendering in real time without requiring additional setup and repositioning of the mobile computing device. These other advantages will be apparent to those skilled in the art when considering the accompanying drawings, specification, and claims of this disclosure.

[0028] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited to these embodiments. Specific embodiments may include all or some of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein, or may exclude components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments of the invention are specifically disclosed in the appended claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim class (e.g., method) may also be claimed in another claim class (e.g., system). Dependencies or references in the appended claims are chosen solely for formal reasons. However, any subject matter arising from intentional references to any prior claim (particularly multiple dependent claims) may also be claimed, such that any combination of multiple claims and their multiple features, regardless of the dependency chosen in the appended claims, is disclosed and can be claimed. Claimable subject matter includes not only combinations of multiple features stated in the appended claims, but also any other combination of multiple features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim, and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature in the appended claims. Attached Figure Description

[0029] Figure 1 An example virtual reality system worn by a user is shown.

[0030] Figure 2 An example of the passthrough function in a virtual reality system is shown.

[0031] Figure 3 An example of a mobile computing device displaying a projected menu screen is shown.

[0032] Figure 4 An example of a mobile computing device displaying a projection mode menu is shown.

[0033] Figure 5 The mobile computing device is shown in the demonstration mode as a display from Figure 4 Examples of the selected patterns.

[0034] Figure 6 An example of a mobile computing device displaying an application library is shown.

[0035] Figure 7An example of a mobile computing device displaying a VR environment is shown.

[0036] Figure 8 An example is shown where a mobile computing device displays an avatar in a VR environment.

[0037] Figure 9 An example of a mobile computing device displaying a personal user interface is shown.

[0038] Figure 10 An example of a mobile computing device displaying a game application is shown.

[0039] Figure 11 An example is shown of a mobile computing device displaying a game application available in its MR view.

[0040] Figure 12 An example of a mobile computing device displaying a notification to set up an MR experience is shown.

[0041] Figure 13 An example is shown where a mobile computing device displays images captured from the first user wearing a VR display in a real-world environment.

[0042] Figure 14 An example is shown where a mobile computing device displays a first user's MR view in a VR environment.

[0043] Figure 15 An example is shown where a mobile computing device displays an MR view of the first user in a VR environment, based on the first pose.

[0044] Figure 16 An example is shown where a mobile computing device displays an MR view of a first user in a VR environment based on a second pose.

[0045] Figure 17 An example is shown where a mobile computing device displays an MR view of a first user in a VR environment based on a third pose.

[0046] Figure 18 An example method for generating MR renders is shown.

[0047] Figure 19 An example network environment associated with a social networking system is shown.

[0048] Figure 20 An example computer system is shown. Detailed Implementation

[0049] In certain embodiments, users of mobile computing devices (e.g., smartphones) may be able to view mixed reality (MR) renderings of a first user immersed in a virtual reality (VR) environment. However, users of mobile computing devices may be limited in the amount of MR rendering content they can view about the first user (e.g., they may be limited to the first user's viewpoint) and how they can interact with the VR environment (e.g., they may be passive viewers who cannot interact with the first user or the VR environment). Therefore, a technical challenge may include providing users of mobile computing devices with more experience than a static viewing experience while the first user is immersed in the VR environment. One solution proposed by the embodiments disclosed herein for addressing the technical challenge of providing an interactive experience for users of mobile computing devices is, for example, by allowing users of mobile computing devices to interact with MR and VR content independently of what the user immersed in the VR environment is doing, and by allowing users of mobile computing devices to dynamically interact with the MR renderings (e.g., the MR renderings of the first user immersed in the VR environment) and the VR environment. By way of example and not limitation, a mobile computing device can render MR images of a first user in a VR environment and allow the user of the mobile computing device to move the mobile computing device to other poses (e.g., position and orientation), and the VR system can dynamically update the MR rendering in real time without additional setup and repositioning processes. Although this disclosure describes a method for generating dynamic MR content in a VR environment in a particular manner, this disclosure contemplates generating dynamic MR content in a VR environment in any suitable manner.

[0050] Figure 1 An example of a virtual reality system 100 worn by user 102 is shown. In a particular embodiment, the virtual reality system 100 may include a head-mounted VR display device 135, a controller 106, and one or more computing systems 110. The VR display device 135 may be worn on the user's eyes and provide visual content to user 102 via an internal display (not shown). The VR display device 135 may have two separate internal displays, one for each eye of user 102 (a single display device is also possible). Figure 1 As shown, the VR display device 135 can completely obscure the user's field of view. By becoming the sole provider of visual information for user 102, the VR display device 135 achieves the goal of providing an immersive artificial reality experience. However, one consequence of this is that user 102 may be unable to see the physical (real-world) environment around them because their field of view is obstructed by the VR display device 135. Therefore, the pass-through function described herein can technically facilitate providing users with real-time visual information about their physical environment.

[0051] Figure 2An example of the pass-through function of a virtual reality system 100 is illustrated. User 102 may be wearing a VR display device 135, immersed in a virtual reality environment. A real-world object 145 is located in the physical environment surrounding user 102. However, because the VR display device 135 obstructs user 102's field of vision, user 102 cannot directly see the real-world object 145. To help the user perceive their physical environment while wearing the VR display device 135, the pass-through function uses, for example, one or more cameras 105 (e.g., outward-facing cameras 105A and 105B) to collect information about the physical environment. The collected information can then be reprojected back to user 102 based on their viewpoint. In a particular embodiment of the VR display device 135 having a right display 136A for the user's right eye and a left display 136B for the user's left eye, the virtual reality system 100 may respectively (1) render a reprojected view 145A of the physical environment for the right display 135A based on the user's right eye viewpoint and (2) render a reprojected view 145B of the physical environment for the left display 135B based on the user's left eye viewpoint.

[0052] Refer again Figure 1 The VR display device 135 may have an outward-facing camera, for example... Figure 1 Two forward-facing cameras, 105A and 105B, are shown. Although only two forward-facing cameras, 105A and 105B, are shown, the VR display device 135 can have any number of cameras facing any direction (e.g., an upward-facing camera for capturing ceiling or room lighting effects, a downward-facing camera for capturing a portion of the user's face and / or body, a rearward-facing camera for capturing a portion of something behind the user, and / or an internal camera for capturing the user's eye gaze for eye-tracking purposes). Outward-facing cameras can be configured to capture the physical environment around the user, and such captures can be made continuously to generate a series of frames (e.g., as video). As previously explained, although the images captured by the forward-facing cameras 105A and 105B can be displayed directly to the user 102 via the VR display device 135, this may not provide the user with an accurate view of the physical environment because cameras 105A and 105B cannot be physically positioned exactly at the same location as the user's eyes. Therefore, the pass-through function described in this paper can use a reprojection technique to generate a three-dimensional (3D) representation of the physical environment, and then render an image based on that 3D representation according to the user's eye viewpoint.

[0053] The 3D representation can be generated based on depth measurements of a physical object observed through cameras 105A and 105B. Depth can be measured in various ways. In a particular embodiment, depth can be calculated based on stereo images. For example, two forward-facing cameras 105A and 105B can share overlapping fields of view and can be configured to acquire images simultaneously. Therefore, the same physical object can be captured simultaneously by both cameras 105A and 105B. For example, in the image captured by camera 105A, a specific feature of the object may appear at one pixel pA, while in the image captured by camera 105B, the same feature may appear at another pixel pB. As long as the depth measurement system knows that these two pixels correspond to the same feature, the virtual reality system 100 can use triangulation techniques to calculate the depth of the observed feature. For example, based on the position of camera 105A in 3D space and the pixel position of pA relative to the field of view of camera 105A, a line passing through pixel pA can be projected from camera 105A. A similar line passing through pixel pB can be projected from the other camera 105B. Since it is assumed that the two pixels correspond to the same physical feature, the two lines should intersect. The two intersecting lines, along with an imaginary line drawn between the two cameras 105A and 105B, form a triangle that can be used to calculate the distance of the observed feature from either camera 105A or 105B, or from a point in space where the observed feature is located.

[0054] In certain embodiments, the pose of the VR display device 135 in the environment (e.g., xyz position and rpy orientation) may be required. For example, in order to render a suitable display for the user 102 as they move around in the virtual environment, the virtual reality system 100 may need to determine their position and orientation at any given time. The virtual reality system 100 may also determine the viewpoint of either of the cameras 105A and 105B or the viewpoint of either of the user's eyes based on the pose of the VR display device. In certain embodiments, the VR display device 135 may be equipped with an inertial measurement unit (IMU). Data generated by the IMU, along with stereoscopic images captured by the outward-facing cameras 105A and 105B, allows the virtual reality system 100 to calculate the pose of the VR display device 135 using, for example, simultaneous localization and mapping (SLAM) or other suitable techniques.

[0055] In a particular embodiment, the virtual reality system 100 may also include one or more controllers 106 that enable the user 102 to provide input. The controllers 106 may communicate with the VR display device 135 or one or more separate computing systems 110 via a wireless or wired connection. The controllers 106 may have any number of buttons or other mechanical input mechanisms. Furthermore, the controllers 106 may have an IMU (Integrated Device Unit) to allow for the tracking of the pose of the controllers 106. The controllers may also be tracked based on predetermined patterns on them. For example, the controllers 106 may have several infrared light-emitting diodes (LEDs) or other known observable features that together form a predetermined pattern. The virtual reality system 100 may be able to use sensors or cameras to acquire images of the predetermined patterns on the controllers. The system can calculate the position and orientation of the controllers relative to the sensors or cameras based on the observed orientation of these patterns.

[0056] The virtual reality system 100 may also include one or more computing systems 110. These computing systems 110 may be separate units physically independent of the VR display device 135, or they may be integrated with the VR display device 135. In embodiments where the one or more computing systems 110 are separate units, they may be communicatively coupled to the VR display device 135 via a wireless or wired link. The one or more computing systems 110 may be high-performance devices (e.g., desktop or laptop computers) or resource-constrained devices (e.g., mobile phones). High-performance devices may have a dedicated graphics processing unit (GPU) and a high-capacity or constant power supply. On the other hand, resource-constrained devices may not have a GPU and may have limited battery capacity. Therefore, the algorithms that can be actually used by the virtual reality system 100 depend on the capabilities of its one or more computing systems 110.

[0057] In embodiments where one or more computer systems 110 are high-performance devices, an embodiment with pass-through functionality can be designed as follows. A series of images of the surrounding physical environment can be captured via outward-facing cameras 105A and 105B of the VR display device 135. However, the information captured by cameras 105A and 105B may not be aligned with the information that the user's eyes can capture because the cameras cannot spatially coincide with the user's eyes (e.g., the cameras may be located at a distance from the user's eyes and therefore have different viewpoints). Therefore, simply displaying the content captured by the cameras to the user may not be an accurate representation of what the user should perceive.

[0058] The pass-through function can reproject information captured by outward-facing cameras 105A and 105B to the user, rather than simply displaying the captured content. Each pair of simultaneously acquired stereo images can be used to estimate the depth of the observed features. As explained above, to measure depth using triangulation, one or more computing systems 110 can find correspondences between the stereo images. For example, the one or more computing systems 110 can determine which two pixels in the pair of stereo images correspond to the same observed feature. One or more high-performance computing systems 110 can use their GPUs and optical flow techniques optimized for such tasks to solve the correspondence problem. The correspondence information can then be used to calculate depth using triangulation. Based on the calculated depth of the observed features, the one or more computing systems 110 can determine the location of these features in 3D space (because the one or more computing systems 110 also know the camera's location in that 3D space). The results can be represented by a dense 3D point cloud, where each point corresponds to an observed feature. The dense point cloud can then be used to generate a 3D model of objects in the environment. When the system renders a scene for display, it can perform a visibility test based on the user's eye perspective. For example, the system can project light into the 3D space based on the viewpoint corresponding to each of the user's eyes. In this way, the rendered scene displayed to the user can be calculated based on the user's eye perspective, rather than based on the perspective of the outward-facing cameras 105A and 105B.

[0059] However, the above process may be infeasible for resource-constrained computing units (e.g., a mobile phone may be the primary computing unit of a VR display device 135). For example, unlike systems with powerful computing resources and ample energy, mobile phones cannot rely on GPUs and computationally expensive algorithms (e.g., optical flow) to perform depth measurements and generate accurate 3D models of the environment. Therefore, an optimized process is needed to provide pass-through on resource-constrained devices.

[0060] In a particular embodiment, as discussed in further detail below, the computing device may be configured to dynamically determine at runtime whether it has the capability or ability to generate depth measurement results using: (1) a GPU and optical flow; or (2) optimization techniques using a video encoder and motion vectors. For example, if the device has a GPU and a sufficient power budget (e.g., the device is plugged into a power source, the battery is fully charged, etc.), the device may use its GPU and optical flow to perform depth measurements. However, if the device does not have a GPU or has a limited power budget, the device may choose an optimized method to calculate the depth.

[0061] Figure 3An example of a projection menu screen 210 displayed on a mobile computing device 200 is shown. The projection menu screen 210 may display information indicating which device is projecting (e.g., VR display device 135a) and which device is receiving the projection (e.g., mobile computing device 200a). The projection menu screen 210 may display a start button 215 to begin projection and a help menu 220 for troubleshooting projection. A cursor 205 may be used to navigate the mobile computing device 200. For example, the cursor 205 may be used to navigate the projection menu screen 210. The cursor 205 may indicate the location where a user of the mobile computing device 200 touches the screen of the mobile computing device 200.

[0062] Figure 4 An example of a projection mode menu 225 displayed on a mobile computing device 200 is shown. The projection mode menu 225 can display information indicating the selection of different modes. For example, a user can select viewing mode 226, where the mobile computing device 200 displays a head-mounted device view based on the perspective of the VR display device 135. That is, the user of the mobile computing device 200 can see a first-person view of what the user 102 wearing the VR display device 135 is viewing. Alternatively, the user of the mobile computing device 200 can see an over-the-shoulder third-person view of the user 102 wearing the VR display device 135. As another example, a user can select demonstration mode 227, where the mobile computing device 200 can display a demonstration mode that teaches the user of the mobile computing device 200 how to use the features and functions. That is, demonstration mode 227 can be used, for example, to guide the user on how to navigate in the VR environment 235 using the mobile computing device 200. As another example, a user can select MR mode 228, where the user can view a first user (e.g., user 102) within the VR environment 235. In other words, the user can view a split view of the first user within VR environment 235 (discussed in further detail below). MR mode 228 allows user 102 to access an MR library 231 of associated MR applications and functions. As another example, the user can select game mode 229, where a user of mobile computing device 200 can interact with a first user (e.g., user 102) wearing VR display device 135. The user of mobile computing device 200 can play multiplayer games with the first user or interact with VR characters and / or VR environment 235. Game mode 229 allows user 102 to access a game mode library 232 of associated multiplayer or interactive VR applications and functions.

[0063] Figure 5 The mobile computing device 200 is shown displaying a demonstration mode 227 as a device from... Figure 4Examples of selected modes are shown below. Demonstration mode 227 may include (for example, selectable using cursor 205) multiple functions, such as stop projection 245, settings 250, switch application 255, transition 260, and recenter view 265. The stop projection 245 function can be selected to end the selected view (here, demonstration mode 227). The settings 250 function can be used to adjust user settings and preferences on the mobile computing device 200. The switch application 255 function can be used to switch between different applications (for example, if in game mode 229, the switch application 255 function can be used to select a different application from the game mode library 232). The transition 260 function can be used to switch between a VR environment 235 and a pass-through view of the real-world environment. Transition 260 can be a binary button (where VR environment 235 is displayed or the real-world environment is displayed), or a slider can be provided to gradually transition from VR environment 235 to the real-world environment and vice versa. The Recenter View 265 function can be used to recenter the view of the VR environment 235 on the mobile computing device 200.

[0064] Figure 6 An example of a mobile computing device 200 displaying an application library is shown. Here, the application library is an MR library 231 that displays one or more applications 270. For example, there may be action game applications, space conquest applications, racing games, sports events (e.g., live basketball games), crime fighting applications, and video streaming applications.

[0065] Figure 7 An example of a mobile computing device 200 displaying a VR environment 235 is shown. While in the VR environment 235, a user of the mobile computing device 200 can use the snapshot 275 function to take a snapshot of the VR environment 235. As another example, the user can use the recording 280 feature to record the VR environment 235. That is, the mobile computing device 200 can be used to create video files or for streaming the VR environment 235. As another example, a user of the mobile computing device 200 can use the image exchange format file (GIF) 285 function to create a GIF file. A user of the VR display device 135 can remotely operate the snapshot 275 function, the recording 280 function, and the GIF 285 function (among other functions). Furthermore, a user of the VR display device 135 can have the option to view a "live selfie" view (e.g., a broadcast to a user wearing the VR display device 135) to see the content captured or recorded by one or more of these functions. Users of VR display device 135 and / or users of mobile computing device 200 can broadcast the collected features to other interactive devices of mobile computing devices, web browsers, or other terminal users, and these other terminal users can interact with users of VR display device 135 and / or users of mobile computing device 200.

[0066] Figure 8 An example is shown of an avatar 290 displayed in a VR environment 235 on a mobile computing device 200. Avatar 290 may be an avatar representing a first user (e.g., user 102) wearing the VR display device 135. The user of the mobile computing device 235 can zoom in and out of the avatar 290 in the VR environment 235 using the zoom 295 function. The user of the mobile computing device 200 can also move within the VR environment 235 by dragging, clicking, pinching, or swiping on the display of the mobile computing device 200.

[0067] Figure 9 An example of a mobile computing device 200 displaying a personal user interface (UI) 305 is shown. The personal UI 305 may have first shape elements and a first pose relative to a user (e.g., the user of the mobile computing device and / or a first user). The personal UI 305 may be a virtual object including one or more 2D virtual displays. The personal UI 305 is operable to execute multiple applications.

[0068] Figure 10 An example of a mobile computing device 200 displaying a game application is shown. Figure 11 An example is shown of a mobile computing device 200 displaying a game application available in its MR view. For example, Figure 11 Application 270 can be Figure 10 The mobile computing device 200 can use mode switching notification 315 to indicate that the MR view is available. That is, the mobile computing device 200 can indicate to its user that the user can view the MR view of another user (e.g., the user of the mobile computing device 200, a first user, another user, etc.) within the VR application. The user of the mobile computing device 200 can select the MR view mode 310 function to see the first user within the VR environment 235 of the VR application. The user of the mobile computing device 200 can select the VR display device view mode 320 function to display the VR application from the perspective of a user wearing the VR display device 135. Figure 12 An example is shown of a mobile computing device 200 displaying a notification to set up an MR experience. The mobile computing device 200 can display an MR setup notification 325 to allow the user of the mobile computing device 200 to view MR content. [The last sentence appears to be incomplete and possibly refers to a different context.] Figure 11 After the MR view mode 310 and / or mode switching notification 315 are displayed, the MR settings notification 325 can be presented to the user of the mobile computing device 200.

[0069] Figure 13An example of image capture of a first user (e.g., user 102) wearing a VR display device in a real-world environment 150 is shown using a mobile computing device 200. The real-world environment may include user 102 and one or more real-world objects 145 (e.g., furniture, windows, walls, door frames, etc.). The mobile computing device 200 may display one or more spatial mapping graphics 330, which may be displayed when the VR system 100 generates a feature map (e.g., a spatial map) of the real-world environment 150. Within the generated feature map of the real-world environment 150, one or more anchor points may exist corresponding to one or more features in the real-world environment 150. The multiple anchor points and the field of view of one or more cameras of the VR display device 135 and the mobile computing device 200 can be used to determine the relative positions of the devices with respect to each other. As an example and not a limitation, features of real-world objects 145 (e.g., the corner of a bookshelf or the end of a sofa) or fixed, static or permanent fixtures (e.g., the corner of a window or the corner of a wall) within the field of view of both the VR display device 135 and the mobile computing device 200 can be used as anchor points.

[0070] In a particular embodiment, one or more cameras of the mobile computing device 200 may capture one or more images of a first user (e.g., user 102) wearing the VR display device 135 in the real-world environment 150. The VR system 100 may instruct the cameras of the mobile computing device 200 to capture one or more images of the real-world environment, including the first user wearing the VR display device 135. One or more cameras may be present on the mobile computing device 200, which has multiple pixel sensors (or other suitable optical sensors), for capturing images of the real-world environment (including multiple pixels), wherein the multiple pixels may be captured by any one of the one or more cameras. This image capture may be a full-resolution image capture of the first user and one or more real-world objects in the real-world environment 150. By way of example and not limitation, the mobile computing device 200 may capture images of user 102 wearing the VR display device 135 in the real-world environment 150. In a particular embodiment, the cameras of the mobile computing device 200 may capture features of the first user and one or more other users. For example, mobile computing device 200 can determine the pose (e.g., xyz position and rpy orientation) of the first user and / or VR display device 135 relative to mobile computing device 200. Although this disclosure describes acquiring one or more images in a particular manner, this disclosure contemplates acquiring one or more images in any suitable manner.

[0071] In a particular embodiment, the mobile computing device 200 may receive one or more anchor points in a real-world environment 150 identified from one or more images by the VR system 100. These anchor points may correspond to any real-world object 145 or distinguishing feature in the real-world environment 150. For example, anchor points may correspond to static or permanent fixtures (e.g., fixed objects) in the real-world environment 150, such as entrances, windows, pieces of furniture, etc. In a particular embodiment, these anchor points may be used to determine the pose of the mobile computing device 200 relative to the VR display device 135. The anchor points may be used to calibrate the pose of the mobile computing device 200 and / or the VR display device 135 by positioning the mobile computing device 200 and the VR display device 135 relative to the identified anchor points. The VR system 100 may automatically perform an initial positioning process (e.g., a calibration process) to position the mobile computing device 200 and the VR display device 135 relative to the real-world environment and / or relative to each other.

[0072] In a particular embodiment, determining the pose of the mobile computing device 200 may include detecting that the mobile computing device 200 has moved (e.g., using a position sensor such as an accelerometer or using an image processing algorithm that determines changes in the position of the mobile computing device 200 based on comparisons of individual images). Then, an updated pose of the mobile computing device 200 relative to the VR display device 135 may be determined based on one or more anchor points. That is, the mobile computing device 200 may reposition itself based on the determination of its position relative to one or more anchor points. In response to detecting that the mobile computing device 200 has moved, the VR system may generate an updated MR rendering of a first user (e.g., user 102) in the VR environment 235 in real time. The updated MR rendering of the first user may be based on the updated pose of the mobile computing device 200 relative to the VR display device 135. That is, the pose of the mobile computing device 200 relative to the VR display device 135 may be determined based on triangulation between: one or more of the one or more anchor points; the mobile computing device 200 (e.g., one or more cameras of the mobile computing device 200); and the VR display device 135. Therefore, based on the updated pose of the mobile computing device 200 after movement, the VR system 100 can generate an updated MR rendering of the first user in the VR environment 235, and the mobile computing device 200 can display this updated MR rendering of the first user in the VR environment 235. For example, if the mobile computing device 200 has moved to the left of the user 102 wearing the VR display device 135, the MR rendering will accordingly show that the mobile computing device 200 has moved to the left of the user 102 wearing the VR display device 135. Therefore, the poses of the mobile computing device 200 and the VR display device can be determined relative to each other and relative to the real-world environment 150. Anchor points can be used to continuously or periodically update the respective poses of the mobile computing device 200 and the VR display device 135 by tracking the relative position of the mobile computing device 200 and the VR display device 135 (e.g., by using simultaneous localization and mapping techniques to track the relative position of the mobile computing device 200 and the VR display device 135). The feature map of the real-world environment 150 generated by the VR system 100 can be used to determine one or more anchor points corresponding to features within the feature map. The feature map and anchor points can be generated and continuously updated by the VR system 100 of the VR display device 135. Technical advantages of various embodiments may include allowing a user of the mobile computing device 200 to move the mobile computing device 200 to other poses (e.g., position and orientation), while the VR system 100 can dynamically update the MR rendering in real time without additional setup and repositioning procedures for the mobile computing device 200.In other words, VR system 100 can reposition mobile computing device 200 only by referring to a feature map, rather than repositioning it relative to VR display device 135. Therefore, this can have the advantage of reducing the computing and processing power required by mobile computing device 200. That is, VR display device 135 can track its pose relative to anchor points in the feature map, and mobile computing device 200 can also track its pose relative to the same anchor points in the feature map. Although this disclosure describes device positioning in a particular manner, this disclosure contemplates device positioning in any suitable manner.

[0073] In a particular embodiment, the mobile computing device 200 can send its pose to the VR system 100 of the VR display device 135. That is, the mobile computing device 200 can send information such as its pose to the VR system 100. The mobile computing device 200 can also send information such as one or more images of a first user wearing the VR display device 135 to the VR system 100. This is by way of example and not limitation, and is for reference only. Figure 13 The mobile computing device 200 can send images of the user 102 in the real-world environment 150 to the VR system 100 on the VR display device 135. Although this disclosure describes the transmission of pose information in a particular manner, this disclosure contemplates the transmission of pose information in any suitable manner.

[0074] Figure 14 An example is shown where a mobile computing device 200 displays a user 102's MR view within a VR environment 235. The user 102, wearing a VR display device 135, may be playing a VR game application set up within the VR environment 235. The user of the mobile computing device 200 can use cursor 205 to view and explore the VR environment 235. The user of the mobile computing device 200 can use the snapshot 275 function to take a photo of the user 102 within the VR environment 235, use the recording 280 function to record the user 102 within the VR environment 235, or use the GIF 285 function to create a GIF file of the user 102 within the VR environment 235.

[0075] In a particular embodiment, the mobile computing device 200 may receive VR rendering of the VR environment 235 from the VR system 100. This VR rendering can be based on the viewing angle of the mobile computing device 200 relative to the VR display device 135. The viewing angle of the mobile computing device 200 can be determined based on its pose relative to the VR display device 135. This is by way of example and not limitation, and reference is made to... Figure 14The mobile computing device 200 can display VR rendering of the VR environment 235 from the perspective of the mobile computing device 200, based on the pose of the mobile computing device 200 relative to the VR display device 135. The mobile computing device 200 and the VR display device 135 can be time-synchronized, so that the rendering of the VR environment 235 displayed on the VR display device 135 is received at the same timestamp as the rendering of the VR environment displayed on the mobile computing device 200. Through this synchronization, the mobile computing device 200 and the VR display device 135 can seamlessly view the same VR environment 235. Although this disclosure describes receiving VR rendering in a specific manner, this disclosure contemplates receiving VR rendering in any suitable manner.

[0076] In a particular embodiment, mobile computing device 200 may receive an indication from the VR system that a first user (e.g., user 102) has activated a specific VR application on VR system 100. As an example, and not a limitation, the first user wearing VR display device 135 may select a VR game application. Mobile computing device 200 may, in response to receiving the indication that the first user has activated a specific VR application on VR system 100, present a notification that the first user wearing VR display device 135 has launched that specific VR application. As an example, and not a limitation, mobile computing device 200 may (e.g., via a banner or prompt popping up on the display of mobile computing device 200) notify the user of mobile computing device 200 that the first user has selected a VR game application. The user of mobile computing device 200 can then (e.g., via...) Figure 11 The mode switching notification 315) allows users to select an option to view the MR view of the first user in the VR environment 235 of the VR game application. Although this disclosure describes ways to facilitate a VR experience, it is contemplated in any suitable way to facilitate a VR experience.

[0077] Figure 15 , Figure 16 and Figure 17An example is shown where a mobile computing device 200 displays an MR view of a first user 102 in a VR environment 235 according to a first pose, a second pose, and a third pose. The mobile computing device 200 displays the first user, segmented from one or more images and rendered in the VR environment 235. A user of the mobile computing device 200 (e.g., an additional user 103) may be using the mobile computing device 200 to view a user (e.g., user 102) wearing a VR display device 135 in a real-world environment 150. The mobile computing device 200 may display a “cutout” of user 102 overlaid on the VR environment 235 (e.g., where user 102 is in a VR gaming application). The VR environment 235 displayed on the mobile computing device 200 may correspond to the VR environment 235 displayed on the VR display device 135 worn by user 102. As the attached user 103 and / or mobile computing device 200 move, and as user 102 and VR display device 135 move, the display of mobile computing device 200 updates the position of user 102 and the pose of mobile computing device 200 "within" VR environment 235. Therefore, as mobile computing device 200 moves around in real-world environment 150 (e.g., from a first pose to an updated pose), the MR rendering of first user 102 in VR environment 235 is automatically and in real-time updated relative to and based on the updated pose of mobile computing device 200, displaying first user 102 in VR environment 235.

[0078] In a particular embodiment, a first user (e.g., user 102) can be segmented from one or more images. That is, the silhouette, body, or "silhouette" of the first user can be segmented from the background real-world environment 150 to overlay or superimpose onto the VR environment 235 displayed on the mobile computing device 200. By way of example and not limitation, segmentation algorithms or image processing algorithms can be used to identify and determine the silhouette or body of user 102 in the real-world environment 150, and the silhouette or body of user 102 can be extracted or segmented from images containing user 102 in the real-world environment 150. Furthermore, in a particular embodiment, only the first user can be segmented from one or more images. That is, the VR system 100 can segment only the first user from one or more images, ignoring (e.g., not segmenting) other users. In a particular embodiment, the VR system 100 can segment the user wearing the VR display device 135 (e.g., the VR system 100 uses depth estimation to create a bounding volume around the display device 135 to determine which user to segment). Segmenting the first user addresses the technical challenge of allowing dynamic movement of the mobile computing device 200 and maintaining the image of the first user immersed in the VR environment 235 within the image frame on the mobile computing device 200. Therefore, the user of the mobile computing device 200 can maintain MR rendering of the first user in the VR environment 235 without requiring a static camera fixed to the first user. The segmentation of the first user can be stored as segmentation data. This segmentation data can be retained at or maintained at the VR system 100. Retaining the segmentation data allows MR or augmented reality (AR) effects to be applied to the segmentation of the first user. For example, if the first user is in a VR sports game environment, MR or AR effects can be applied to the clothing worn by the segmented first user, making the first user appear to be wearing a sports team jersey. Although this disclosure describes segmenting a user from one or more images in a specific manner, this disclosure contemplates segmenting a user from one or more images in any suitable manner.

[0079] In a particular embodiment, VR system 100 may, in response to acquiring one or more images containing a first user (e.g., user 102), generate an MR rendering of the first user (e.g., user 102) in VR environment 235 in real time at mobile computing device 200. The MR rendering of the first user may be based on a composite of one or more images of the segmented first user with the VR rendering. That is, the MR rendering may include the first user (e.g., user 102), segmented from one or more images of user 102 located within the field of view (FOV) of mobile computing device 200 in the real-world environment 235, and rendered (e.g., overlaid or superimposed) onto the VR environment 235 displayed on mobile computing device 200. As user 102 moves and / or interacts with the VR environment 235 displayed on VR display device 135, a user of mobile computing device 200 (e.g., an additional user 103) may view a 2D rendering of user 102 overlaid on VR environment 235. In a particular embodiment, the mobile computing device 200 can be used to interact with the VR environment 235 via MR rendering of user 102 within the VR environment 235 displayed on the mobile computing device 200. That is, while user 102 moves or interacts with the VR environment 235, a user of the mobile computing device 200 (e.g., an additional user 103) may be able to view and interact with user 102. The additional user 103 of the mobile computing device 200 can move and interact with the VR environment 235 independently of user 102 wearing the VR display device 135. For example, user 102 can play a VR game application, while the additional user 103 of the mobile computing device 200 can view the VR environment 235 behind user 102. The VR system can synthesize one or more images, which can be performed at the mobile computing device 200 or at the VR display device 135. The technical advantages of this embodiment may include: generating MR rendering of a first user (e.g., user 102) in VR environment 235 on a separate display (e.g., mobile computing device 200) without using a green screen or other professional equipment.

[0080] Technical challenges may include interacting with generated MR and VR content without requiring specialized equipment, such as not requiring the user of mobile computing device 200 to use specialized VR gaming equipment to interact with a first user in a VR game application. The solution to this challenge proposed in the embodiments disclosed herein may be to leverage the rendering and spatial mapping capabilities of a VR system to allow the user of mobile computing device 200 to interact with the generated MR and VR content through mobile computing device 200. That is, the user of mobile computing device 200 can interact with a first user in VR environment 235 using only their mobile computing device 200 (e.g., their smartphone). For example, the user of mobile computing device 200 can click icons or buttons on the display of mobile computing device 200 to set up obstacles for the first user to interact in a VR game application. Interacting with the VR environment 235 may include one or more of the following: modifying a VR object in the VR environment 235; creating a VR object in the VR environment 235; modifying an MR object in the VR environment 235; creating an MR object in the VR environment 235; modifying one or more attributes of the VR environment 235; changing the VR environment 235 from a first VR environment to a second VR environment; changing the VR application; recording MR rendering; or streaming MR rendering. In a particular embodiment, the mobile computing device 200 may send the generated MR rendering of the first user in the VR environment 235 to an online social network in real time for live display to one or more additional users. That is, the mobile computing device 200 may live stream the generated MR rendering on an online social network or other streaming service (e.g., using the recording 280 function button). In addition, users of the mobile computing device 200 may interact with the VR environment by creating a VR avatar of a second user or additional user (e.g., additional user 103) corresponding to the user of the mobile computing device 200. Therefore, a user of mobile computing device 200 can interact with a first user within the same VR environment 235 (e.g., a user of VR display device 135 and a user of mobile computing device 200 can "sit" on the same VR sofa and watch a movie together). A user of mobile computing device 200 can also interact with a VR avatar of another user (different from the user of VR display device 135 and the user of mobile computing device 200). In a particular embodiment, and referring to... Figure 11The mobile computing device 200 can be used to switch between VR rendering and MR rendering in response to a selection on the mobile computing device 200 (e.g., selection via cursor 205). The mobile computing device 200 can be used to select between: VR rendering of the VR environment 235 based on the perspective of a first user (e.g., by selecting VR display device view mode 320); and MR rendering of the first user in the VR environment 235 based on the perspective of the mobile computing device 200 relative to the VR display device 135 (e.g., by selecting MR view mode 310). Although this disclosure describes the generation of MR rendering, this disclosure contemplates generating MR rendering in any suitable manner.

[0081] Figure 18 An example method 1800 for generating MR renderings is shown. The method may begin at step 1810, in which one or more cameras of a mobile computing device may capture one or more images of a first user wearing a VR display device 135 in a real-world environment. At step 1820, the mobile computing device may send its pose to the VR system of the VR display device 135. At step 1830, the mobile computing device may receive a VR rendering of the VR environment from the VR system. This VR rendering can be based on the viewpoint of the mobile computing device relative to the VR display device 135. At step 1840, the first user may be segmented from the one or more images. At step 1850, the mobile computing device may generate an MR rendering of the first user in the VR environment in real time in response to the capture of one or more images. The MR rendering of the first user may be based on a composite of the segmented one or more images of the first user and the VR rendering. Certain embodiments may be repeated where appropriate. Figure 18 One or more steps in the method. Although this disclosure will Figure 18 The specific steps of the method are described and shown as being performed in a particular order, but this disclosure contemplates performing them in any suitable order. Figure 18 Any suitable steps of the method. Furthermore, although this disclosure describes and illustrates methods for generating MR renderings, including... Figure 18 This disclosure describes example methods for specific steps of a method, but it contemplates any suitable method for generating MR renders that includes any appropriate steps, and where appropriate, the method may include... Figure 18 The method may include all steps, some steps, or none of the steps. Figure 18 Any steps of the method. Furthermore, although this disclosure describes and illustrates the implementation of... Figure 18 The method may refer to a specific component, device, or system of a particular step, but this disclosure is contemplated for the execution of... Figure 18 Any suitable step of the method, any suitable component, device or system, or any suitable combination thereof.

[0082] Figure 19 An example network environment 1900 associated with a social networking system is shown. Network environment 1900 includes client systems 1930, a social networking system 1960, and a third-party system 1970 connected to each other via network 1910. Although Figure 19 A specific arrangement of client system 1930, social networking system 1960, third-party system 1970, and network 1910 is shown, but this disclosure considers any suitable arrangement of client system 1930, social networking system 1960, third-party system 1970, and network 1910. By way of example and not limitation, two or more of client system 1930, social networking system 1960, and third-party system 1970 may bypass network 1910 and be directly connected to each other. As another example, two or more of client system 1930, social networking system 1960, and third-party system 1970 may be physically or logically located in the same place, either wholly or partially. Furthermore, although... Figure 19 A specific number of client systems 1930, social networking systems 1960, third-party systems 1970, and networks 1910 are shown, but this disclosure contemplates any suitable number of client systems 1930, social networking systems 1960, third-party systems 1970, and networks 1910. As an example and not a limitation, network environment 1900 may include multiple client systems 1930, multiple social networking systems 1960, multiple third-party systems 1970, and multiple networks 1910.

[0083] This disclosure considers any suitable network 1910. By way of example and not limitation, one or more portions of network 1910 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 1910 may include one or more networks 1910.

[0084] Link 1950 can connect client system 1930, social networking system 1960, and third-party system 1970 to communication network 1910 or to each other. This disclosure contemplates any suitable link 1950. In certain embodiments, one or more links 1950 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In a particular embodiment, one or more links 1950 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular-based network, a satellite-based network, another link 1950, or a combination of two or more such links 1950. The links 1950 need not be identical throughout the network environment 1900. In one or more aspects, one or more first links may differ from one or more second links.

[0085] In a particular embodiment, client system 1930 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of both or more, and is capable of performing suitable functions implemented or supported by client system 1930. By way of example and not limitation, client system 1930 may include computer systems such as desktop computers, laptop or notebook computers, netbooks, tablet computers, e-book readers, global positioning system (GPS) devices, cameras, personal digital assistants (PDAs), handheld electronic devices, cellular phones, smartphones, augmented / virtual reality devices, other suitable electronic devices, or any suitable combination thereof. This disclosure contemplates any suitable client system 1930. Client system 1930 enables network users at client system 1930 to access network 1910. Client system 1930 enables its users to communicate with other users at other client systems 1930.

[0086] In a particular embodiment, client system 1930 may include web browser 1932 and may have one or more add-ons, plugins, or other extensions. A user at client system 1930 may enter a Uniform Resource Locator (URL) or direct web browser 1932 to a specific server (e.g., server 1962, or a server associated with third-party system 1970), and web browser 1932 may generate a Hypertext Transfer Protocol (HTTP) request and send that HTTP request to the server. The server may receive the HTTP request and, in response, send one or more Hypertext Markup Language (HTML) files to client system 1930. Client system 1930 may render the webpage to the user based on the HTML files from the server. This disclosure considers any suitable webpage file. By way of example and not limitation, webpages may be rendered from HTML files, Extensible Hypertext Markup Language (XHTML) files, or Extensible Markup Language (XML) files, depending on specific needs. Such pages can also execute scripts, as well as combinations of markup languages ​​and scripts. In this article, where appropriate, a reference to a webpage includes one or more corresponding webpage files (which the browser can use to render the webpage), and vice versa.

[0087] In a particular embodiment, the social networking system 1960 may be a network-addressable computing system capable of hosting online social networks. The social networking system 1960 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to the online social network. The social networking system 1960 may be accessed directly by other components in the network environment 1900 or via network 1910. By way of example and not limitation, the client system 1930 may use a web browser 1932 or a native application associated with the social networking system 1960 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof) to access the social networking system 1960 directly or via network 1910. In a particular embodiment, the social networking system 1960 may include one or more servers 1962. Each server 1962 may be a single server or a distributed server spanning multiple computers or multiple data centers. Server 1962 can be of various types, including, but not limited to, web servers, news servers, mail servers, messaging servers, advertising servers, file servers, application servers, exchange servers, database servers, proxy servers, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 1962 may include hardware, software, or embedded logic components, or a combination of both or more of these components, for performing suitable functions implemented or supported by server 1962. In a particular embodiment, social networking system 1960 may include one or more data repositories 1964. Data repositories 1964 can be used to store various types of information. In a particular embodiment, the information stored in data repositories 1964 may be organized according to a particular data structure. In a particular embodiment, each data repository 1964 may be a relational database, columnar database, association database, or other suitable database. Although this disclosure describes or illustrates specific types of databases, this disclosure contemplates any suitable type of database. Specific embodiments may provide an interface that enables client system 1930, social network system 1960, or third-party system 1970 to manage, retrieve, modify, add, or delete information stored in data repository 1964.

[0088] In a particular embodiment, the social network system 1960 may store one or more social graphs in one or more data repositories 1964. In a particular embodiment, the social graph may include multiple nodes—which may include multiple user nodes (each user node corresponding to a specific user) or multiple concept nodes (each concept node corresponding to a specific concept)—and multiple edges connecting the multiple nodes. The social network system 1960 may provide users of the online social network with the ability to communicate and interact with other users. In a particular embodiment, a user can join an online social network through the social network system 1960 and subsequently add connections (e.g., relationships) to multiple other users they wish to connect to within the social network system 1960. In this document, the term "friend" may refer to any other user in the social network system 1960 with whom the user has already formed a connection, association, or relationship through the social network system 1960.

[0089] In a particular embodiment, the social networking system 1960 may provide users with the ability to take actions on various types of items or objects supported by the social networking system 1960. By way of example, and not limitation, these items and objects may include groups or social networks to which the user of the social networking system 1960 may belong, events or calendar entries that the user may be interested in, computer-based applications available to the user, transactions that allow the user to buy or sell items through services, user-executable interactions with advertisements, or other suitable items or objects. Users may interact with anything that can be represented in the social networking system 1960 or by an external system of a third-party system 1970, which is separate from the social networking system 1960 and coupled to the social networking system 1960 via network 1910.

[0090] In a particular embodiment, the social networking system 1960 may be able to link various entities. By way of example and not limitation, the social networking system 1960 may enable users to interact with each other and receive content from third-party systems 1970 or other entities, or allow users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0091] In a particular embodiment, third-party system 1970 may include one or more types of servers, one or more data repositories, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable component (e.g., with which the server can communicate). Third-party system 1970 may be operated by an entity different from the entity operating social networking system 1960. However, in a particular embodiment, social networking system 1960 and third-party system 1970 may operate collaboratively to provide social networking services to users of either social networking system 1960 or third-party system 1970. In this sense, social networking system 1960 may provide a platform or backbone that other systems (e.g., third-party system 1970) can use to provide social networking services and functionality to users via the Internet.

[0092] In a particular embodiment, third-party system 1970 may include a third-party content object provider. The third-party content object provider may include one or more sources of content objects that can be delivered to client system 1930. As an example, and not a limitation, content objects may include information related to things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example, and not a limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentives.

[0093] In a particular embodiment, the social networking system 1960 also includes user-generated content objects that can enhance user interaction with the social networking system 1960. User-generated content can include anything a user can add, upload, send, or "post" to the social networking system 1960. As an example, and not a limitation, a user transmits a post to the social networking system 1960 from a client system 1930. Posts can include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the social networking system 1960 by third parties via a "communication channel" (e.g., a news feed or stream).

[0094] In a particular embodiment, the social networking system 1960 may include various servers, subsystems, programs, modules, logs, and data repositories. In a particular embodiment, the social networking system 1960 may include one or more of the following: a web server, an operation log recorder, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an operation log, a third-party content object public log, an inference module, an authorization / privacy server, a search module, an ad targeting module, a user interface module, a user profile repository, a connection repository, a third-party content repository, or a location repository. The social networking system 1960 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof. In a particular embodiment, the social networking system 1960 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biometric information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, close relationships, or location). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example, and not a limitation, if a user “likes” items related to a shoe brand, the category could be that brand, or it could be a generic category like “shoes” or “clothing.” A connection repository can be used to store connection information about users. Connection information can indicate users who have similar or shared work experience, group memberships, hobbies, educational history, or are in any way related to or share common attributes. Connection information can also include user-defined connections between different users and content (both internal and external). A web server can be used to link the social networking system 1960 to one or more client systems 1930 or one or more third-party systems 1970 via network 1910. The web server can include a mail server, or other messaging functionality for receiving and routing messages between the social networking system 1960 and one or more client systems 1930. An API request server can allow third-party systems 1970 to access information from the social networking system 1960 by calling one or more APIs. An operation logger can be used to receive information from the web server about user actions of starting or closing the social networking system. In conjunction with the operation log, a log of third-party content objects exposed by users can be maintained. The notification controller can provide information about content objects to client system 1930. This information can be pushed to client system 1930 as a notification, or retrieved from client system 1930 in response to a request received from client system 1930. The authorization server can be used to implement one or more privacy settings for users of social networking system 1960.A user's privacy settings determine how specific information associated with that user can be shared. The authorization server allows users to choose, for example, by setting appropriate privacy settings, whether or not the social network system 1960 can record their actions, or share their actions with other systems (e.g., third-party system 1970). A third-party content object repository can be used to store content objects received from third parties (e.g., third-party system 1970). A location repository can be used to store location information received from the client system 1930 associated with the user. The advertising targeting module can combine social information, current time, location information, or other suitable information to deliver relevant advertisements to users in the form of notifications.

[0095] Figure 20 An example computer system 2000 is illustrated. In a particular embodiment, one or more computer systems 2000 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 2000 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 2000 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. The particular embodiments include one or more portions of one or more computer systems 2000. Herein, references to computer systems may cover computing devices and vice versa, where appropriate. Furthermore, references to computer systems may cover one or more computer systems, where appropriate.

[0096] This disclosure contemplates any suitable number of computer systems 2000. This disclosure contemplates computer systems 2000 employing any suitable physical form. By way of example and not limitation, computer system 2000 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, computer system 2000 may include one or more computer systems 2000; may be single or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 2000 may perform one or more steps of the methods described or shown herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 2000 may perform one or more steps of the methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 2000 may perform one or more steps of the methods described or shown herein at different times or in different locations.

[0097] In a particular embodiment, computer system 2000 includes a processor 2002, memory 2004, storage 2006, input / output (I / O) interface 2008, communication interface 2010, and bus 2012. Although this disclosure describes and illustrates a particular computer system having a particular number of components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of components in any suitable arrangement.

[0098] In a particular embodiment, processor 2002 includes hardware for executing instructions (e.g., those that constitute a computer program). By way of example, and not limitation, to execute instructions, processor 2002 may retrieve (or read) multiple instructions from internal registers, internal cache, memory 2004, or memory 2006; decode and execute those instructions; and subsequently write one or more results to internal registers, internal cache, memory 2004, or memory 2006. In a particular embodiment, processor 2002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, this disclosure contemplates processor 2002 including any suitable number of suitable internal caches. By way of example, and not limitation, processor 2002 may include one or more instruction caches, one or more data caches, and one or more page table caches (TLBs). Instructions in the instruction cache may be copies of instructions in memory 2004 or memory 2006, and the instruction cache may accelerate the retrieval of those instructions by processor 2002. Data in the data cache may be: a copy of data in memory 2004 or memory 2006 for operation by instructions executed at processor 2002; the result of a previous instruction executed at processor 2002 for access by subsequent instructions executed at processor 2002 or for writing to memory 2004 or memory 2006; or other suitable data. The data cache can accelerate read or write operations of processor 2002. The TLB can accelerate virtual address translation of processor 2002. In a particular embodiment, processor 2002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, this disclosure contemplates processor 2002 including any suitable number of suitable internal registers. Where appropriate, processor 2002 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 2002. Although this disclosure describes and illustrates specific processors, this disclosure contemplates any suitable processor.

[0099] In a particular embodiment, memory 2004 includes main memory for storing instructions executed by processor 2002 or data operated on by processor 2002. By way of example and not limitation, computer system 2000 may load instructions from memory 2006 or another source (e.g., another computer system 2000) into memory 2004. Processor 2002 may then load these instructions from memory 2004 into internal registers or internal cache memory. To execute these instructions, processor 2002 may retrieve and decode these instructions from internal registers or internal cache memory. During or after the execution of these instructions, processor 2002 may write one or more results (which may be intermediate or final results) to internal registers or internal cache memory. Processor 2002 may then write one or more of these results into memory 2004. In a particular embodiment, processor 2002 executes only instructions in one or more internal registers or internal caches or memory 2004 (as opposed to memory 2006 or elsewhere), and operates only on data in one or more internal registers or internal caches or memory 2004 (as opposed to memory 2006 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) couple processor 2002 to memory 2004. As described below, bus 2012 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between processor 2002 and memory 2004 and facilitate access to memory 2004 requested by processor 2002. In a particular embodiment, memory 2004 includes random access memory (RAM). Where appropriate, RAM may be volatile memory. Where appropriate, RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, RAM may be single-port RAM or multi-port RAM. This disclosure considers any suitable RAM. Where appropriate, memory 2004 may include one or more memory modules. Although this disclosure describes and illustrates specific memory, it considers any suitable memory.

[0100] In a particular embodiment, memory 2006 includes a large-capacity storage for data or instructions. By way of example and not limitation, memory 2006 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 2006 may include removable or non-removable (or fixed) media. Where appropriate, memory 2006 may be located internally or externally to computer system 2000. In a particular embodiment, memory 2006 is a non-volatile solid-state memory. In a particular embodiment, memory 2006 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates a large-capacity memory 2006 in any suitable physical form. Where appropriate, memory 2006 may include one or more memory control units that facilitate communication between processor 2002 and memory 2006. Where appropriate, memory 2006 may include one or more memory units 2006. Although this disclosure describes and illustrates specific memories, it contemplates any suitable memory.

[0101] In a particular embodiment, I / O interface 2008 includes hardware, software, or both that provide one or more interfaces for communication between computer system 2000 and one or more I / O devices. Where appropriate, computer system 2000 may include one or more of these I / O devices. These one or more I / O devices enable communication between a person and computer system 2000. By way of example and not limitation, I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touchscreen, trackball, video camera, another suitable I / O device, or a combination of two or more of these I / O devices. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O device and any suitable I / O interface 2008 for such I / O devices. Where appropriate, I / O interface 2008 may include one or more device or software drivers that enable processor 2002 to drive one or more of these I / O devices. Where appropriate, I / O interface 2008 may include one or more I / O interfaces 2008. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure considers any suitable I / O interface.

[0102] In a particular embodiment, the communication interface 2010 includes hardware, software, or both of these providing one or more interfaces for communication (e.g., packet-based communication) between the computer system 2000 and one or more other computer systems 2000 or with one or more networks. By way of example, and not limitation, the communication interface 2010 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or may include a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 2010 for that network. By way of example, and not limitation, the computer system 2000 may communicate with one or more portions of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these networks. One or more portions of one or more of these networks may be wired or wireless. For example, computer system 2000 may communicate with networks such as wireless PAN (WPAN) (e.g., Bluetooth WPAN), Wi-Fi networks, Wi-MAX networks, cellular telephone networks (e.g., Global System for Mobile Communications (GSM) networks), or other suitable wireless networks, or combinations of two or more of these. Where appropriate, computer system 2000 may include any suitable communication interface 2010 for any of these networks. Where appropriate, communication interface 2010 may include one or more communication interfaces 2010. Although specific communication interfaces are described and illustrated in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0103] In a particular embodiment, bus 2012 includes hardware, software, or both, that couples the components of computer system 2000 to each other. By way of example and not limitation, bus 2012 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Where appropriate, bus 2012 may include one or more buses 2012. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnection.

[0104] In this document, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based circuits or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disc drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0105] In this document, unless otherwise expressly indicated or the context otherwise indicates, "or" is open-ended rather than exclusive. Therefore, in this document, unless otherwise expressly indicated or the context otherwise indicates, "A or B" means "A, B, or both." Furthermore, unless otherwise expressly indicated or the context otherwise indicates, "and" is both common and separate. Therefore, in this document, unless otherwise expressly indicated or the context otherwise indicates, "A and B" means "A and B, commonly or separately."

[0106] The scope of this disclosure covers all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or shown herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or shown herein. Furthermore, although this disclosure describes and illustrates various embodiments herein as including specific components, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the components, elements, features, functions, operations, or steps described or shown anywhere herein that will be understood by those skilled in the art. Moreover, references in the appended claims to means or systems, or components of means or systems, suitable for, arranged as, capable of, configured as, implemented as, operable as, or usable to perform a particular function cover such means, systems, or components (whether or not the means, systems, components, or the particular function are activated, turned on, or unlocked), provided that the means, systems, or components are so suitable for, arranged as, capable of, configured as, implemented as, operable as, or usable. Furthermore, while this disclosure describes or illustrates specific embodiments to provide particular advantages, specific embodiments may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A method for rendering dynamic mixed reality content in virtual reality, comprising: From mobile computing devices: Using one or more cameras of the mobile computing device, one or more images of a first user wearing a virtual reality (VR) display device are captured in a real-world environment, wherein the VR display device is displaying a first VR rendering of the VR environment, and wherein the first VR rendering is from the viewpoint of the VR display device; Sends the pose of the mobile computing device relative to the VR display device to the VR system of the VR display device; The mobile computing device receives a second VR rendering of the VR environment from the VR system, wherein the second VR rendering is based on the viewing angle of the mobile computing device relative to the VR display device; Segmenting the first user from the one or more images; and In response to the acquisition of the one or more images, a mixed reality (MR) rendering of the first user in the VR environment is generated in real time at the mobile computing device, wherein the MR rendering of the first user is based on the synthesis of one or more segmented images of the first user with the second VR rendering.

2. The method according to claim 1, further comprising: The mobile computing device receives one or more anchor points in the real-world environment identified from the one or more images by the VR system.

3. The method according to claim 2, further comprising: The pose of the mobile computing device relative to the VR display device is determined based on one or more of the one or more anchor points.

4. The method according to claim 3, further comprising: The mobile computing device has been detected as having been moved; Based on one or more of the one or more anchor points, determine the updated pose of the mobile computing device relative to the VR display device; as well as In response to detecting that the mobile computing device has moved, an updated MR rendering of the first user in the VR environment is generated in real time at the mobile computing device, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device relative to the VR display device.

5. The method according to any one of claims 1 to 4, wherein, The pose of the mobile computing device relative to the VR display device is determined based on triangulation of one or more of the following: one or more anchor points; one or more cameras of the mobile computing device; and the VR display device.

6. The method according to claim 5, wherein, The one or more anchor points are fixed objects in the real-world environment.

7. The method according to any one of claims 1 to 4, further comprising: In response to a selection on the mobile computing device, switching between the following: The first VR rendering of the VR environment based on the first user's perspective; and The MR rendering of the first user in the VR environment, based on the perspective of the mobile computing device relative to the VR display device.

8. The method according to any one of claims 1 to 4, further comprising: The generated MR rendering of the first user in the VR environment, originating from the mobile computing device, is sent in real time to an online social network for live display to one or more additional users.

9. The method according to any one of claims 1 to 4, wherein, Segmenting the first user from the one or more images includes: Using a segmentation algorithm, the body of the first user is identified within each of the one or more images.

10. The method according to any one of claims 1 to 4, further comprising: The mobile computing device receives an instruction from the VR system that the first user has activated a specific VR application on the VR system. as well as In response to the received instruction, a notification is displayed on the mobile computing device that the first user wearing the VR display device has launched the specific VR application.

11. The method according to any one of claims 1 to 4, wherein, The mobile computing device can be used to interact with the VR environment through the MR rendering.

12. The method according to claim 11, wherein, Interacting with the VR environment includes one or more of the following: Modify VR objects within the VR environment; Create VR objects within the VR environment; Modify the MR object in the VR environment; Create MR objects in the VR environment; Modify one or more properties of the VR environment; Change the VR environment from the first VR environment to the second VR environment; Modify the VR application; Record the MR rendering; Streaming the MR rendering; Create a VR avatar corresponding to a second user of the mobile computing device; or Interact with a VR avatar of a third user.

13. One or more computer-readable nontransitory storage media, the one or more computer-readable nontransitory storage media comprising software, the software being operable when executed by a mobile computing device to: Using one or more cameras of the mobile computing device, one or more images are captured in a real-world environment of a first user wearing a virtual reality (VR) display device, wherein... The VR display device is displaying a first VR rendering of a VR environment, wherein the first VR rendering is from the viewpoint of the VR display device; Sends the pose of the mobile computing device relative to the VR display device to the VR system of the VR display device; The mobile computing device receives a second VR rendering of the VR environment from the VR system, wherein the second VR rendering is based on the viewing angle of the mobile computing device relative to the VR display device; Segmenting the first user from the one or more images; and In response to the acquisition of the one or more images, a mixed reality (MR) rendering of the first user in the VR environment is generated in real time at the mobile computing device, wherein the MR rendering of the first user is based on the synthesis of one or more segmented images of the first user with the second VR rendering.

14. The medium according to claim 13, wherein, When the software is executed, it can also operate as follows: The mobile computing device receives one or more anchor points in the real-world environment identified from the one or more images by the VR system.

15. The medium according to claim 14, wherein, When the software is executed, it can also operate as follows: The pose of the mobile computing device relative to the VR display device is determined based on one or more of the one or more anchor points.

16. The medium according to claim 15, wherein, When the software is executed, it can also operate as follows: The mobile computing device has been detected as having been moved; Based on one or more of the one or more anchor points, determine the updated pose of the mobile computing device relative to the VR display device; as well as In response to detecting that the mobile computing device has moved, an updated MR rendering of the first user in the VR environment is generated in real time at the mobile computing device, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device relative to the VR display device.

17. The medium according to claim 15 or 16, wherein, The pose of the mobile computing device relative to the VR display device is determined based on triangulation of one or more of the following: one or more anchor points; The mobile computing device includes one or more cameras; and the VR display device.

18. The medium according to claim 14 or 15, wherein, The one or more anchor points are fixed objects in the real-world environment.

19. A mobile computing device, comprising: One or more processors; and non-transitory memory coupled to the processor, the non-transitory memory including instructions executable by the processor, the processor being able to operate to: Using one or more cameras of the mobile computing device, one or more images of a first user wearing a virtual reality (VR) display device are captured in a real-world environment, wherein the VR display device is displaying a first VR rendering of the VR environment, and wherein the first VR rendering is from the viewpoint of the VR display device; Sends the pose of the mobile computing device relative to the VR display device to the VR system of the VR display device; The mobile computing device receives a second VR rendering of the VR environment from the VR system, wherein the second VR rendering is based on the viewing angle of the mobile computing device relative to the VR display device; Segmenting the first user from the one or more images; and In response to the acquisition of the one or more images, a mixed reality (MR) rendering of the first user in the VR environment is generated in real time at the mobile computing device, wherein the MR rendering of the first user is based on the synthesis of one or more segmented images of the first user with the second VR rendering.

Images