Detection and display of mixed 2d / 3d content

Abstract

To provide a system, a method and a device for sharing and displaying 2D and 3D content.SOLUTION: Provided are a device, a system, and a method that enable a first device to obtain two-dimensional (2D) content referenced by a webpage and a second device in communication with the first device to display 3D content referenced by the webpage. In an example, it is detected that webpage references the 3D content associated with the 2D content and detected that the second device that can display the 3D content referenced by the webpage is linked to or in communication with the first device. In one example, the second device receives input corresponding to a user selection to view the 3D content and, in response, displays a 3D environment including the 3D content.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to displaying two-dimensional (2D) and three-dimensional (3D) content, and more particularly to systems, methods, and devices for sharing and displaying 2D and 3D content.

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 715,331, filed Aug. 7, 2018, which is incorporated herein in its entirety. [Background technology]

[0003] Electronic devices have different capabilities for viewing and interacting with electronic content. For example, many desktop and laptop computers utilize a single flat screen display that is generally suitable for viewing 2D content, while the device remains stationary in a fixed position on a desk or other flat surface. Traditional web pages contain 2D content that can be easily viewed on such devices. However, such devices are limited with respect to viewing 3D content. In contrast, mobile devices, head-mounted devices (HMDs), and other such devices often have attributes and capabilities that are better suited for viewing 3D content. For example, 3D content (e.g., virtual objects) can be viewed on a mobile phone, and the orientation / position of the mobile phone can be used to change the view of the 3D content. As the user walks around and tilts the phone, the user can view the 3D content from different relative positions and angles. As another example, some devices, such as HMDs, include multiple screens that provide stereoscopic rendering capabilities for 3D content. This allows the user to better perceive depth when viewing 3D content. Existing technologies and systems do not adequately facilitate the distribution, viewing and sharing of 2D and 3D content to enable efficient and intuitive use of the different capabilities of these electronic devices. Summary of the Invention

[0004] Various implementations disclosed herein include devices, systems, and methods that allow a first device to display 2D content referenced by a content item (e.g., a web page) and a second device in communication with the first device to display 3D content referenced by the content item. In one embodiment, the first device displays the 2D content of the web page. The first device or the second device detects that the web page references 3D content associated with the 2D content. The second device can display the 3D content referenced by the web page and may be linked or otherwise in communication with the first device. The first or second device determines whether the second device should display the 3D content. The determination of whether to display the 3D content on the second device may be based on detecting that the devices are in communication with each other, detecting that the devices are in close proximity to each other, detecting that the first device is within an image range captured by the second device, detecting a user input, or detecting other relevant conditions or information. After determining that the 3D content should be displayed on the second device, the second device receives the 3D content referenced by the web page from the first device or another web page source and displays the 3D content. In one embodiment, the first device pre-downloads the 3D content as part of the web page content and automatically transmits the 3D content to the second device for display. In another embodiment, the second device requests the 3D content from the first device or from a web server that hosts the web page or the 3D content of the web page.

[0005] After receiving the 3D content, the second device displays the 3D content in various implementations. It may utilize one or more features in displaying the 3D content to display the 3D content not available on the first device. For example, the second device may display the 3D content from a user / device position-based viewpoint (e.g., changes as the device moves and tilts) using stereoscopic rendering or in a computer-generated reality (CGR) environment. The second device may display the 3D content using one or more of these display modes or other methods not available on the first device. In some implementations, the second device captures images using a camera and uses the 3D content and the captured images to build a CGR environment.

[0006] In some implementations, a user of the second device can simply and easily receive, view, and use the 3D content without having to move the 3D content to a specific storage location on the second device, identify an app that can experience the 3D content on the second device, identify an app to experience the 3D content, launch such an app, or import the received 3D content or add it to the CGR environment provided by such an app. The user experience of experiencing the 2D and 3D content of a website using two or more devices is efficient and intuitive.

[0007] The devices, systems, and methods disclosed herein allow for the display of 3D content corresponding to 2D web page content viewed on a first device. The devices, systems, and methods enhance a user's ability to use, share, and interact with 2D content combined with 3D content. For example, a first user using a first device can view a web page that includes 2D content corresponding to a sofa sold by an online retailer. The web page can include a 3D content indicator, such as, for example, a markup language statement indicating availability of 3D content that is part of the web page that corresponds to a 2D image of the sofa within the 2D content. The 2D content can include a visual indicator that the 3D content is available. The first user or a second user can then view the 3D content using a second device. In the sofa example, the second device is used to access the 3D content corresponding to the sofa, such as, for example, accessing a data object or file that includes a 3D model of the sofa or identifying a data storage address of a file or file that includes the 3D model of the sofa. In one embodiment, the first device initiates a communication to the second device that includes or provides access to the 3D content.

[0008] Based on receiving a communication from the first device, the second device allows the 3D content to be viewed or used. In one embodiment, the second device receives an input corresponding to a user selection to view the 3D content within the CGR environment. In response to detecting the input, the second device displays the 3D content as part of the CGR environment. Such a CGR environment can be provided by capturing images (e.g., video) using a camera and constructing the CGR environment using the 3D content and the captured images. In some implementations, the second device overlays the 3D content on the captured images.

[0009] In some implementations, the method is performed on a first device having one or more processors and a computer-readable storage medium, such as a desktop computer, a laptop computer, a tablet, etc. The method involves obtaining, by the first device, 2D content referenced by a webpage that also references 3D content. The method also involves a second device, such as a tablet, an HMD, a mobile phone, that can present the 3D content (e.g., can provide a view from a user / device position-based perspective using stereoscopic rendering or within a CGR environment). The second device receives and presents the 3D content referenced by the webpage. The method may include detecting that the webpage references 3D content that cannot be displayed by the first device or detecting other circumstances that trigger automatic communication between the first device and the second device or result in automatic display of the 3D content on the second device.

[0010] Thus, some implementations, as illustrated in the above examples and elsewhere herein, allow 2D content referenced by a web page on a first device and 3D content referenced by a web page on a second device to be viewed. These implementations improve the user's visual experience. For example, a single user with two devices (e.g., a laptop computer and an HMD) can browse a web page, and when the web page references 3D content of interest, he or she can turn on the HMD, access it automatically or through an intuitive user experience, and view the 3D content. In another example, multiple users can experience the 2D and 3D content referenced by a web page under a similarly convenient shared experience. For example, one user may view a web page displaying cars on a laptop computer, while another user may use the HMD to experience the 3D content of the car-related website.

[0011] According to some implementations, a non-transitory computer-readable storage medium has computer-executable instructions stored thereon to perform or cause the execution of any of the methods described herein. According to some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs, the one or more programs being stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause the execution of any of the methods described herein. The present disclosure may have a more detailed description by reference to aspects of several exemplary implementations, some of which are illustrated in the accompanying drawings, as can be understood by those skilled in the art. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an example system in which a first device displays 2D content referenced by a web page that also references 3D content, and a second device displays 3D content, according to some implementations. [Diagram 2] 2 illustrates the system of FIG. 1, where the second device is an HMD, according to some implementations. [Diagram 3] 1 illustrates an example system in which a first device displays 2D content, including 3D content, and a second device displays 3D content, according to some implementations. [Figure 4] 1 is a flowchart illustrating a flow of communication related to sharing of 3D content according to some implementations. [Diagram 5] FIG. 2 is a block diagram illustrating device components of an exemplary first device in accordance with some implementations. [Figure 6] FIG. 2 is a block diagram illustrating device components of an exemplary second device in accordance with some implementations. [Figure 7]1 is a flowchart illustrating a method for enabling a first device to obtain 2D content referenced by a web page and for enabling a second device in communication with the first device to display 3D content referenced by the web page. [Figure 8] 1 is a flowchart illustrating a method for displaying a CGR environment on a second device with 3D content received from a first device according to some implementations. [Figure 9] A flowchart illustrating a method for providing a view of a 3D environment on a first device that references 3D content referenced by a web page, the 3D content being obtained based on detecting a second device displaying 2D content referenced by the web page, in accordance with some implementations. [Figure 10] 1 is a flowchart illustrating a method for enabling a first device to update 2D or 3D content based on information associated with a current state of a 3D environment according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] According to common practice, the various features illustrated in the figures may not be drawn to scale. Thus, for clarity, dimensions of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all of the components of a given system, method, or device. Finally, like reference numbers may be used to denote like features throughout the specification and figures.

[0014] Numerous details are described to provide a thorough understanding of the exemplary implementations shown in the drawings. However, the drawings merely illustrate some exemplary aspects of the present disclosure and should not be considered as limiting. Those skilled in the art will understand that other effective aspects or variations do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the exemplary implementations described herein.

[0015] 1, an exemplary operating environment 5 for implementing aspects of the present invention is shown. In general, the operating environment 5 represents two devices, a first device 10 and a second device 15, that are involved in experiencing content items (e.g., web pages) that reference 2D and 3D content. While the examples herein utilize web pages as exemplary content items, the techniques disclosed herein apply to content items that reference 2D and 3D content that can be viewed by applications on one or more devices.

[0016] In this embodiment, the first device 10 and the second device 15 are connected via a link 20, which may include wired communications using, for example, a Universal Serial Bus (USB) cable / interface, a USB-C cable / interface, a THUNDERBOLT v1, v2 or v3 cable / interface, an IEEE 1394 cable / interface (e.g., FIREWIRE, i.LINK, LYNX), an IEEE 802.3x cable / interface (e.g., Ethernet), and the like, as well as other standards such as IEEE 803.11 transmissions (e.g., WiFi), IEEE 802.11 transmissions (e.g., wireless LAN), IEEE 802.16x transmissions (e.g., WiMAX), shortwave transmissions (e.g., BLUETOOTH), IEEE 802.15.4 transmissions (e.g., ZIGBEE), GSM transmissions, ECMA-340, and ISO / IEC 19001-10. The link 20 may be a wired or wireless link, including, but not limited to, wireless communication using, for example, IEEE 18092 (e.g., Near Field Communication (NFC)). The link 20 may be direct, i.e., there may be no intervening device or network node between the first device 10 and the second device 15. For example, the link 20 may include directly connecting the first device 10 to the second device 15 via a single cable connecting to each device or via Bluetooth communication between the first device 10 and the second device 15. The link 20 may be indirect, i.e., there may be one or more intervening devices or network nodes. For example, the link 20 may connect the first device 10 to the second device 15 via communications transmitted over the Internet. Furthermore, interactions between the devices, e.g., between the first device 10 and the second device 15, may be bidirectional, e.g., one device may send a communication to the other device to perform an action on a web page.

[0017] In the example of Figure 1, the first device 10 retrieves web page content 25 (e.g., on a web browser) that includes 2D content 30 (e.g., an image of a sofa) and an indicator 35 of 3D content 50 associated with the 2D content 30. For example, the 3D content 50 may include a 3D model of the sofa. The 3D content 50 is viewed on the second device 15 in the CGR environment. For example, the second device 20 can use a camera to capture a real-world scene 45 including the nearby environment, e.g., an end table and a wall photo, and display the captured view along with the 3D model of the sofa.

[0018] The devices, systems and methods allow the user to see an indicator 35 that associated 3D content 50 is available and to transmit the 3D content 50, e.g., 3D content representing a sofa, to the second device 15. A user of the second device 15 (e.g., the same or a different user as the user of the first device 10) can then experience the 3D content 50 on the display 40 of the second device 15. In this example, the 3D content 50 is displayed as part of the CGR environment.

[0019] A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's physical actions or representations thereof are tracked, and one or more characteristics of one or more virtual objects simulated within the CGR environment are adjusted accordingly to obey at least one law of physics. For example, a CGR system may detect a person's head rotation and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to the way such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to a characteristic(s) of a virtual object(s) in a CGR environment may be made in response to a representation of a physical action (e.g., a voice command).

[0020] A person may sense and / or interact with a CGR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. In another embodiment, the audio object enables audio transparency, selectively incorporating ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only the audio object.

[0021] Examples of CGRs include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment comprises virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0022] In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to incorporate sensory input from, or representations of, the physical environment in addition to including computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, mixed reality environments lie between, but do not include, a fully physical environment at one end and a virtual reality environment at the other end.

[0023] In some CGR environments, computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Some electronic systems for presenting CGR environments may also track position and / or orientation relative to the physical environment, allowing virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0024] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present the virtual objects on the transparent or semi-transparent display, so that the person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are a representation of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, the video of the physical environment displayed on the opaque display is referred to as "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and use those images in presenting the AR environment on the opaque display. As a further alternative, the system may have a projection system that projects virtual objects into the physical environment, for example as a hologram or onto a physical surface, so that a person using the system perceives the virtual objects as superimposed on the physical environment.

[0025] Augmented reality environments also refer to simulated environments in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, a system may transform one or more sensor images to provide a selected perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of the physical environment, so that the modified portion is represented but is not a photorealistic version of the original captured image. As a further example, the representation of the physical environment may be transformed by graphically removing or obscuring a portion of the physical environment.

[0026] An augmented virtual sensory (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but people with faces are realistically recreated from images of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that correspond to the position of the sun in the physical environment.

[0027] A wide variety of electronic systems exist that allow a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated viewing capabilities, windows with integrated viewing capabilities, displays formed as lenses designed to rest on a person's eye (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integral opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Rather than an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or semi-transparent display may comprise a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid-crystal on silicon (LCOS), laser scanning light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, a transparent or semi-transparent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects into a physical environment, for example as a hologram or onto a physical surface.

[0028] In the example of FIG. 1, the CGR environment combines the real world (e.g., via locally captured images of a real-world scene 45) with 3D content 50 (e.g., a file or model defining the 3D appearance of a sofa).

[0029] In this example, the second device 15 has received a data object corresponding to the 3D content 50 (e.g., the sofa in FIG. 1). The web page content 25 retrieved by the first device 10 may include an indicator 35 (e.g., a thumbnail image, an icon, a link, or a button) that represents or corresponds to the 3D content 50. In the case of an icon or thumbnail image, for example, the icon or thumbnail image may include a 2D image of the content of the 3D object from a single or standard perspective. Thumbnail images, icons, links, and buttons are examples of graphic features that may indicate that additional 3D content is available. Additionally or alternatively, the visual indicator of available 3D content may be configured to receive an input (e.g., a click, a touch, etc.) corresponding to a user selection to view the 3D content 50 on the display 40 of the second device 15. For example, the first device 10 may receive an input to select to display the 3D content 50 (e.g., the sofa) on the display 40 of the second device 15 by receiving a mouse click on the indicator 35 (e.g., a thumbnail image corresponding to the 3D content 50).

[0030] The second device 15 controls one or more cameras on the second device 15 to capture images of the real-world scene 45 and uses the 3D content 50 and the captured images to display the CGR content on the display 40 of the second device 15. In some implementations, the second device 15 includes a suitable combination of software, firmware, or hardware for providing a CGR environment. In other implementations, the second device 15 interacts with one or more other devices (local or remote) to provide the CGR environment on the display 40 of the second device 15. For example, the second device 15 may communicate with a separate controller (not shown) that performs some or all of the processing and storage required to provide the CGR environment. According to some implementations, the second device 15 displays the CGR environment based on the real-world scene 45 while the second device 15 is physically present within the real-world scene 45. In some implementations, the second device 15 (e.g., glasses, a head-up display, etc.) is configured to provide the CGR environment on the display 40 of the second device 15 using optical transmission of the real-world scene 45. In some implementations, the second device 15 is configured to provide a CGR environment on the display 40 of the second device 15 using video pass-through of the real-world scene 45. In some implementations, the second device 15 need not have a display screen and can project virtual elements onto the user's retina.

[0031] In some implementations, as shown in FIG. 1 , the second device 15 is a handheld electronic device (e.g., a smart phone or tablet) configured to display a CGR environment on a display 40 of the second device 15. In some implementations, the second device 15 is a chamber, enclosure, or room configured to present a CGR environment without a user wearing or holding the second device 15. In some implementations, the second device 15 can detect that it is capable of displaying 3D content 50 by detecting the type of the second device 15, i.e., by identifying that the type of the second device 15 is a type of device that can display 3D content 50. Furthermore, the second device 15 can detect that it is capable of displaying 3D content 50 by detecting an application stored on the second device 15 that can display 3D content 50.

[0032] In the example of FIG. 1, the second device 15 is configured to use images or other real-world information detected based on a camera or other sensor on the second device 15. In some implementations, the second device 15 uses at least a portion of one or more camera images captured by the camera to provide a CGR environment on the display 40 of the second device 15. In the example of FIG. 1, the display 40 of the second device 15 includes depictions of items captured by the camera of the second device 15 (e.g., a floor, a wall, a wall photo, two end tables, and two vases). The display 40 of the second device 15 also represents 3D content 50 (e.g., a sofa) that corresponds to the 2D content 30 displayed on the first device 10.

[0033] In some implementations, the second device 15 allows the user 55 to change the viewpoint or otherwise modify or interact with the 3D content 50. For example, the viewpoint may change as the user moves the second device 15. In some implementations, the second device 15 may be configured to receive user input that repositions the received 3D content 50 relative to the representation of real-world items of the real-world scene 45 shown on the display 40 of the second device 15 (e.g., repositioning a sofa relative to a wall picture or an end table).

[0034] In some implementations, the first device 10 receives or stores 3D content 50 corresponding to the sofa which is provided to the second device 15. In some implementations, the first device 10 then provides the 3D content 50 to the second device 15 so that the second device 15 can display the 3D content 50. In other implementations, the first device 10 does not receive or store the 3D content 50, and the second device 15 obtains the 3D content 50 from another source, such as a web server serving web pages.

[0035] In some implementations, multiple devices can identify 3D content in the same or different display modes. In one embodiment, the first device 10 provides a 2D display mode that does not include the 3D content 50, the second device 15 provides an MR display mode of the 3D content 50 combined with real-world content, and a third device (not shown) provides a VR display mode of the 3D content 50. In another embodiment, both the first device 10 and the second device 15 can provide an MR view of the 3D content 50. In one embodiment, the second device 15 displays a view of the 3D content 50 that includes various real-world content that is dependent on real-world content that surrounds or is otherwise observed by the respective device. One or both of the first device 10 and the second device 15 may be configured to use received data or images or other real-world information detected using their own cameras or other sensors. In some implementations, to provide the MR display mode, the second device 15 uses at least a portion of one or more camera images captured by a camera on the first device 10. In this example, the second device 15 provides a view using real-world information of the real world surrounding the first device 10 .

[0036] In some implementations, as shown in FIG. 2, the second device 15 is an HMD configured to be worn on the head of the user 55. Such an HMD can surround the field of view of the user 55. As shown in the example of FIG. 2, the HMD can include one or more screens or other displays configured to display 3D content within the field of view of the user 55. As shown in FIG. 2, in some implementations, the second device 15 is worn in a manner that the screens are positioned to display 3D content within the field of view of the user 55. Additionally, the HMD can include a housing (or enclosure) that houses various components. The housing can include (or can be coupled to) an eye pad located at a proximal end (of the user 55) of the housing. In some implementations, the THUNDERBOLT is a piece of plastic or rubber that holds the HMD comfortably and snugly in the appropriate position on the face of the user 55 (e.g., surrounding the eyes of the user 55). The housing can house a display that displays an image and emits light toward one or both eyes of the user 55.

[0037] FIG. 3 illustrates an exemplary system in which a first device 10 displays 2D content from a webpage that also references 3D content, and a second device 15 displays 3D content, according to some implementations. As illustrated in FIG. 3, a user 55 may wear a second device 15 (e.g., an HMD) while browsing the Internet using the first device 10, e.g., using both devices 10, 15 simultaneously. For example, the user 55 may wear the second device 15 while using the first device 10 to view webpage content 25 that includes 2D content 30 (e.g., a flower pot) and an indicator 35 of 3D content 50 associated with the 2D content 30. The indicator 35 may be a visible part of the 2D content 30 displayed on the first device 10, and thus may be visible as part of the image captured and displayed by the second device 15. Alternatively, the indicator 35 may not be included in the 2D content 30 on the first device 10, but may be displayed only on the second device 15. In one embodiment, the indicator 35 is displayed on the second device 15 based on the second device 15 detecting the first device 10 within an image captured by the second device 15 .

[0038] In some implementations, the second device 15 is configured to detect other devices shown within the captured image of the second device 15, determine whether such devices are in communication (e.g., linked) with the second device 15, establish communication (e.g., link) with the other devices, identify whether any of the other devices are displaying 2D content associated with the 3D content, obtain the 3D content, and display the 3D content.

[0039] Various techniques can be used by the second device 15 to detect the presence of the first device 10 in an image or view. For example, the second device 15 may detect that the first device 10 is in the current image / view of the second device 15. Such detection may include detecting a representation of the first device 10 based on the shape, color, or other attributes displayed in the pixels of the image or view of the real-world environment at the second device 15. Object detection techniques can involve the use of neural networks or other machine learning based techniques, for example, to detect the first device 10 in a captured image of the real-world environment around the second device 15. Such machine learning based techniques can be trained using sample images of devices (e.g., images labeled with shapes and other spatial characteristics of laptop computers, desktop computers, and mobile devices). In another embodiment, the second device 15 uses images of its environment to determine a 3D model of the real-world environment, and the 3D shape of the first device 10 is detected in the 3D model.

[0040] Detection of the first device 10 in an image / view at the second device 15 may include content displayed on a display of the first device 10. For example, the first device 10 may display 2D content of a web page on the display of the first device 10, and the second device 15 may capture an image of a real-world environment including the first device 10 and recognize the 2D content of the web page displayed on the display of the first device 10. In one embodiment, the display of the first device 10 includes text-based or graphic-based markers or codes (which may or may not be part of the 2D content of the web page) that are used by the second device 15 to detect the presence or spatial relationship of the first device 10 in the image / view of the real-world environment.

[0041] In some implementations, the second device 15 detects the spatial relationship between the first device 10 and the second device 15, for example, based on an image / view of the real-world environment. In some implementations, the second device 15 determines the position, orientation, size, or other attributes of the first device 10 in a 3D coordinate system associated with the real-world environment. The second device 15 can determine the spatial relationship between the first device 10 and the second device 10 by knowing the pose (e.g., position and orientation) of the second device 15 itself in that 3D coordinate system. In some implementations, the first device 10 and the second device 20 each include a sensor, allowing each device to track its own changes in position and orientation over time. Thus, once an initial spatial relationship between the devices is determined (e.g., based on image data), changes in the spatial relationship can be determined based on additional image-based determinations, devices tracking changes in their own position / orientation, or a combination of such tracking techniques.

[0042] In some implementations, the spatial relationship between the first device 10 and the second device 15 is used to present 3D content to the second device 15. In an example where the first device is a laptop computer and the second device is an HMD, the relative position, orientation, size, or other attributes of the laptop computer are used to present the 3D content on the HMD. As a specific example, the display of the 3D content on the HMD may depend on the spatial relationship (e.g., the 3D content may be displayed only when the laptop computer is the current image / view or within a certain threshold distance of the HMD).

[0043] In some implementations, the second device 15 captures images of the real-world environment surrounding the second device 15, including the first device 10 and the representation of the first device 10 in 3D content of a web page. The second device 15 displays these images with the additional 3D content of the web page having added position, orientation, size, and other attributes to complement the 2D content on the display of the first device 10 and seen in the images captured and displayed by the second device 15.

[0044] In one embodiment, based on the position, size, or other attributes of the first device 10, the second device 15 determines a position, size, or other attributes for displaying the 3D content via the second device 15. For example, the second device 15 may determine to display the 3D content in a manner that does not obstruct the line of sight of the user 55 so that the user 55 maintains an unobstructed view of the display of the first device 10. The second device 15 may determine to display the 3D content in a location adjacent to the first device 10 so that the user can easily understand the contextual relationship between the 2D content displayed on the first device 10 and the nearby 3D content displayed by the second device 15. The second device 15 may determine to display the 3D content such that the size of the 3D content corresponds to the size of the 2D content displayed on the first device 10, e.g., text in the 2D and 3D content has the same size. The second device 15 may determine to display a visual indicator connecting or otherwise graphically associating the 2D content displayed on the first device 10 with the 3D content displayed via the second device 15.

[0045] In another embodiment, the second device 15 detects the pose (e.g., position and orientation) of the first device 10 in one or more images captured by the second device 15 and uses this information to display the 3D content. For example, the orientation of the face of the display (e.g., monitor) of the first device 10 can be detected, and this face is used to position the 3D content aligned with the displayed 2D content.

[0046] In some implementations, the second device 15 is configured to detect or identify a position change of the first device 10 relative to the second device 15 and determine a second spatial relationship between the first device 10 and the second device 15. The second device 15 may then update the display of the 3D content based on the second spatial relationship. For example, if the user 55 shifts the position of the first device 10 relative to the second device 15, the second device 15 may change the display of the 3D content such that the user maintains an unobstructed view of the display of the first device 10.

[0047] In some implementations, the second device 15 automatically retrieves and displays the 3D content 50 without requiring user input. For example, the second device 15 displays the 3D content 50 when it detects that the second device 15 is worn by a user 55 and that the second device 15 is within a threshold distance from the first device 10. In other implementations, the second device 15 provides an indication that 3D content associated with 2D content on another device is available and receives input to initiate a search and display of the 3D content.

[0048] The second device 15 may be configured to determine display characteristics for displaying the 3D content. In some implementations, the second device determines a location to display the 3D content 50 relative to other displayed content, e.g., relative to real-world content or other virtual objects, based on predefined or user-specified display criteria. In one embodiment, based on recognition of the spatial relationship between the first device 10 and the second device 15, the 3D content 50 is displayed to the user 55 on a second device (e.g., an HMD) that is within the field of view of the user 55. In some implementations, the second device 15 determines a location to display the 3D content 50 by identifying a location where the virtual content can be displayed without obstructing a view of the web page content 25 displayed on the first device 10. For example, as shown in FIG. 3, the 3D content 50 is displayed on a table 57 next to the first device 10 without obstructing the view of the first device 10.

[0049] In some implementations, for example in the case of an interactive web page, changes made to the 3D content 50 are maintained or otherwise synchronized on both the first device 10 and the second device 15, either through the user interface of the first device 10 or the user interface of the second device 15. For example, a user of the first device 10 or the second device 15 can change the color of a sofa on either the first device 10 or the second device 15, and the change can be detected and displayed on the other device. The state of the web page maintained on either the first device 10 or the second device 15 may be shared between the devices 10 and 15 such that the devices 10, 15 maintain the same state and synchronization, e.g., display the same instance of the web page. As an example, an item highlighted on the first device 10 based on a user interaction is also highlighted on the second device 15, and any changes in the 3D content 50 cause a corresponding change in the 2D content 30, and vice versa.

[0050] 4 is a flow chart illustrating the flow of communications 60 involved in sharing 3D content 50 according to some implementations. In FIG. 4, a web server 70 provides a web page transmission 75 including 2D content 30 and 3D content 50 to a first device 10. The first device 10 communicates with a second device 15 via a link 20, and the second device 15 receives the 3D content 50. In some implementations, the first device 10 identifies which 3D content 50 to provide based on information associated with the second device 15. For example, the second device 15 may be a phone, tablet, or HMD, and the 3D content 50 may be provided in a form appropriate for viewing on such device based on the 3D content viewing capabilities available on such device.

[0051] Examples of objects that may be depicted in 2D and 3D content include, but are not limited to, a table, a floor, a wall, a desk, a book, a body of water, a mountain, a field, a vehicle, a counter, a human face, a human hand, human hair, another human body part, an entire human body, an animal or other living thing, clothing, paper, a magazine, a book, a vehicle, a machine or other man-made object, and any other 3D item or group of items that can be identified and represented. The 2D and 3D content may additionally or alternatively include created content that may or may not correspond to real world content, including, but not limited to, aliens, wizards, spaceships, unicorns, computer-generated graphics, and other such items.

[0052] 5 is a block diagram illustrating device components of the first device 10 according to some implementations. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, in some embodiments the first device 10 may include one or more processing units 102 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 106, one or more communications interfaces 108 (e.g., USB, FIREWIRE®, THUNDERBOLT®, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM®, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, SPI, I2C or similar type interfaces), one or more programming (e.g., I / O) interfaces 110, one or more displays 112, one or more interior or exterior facing image sensor systems 114, memory 120, and one or more communications buses 104 for interconnecting these and various other components.

[0053] In some implementations, the one or more communication buses 104 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 106 include at least one touch screen, soft keys, keyboard, virtual keyboard, buttons, knobs, joysticks, switches, dials, inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc. In some implementations, the movement, rotation, or position of the first device 10 detected by the one or more I / O devices and sensors 106 provides input to the first device 10.

[0054] In some implementations, the one or more displays 112 are configured to present a user interface. In some implementations, the one or more displays 112 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electro-mechanica system (MEMS), or similar display types. In some implementations, the one or more displays 112 correspond to a waveguide display, such as diffractive, reflective, polarizing, holographic, etc. In one embodiment, the first device 10 includes a single display. In another embodiment, the first device 10 includes a display for each eye. In some implementations, the one or more displays 112 can present MR or VR content.

[0055] In some implementations, the one or more image sensor systems 114 are configured to acquire image data corresponding to at least a portion of a scene local to the first device 10. The one or more image sensor systems 114 can include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, infrared cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 114 further include an illumination source that emits light, such as a flash.

[0056] The memory 120 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, the memory 120 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. The memory 120 optionally includes one or more storage devices located remotely from the one or more processing units 102. The memory 120 includes a non-transitory computer-readable storage medium. In some implementations, the memory 120 or the non-transitory computer-readable storage medium of the memory 120 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 130 and a web browser 132. The operating system 130 includes procedures for handling various basic system services and for performing hardware-dependent tasks. Furthermore, in some implementations, the functionality of the operating system 130 may also be located in or separate from a browser or browser plug-in.

[0057] In some implementations, the operating system 130 includes a web browser 132 for receiving or displaying web page content. Additionally, the inspection unit 134 may be used by the web browser 132 to inspect markup language or other web page content for tags or other indications that 3D content is available. In some implementations, the operating system 130 includes an indicator unit 136 that adds an indicator to the displayed content to indicate the presence of 3D content. Additionally, the operating system 130 may include an interface unit 138 for communicating with the second device 15, including transferring 3D content or exchanging other information. In some implementations, one or more of the inspection unit 134, the indicator unit 136, and the interface unit 138 may be part of the web browser 132 or may be separate. In some implementations, one or more of the inspection unit 134, the indicator unit 136, and the interface unit 138 are provided as part of an application separate from the operating system 130 or are implemented on a separate device local to or remote from the first device 10.

[0058] FIG. 5 is further intended as a functional description of various features present in a particular implementation, as opposed to a structural overview of the implementations described herein. As will be appreciated by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 5 can be implemented within a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of specific functions and how functions are allocated among them will vary from implementation to implementation, and in some implementations will depend in part on the particular combination of hardware, software, or firmware selected for a particular implementation.

[0059] 6 is a block diagram illustrating device components of second device 15, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, in some embodiments, second device 15 may include one or more processing units 152 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 156, one or more communications interfaces 158 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, SPI, I2C or similar type interfaces), one or more programming (e.g., I / O) interfaces 160, one or more displays 162, one or more internally or externally facing image sensor systems 164, memory 170, and one or more communications buses 154 for interconnecting these and various other components.

[0060] In some implementations, the one or more communication buses 154 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 156 include at least one touch screen, soft keys, keyboard, virtual keyboard, buttons, knobs, joysticks, switches, dials, inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc. In some implementations, the movement, rotation, or position of the second device 15 detected by the one or more I / O devices and sensors 156 provides input to the second device 15.

[0061] In some implementations, the one or more displays 162 are configured to present a view of a 3D model being viewed or edited on another device. In some implementations, the one or more displays 162 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), LCoS, organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, or similar display types. In some implementations, the one or more displays 162 correspond to a waveguide display, such as diffractive, reflective, polarizing, holographic, etc. In one embodiment, the second device 15 includes a single display. In another embodiment, the second device 15 includes a display for each eye. In some implementations, the one or more displays 162 can present CGR content.

[0062] In some implementations, the one or more image sensor systems 164 are configured to acquire image data corresponding to at least a portion of a scene local to the second device 15. The one or more image sensor systems 164 can include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, infrared cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 164 further include an illumination source that emits light, such as a flash.

[0063] Memory 170 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, memory 170 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 170 optionally includes one or more storage devices located remotely from one or more processing units 152. Memory 170 includes a non-transitory computer-readable storage medium. In some implementations, memory 170 or the non-transitory computer-readable storage medium of memory 170 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 180 and a web browser 182. Operating system 180 includes procedures for handling various basic system services and for performing hardware-dependent tasks. Furthermore, in some implementations, the functionality of operating system 180 may also be located in a browser, browser plug-in, or separate from the browser.

[0064] In some implementations, the operating system 180 includes a web browser 182 for receiving or displaying web page content. Additionally, an inspection unit 184 may be used by the web browser 182 to inspect markup language or other web page content for tags or other indications that 3D content is available. In implementations, the operating system 130 includes an interface unit 186 for receiving information, such as 3D content, from the first device 10 or exchanging information related to receiving or displaying 3D content. Additionally, the operating system 130 may include a CGR content creation unit 188 for generating CGR content or a view of a CGR environment using multiple sources, for example, real-world data from a camera and data from the 3D content received from the first device 10. In some implementations, one or more of the inspection unit 184, the interface unit 186, and the CGR content creation unit 188 are part of or separate from the web browser 182. In some implementations, one or more of the inspection unit 184, interface unit 186, and CGR content creation unit 188 are provided as part of an application separate from the operating system 180 or are implemented on a separate device local to the first device 15 or remote from the first device 15.

[0065] FIG. 6 is further intended as a functional description of the various features present in a particular implementation, as opposed to a structural overview of the implementations described herein. As will be appreciated by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 6 can be implemented within a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of specific functions and how functions are allocated among them will vary from implementation to implementation, and in some implementations will depend in part on the particular combination of hardware, software, or firmware selected for a particular implementation.

[0066] FIG. 7 is a flow chart illustrating a method 200 for enabling a first device to present 2D content referenced by a web page and for enabling a second device in communication with the first device to display 3D content referenced by the web page. In some implementations, the method 200 is performed by one or more devices, such as the first device 10 of FIGS. 1-5. The method 200 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. In some implementations, the method 200 is performed by processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 200 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0067] At block 210, method 200 retrieves 2D content referenced by a web page for display on a display of the first device. In some implementations, method 200 interprets HTML tags or JavaScript to display the 2D content, such as titles, text, images, etc. For example, method 200 may retrieve tags (e.g., ") associated with image elements. ") can be processed to obtain 2D images embedded within a web page. In implementations, the 2D content includes a 2D view of an object. In other implementations, the 2D content includes content that is not formatted for 3D viewing.

[0068] At block 220, the method 200 detects that the web page references 3D content associated with the 2D content. In some implementations, this includes inspecting the markup language, for example via a Mixed Reality Markup Language (MRML) protocol or an application program interface (API), or inspecting other aspects of the web page to identify markup language statements or other indicators of the 3D content. Thus, the markup language file may include one or more statements that correspond to the 3D content or that otherwise indicate the availability of the 3D content associated with the web page. The markup language indicator may be associated with a visual indicator that is displayed as part of the 2D content referenced by the web page, for example in the form of a thumbnail image, link, icon, or button that indicates the availability of the 3D content.

[0069] At block 230, the method 200 detects that the second device is communicating with the first device. The communication may be detected based on a link between the first and second devices. In some implementations, detecting a link between the first and second devices involves detecting that the second device is connected to the first device via a USB or other cable. In some implementations, detecting a link between the first and second devices involves detecting that a wireless communication channel has been established between the devices. In some implementations, a wireless or wired connection or other link between the first and second devices is automatically detected by the first or second device. In some implementations, this may additionally or alternatively involve recognizing that the connected device is a particular device, type of device, or device associated with a particular user, owner, or account.

[0070] If a connection does not exist, the connection may be established. In some implementations, a message is presented to the user of the first device identifying an option to establish the connection. For example, a text, graphical or audio message is presented asking if the user would like to establish the connection. In other embodiments, the connection is automatically established based on both the first device and the second device being associated with the same account or user profile. In another embodiment, the connection is automatically established based on detecting that 2D content on one device is associated with 3D content that may be displayed on another device. In another embodiment, the connection is automatically established based on detecting a user's intent to share 3D content with each other's devices. In another embodiment, the connection is automatically established based on detecting or estimating the proximity between the devices, e.g., the devices are within a threshold distance of each other. In some implementations, the connection is established via an operating system (OS) level service call. The connection may also invoke or access shared memory on the second device. The daemon can map this shared memory into the daemon's processing space, so that the daemon becomes a conduit for the first device to seamlessly link to the second device to provide a 3D content viewing / editing experience.

[0071] At block 240, the method 200 provides the 3D content referenced by the web page to the second device. In some implementations, the first device sends a communication to the second device to automatically launch a 3D viewing interface on the second device.

[0072] At block 250, method 200 receives a communication corresponding to an input received at the second device. For example, a user of the second device may interact with the second device via, for example, typing, waving, selecting, etc., and the input at the second device may then be communicated to the first device.

[0073] At block 260, the method 200 applies the input received at the second device to the 2D content referenced by the web page on the display of the first device. In some implementations, the 3D content may additionally or alternatively be refreshed, updated, selected, or otherwise modified based on the input received from the second device. The method may then provide the updated 3D content to the second device at block 240.

[0074] FIG. 8 is a flow chart illustrating a method 300 for displaying a CGR environment on a second device with 3D content received from a first device, according to some implementations. In some implementations, the method 300 is performed by one or more devices (e.g., the second device 15 of FIGS. 1-4 and 6). The method 300 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. The method 300 can be performed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some implementations, the method 300 is performed by processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0075] At block 310, the method 300 receives 3D content corresponding to the 2D content. The 3D content is received from a separate device, such as from a device used by another user to transmit the 3D content or from a web page server device. In some implementations, the 3D content is received based on user input. For example, the second device may display an indication that the 3D content is available via a selectable option and detect a user selection of that option.

[0076] At block 320, the method 300 receives input corresponding to a user selection to view the 3D content. In some implementations, the method 300 presents an icon, graphic, or text representing the 3D content on a user interface and receives input corresponding to a user selection of the icon, graphic, or text. In some implementations, the user interface displays an icon, graphic, or text representing the received 3D content and also displays an indicator (e.g., text, graphic, etc.) indicating that selection of the icon, graphic, or text launches a viewer to view the received 3D content.

[0077] Blocks 330 and 340 are executed in response to detecting an input to view 3D content. In block 330, method 300 captures an image using a camera. The image captured by the camera depicts real-world content in a scene of the device that may be included in the CGR environment.

[0078] At block 340, the method 300 constructs a CGR environment using the 3D content and the captured images. In some implementations, constructing the CGR environment is performed by overlaying the 3D content on the captured images. In some implementations, constructing the CGR environment is performed by detecting one or more planes in the real-world content shown in the images and positioning the 3D content in the CGR environment based on the detected planes. For example, at a first time, an image of the received 3D content can be placed on the most recently captured image of the captured images, and at a second time after capturing additional captured images, an image (same or different) of the received 3D content can be placed on the newly captured image, etc. In some implementations, constructing the CGR environment is performed by building a 3D model corresponding to some or all of the real-world content shown in the images, adding the 3D content to the model, and then creating an image of the synthetic content from a particular viewpoint.

[0079] A user can interact with and change the CGR environment. In one embodiment, the user changes the position or rotation of the 3D content of a web page. For example, the user can move a depiction of a sofa to another location in the scene. In some implementations, the change involves a change in viewpoint. For example, this can involve receiving a device movement or rotation and updating the display of the CGR environment based on the movement. As the user moves their head, moves around the room, jumps around, etc., the user's viewpoint changes in the CGR environment. However, the position and rotation of the 3D content relative to real-world objects shown in the CGR environment can remain constant, e.g., the sofa stays in its position relative to the floor and other real-world objects shown in the scene. To maintain a constant relative position, flat surfaces or features are identified in the image and used to maintain the relative position of the received 3D content.

[0080] FIG. 9 is a flowchart illustrating a method 400 for providing a view of a 3D environment on a first device including 3D content referenced by a web page, the 3D content being obtained based on detecting a second device displaying 2D content referenced by the web page, according to some implementations. In some implementations, the method 400 is performed by one or more devices (e.g., the second device 15 of FIGS. 1-4 and 6). The method 400 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. The method 400 can be performed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some implementations, the method 400 is performed by processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0081] At block 410, the method 400 acquires image data representative of the physical environment using an image sensor. The image data captured by the image sensor may include real-world content that includes the second device.

[0082] At block 420, method 400 detects that a second device is depicted in the image data. Method 400 can further detect that the second device displays 2D content referenced by the web page without 3D content referenced by the web page. In some implementations, method 400 detects an icon, graphic, or text representing the 3D content. In other implementations, method 400 detects a markup statement in the web page's definition HTML or other markup language file that corresponds to the 3D content.

[0083] At block 430, the method 400 obtains 3D content referenced by the web page based on detecting that the display of the second device includes 2D content referenced by the web page without 3D content referenced by the web page. In some implementations, the 3D content is obtained from a second device that displays the 2D content. In some implementations, the 3D content is obtained from a web address associated with the web page displayed on the second device.

[0084] At block 440 of method 400, the first device provides a view of the 3D environment including the 3D content referenced by the web page for display. Method 400 may construct the view of the 3D environment using the 3D content and the image data. In some implementations, the view of the 3D environment is constructed using real-time or pre-recorded image data. In some implementations, constructing the view of the 3D environment is performed by overlaying the 3D content on the image data. In some implementations, constructing the view of the 3D environment is performed by building a 3D model that corresponds to some or all of the real-world content depicted in the image data, adding the 3D content to the model, and then creating an image of the synthetic content from a particular viewpoint.

[0085] At block 450 of method 400, the first device receives input (e.g., a typing, a gesture, a selection, etc.) from a user. At block 460 of method 400, the first device communicates the received input to the second device so that the second device can apply the input to the webpage content displayed at the second device. In some implementations, the 3D content may additionally or alternatively be refreshed, updated, selected, or otherwise modified based on the input. The first device may receive a response to the input from the second device, e.g., a notification that the input has been received or applied, and as at block 430, the first device may repeatedly obtain 3D content referenced by the webpage from the second device.

[0086] FIG. 10 is a flow chart illustrating a method 500 for enabling a first device to update 2D or 3D content based on information associated with a current state of a 3D environment. In some implementations, the method 500 is performed by three or more devices, such as the first device 10 and the second device 15 of FIGS. 1-5, and a third device. The method 500 can be performed by a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. In some implementations, the method 500 is performed by processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 500 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0087] At block 510, the method 500 provides the 3D content referenced by the web page to the second device and the third device. In some implementations, the first device sends a communication to the second device or the third device to automatically launch the 3D viewing interface. In some implementations, the third device does not include a viewing interface, e.g., a web-enabled camera.

[0088] At block 520, the method 500 grants modification rights to the third device. In some implementations, the third device may launch the 3D viewing interface in a read-only mode, i.e., the third device may only be allowed to view the 3D content or comments on the 3D content using text messages or emojis, and the third device may not be allowed to modify the state of the 3D content or the 3D environment. The first device or the second device may then grant modification rights to the third device via their respective user interfaces. Thus, at block 520, the method 500 grants modification rights to the third device.

[0089] At block 530, method 500 obtains information associated with the current state of the 3D environment from the third device. In some embodiments, the information associated with the 3D environment includes images captured by an integrated camera of the third device that show real-world content in the scene of the third device, which images may be included in the CGR environment or used to create the CGR environment.

[0090] At block 540, the method 500 updates the 2D or 3D content referenced by the webpage based on information associated with the current state of the 3D environment, e.g., an image or model of the real-world environment. In some implementations, the 2D or 3D content is refreshed, updated, selected, or otherwise modified based on information received from the third device. In some implementations, the first device provides a view of webpage content that includes, incorporates, or uses information associated with the current state of the 3D environment. In one embodiment, the first device is a laptop computer without a built-in camera and the connected third device is a mobile phone with a built-in camera. The mobile phone provides an image of the 3D environment to the laptop computer, which provides a view of the webpage content incorporating the 3D environment. In another embodiment, the first device is a desktop computer and the third device is a web-enabled camera. The camera provides an image of the 3D environment to the desktop computer. The desktop computer includes the image received from the camera in the background of the 2D content referenced by the webpage that the desktop computer is displaying.

[0091] At block 550, the method 500 shares the updated content referenced by the web page with the second device or the third device. For example, the 3D environment information received at the first device from the third device can be used to update the web page content delivered for viewing on the second device. In one embodiment, the first device is a laptop computer, the second device is a mobile phone, and the third device is an HMD. In this embodiment, the HMD captures images of the 3D environment around the HMD and shares those images for incorporation into the web page experience on the first device, the second device, or both.

[0092] Numerous specific details are described herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatus, or systems that are known by those skilled in the art have not been described in detail so as not to obscure the claimed subject matter.

[0093] Unless otherwise indicated, throughout the description herein, the use of terms such as "processing," "computing," "calculating," "determining," and "identifying" will be understood to refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing device(s) that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register or other information storage device, transmission device, or display device of a computing platform.

[0094] The system(s) discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provides a result conditioned by one or more inputs. Suitable computing devices include general purpose microprocessor-based computer systems that access stored software to program or configure the computing system, ranging from general purpose computing devices to special purpose computing devices that implement one or more implementations of the present subject matter. The teachings contained herein may be implemented in software and used to program or configure a computing device using any suitable programming, scripting or other type of language or combination of languages.

[0095] Implementations of the methods disclosed herein may be performed in operation of such a computing device. The order of the blocks presented in the above examples may, for example, be rearranged, combined, or divided into sub-blocks. Certain blocks or processes may be performed in parallel.

[0096] The use of "compatible with" or "configured to" herein means open-ended and inclusive language that does not exclude devices that are adapted or configured to perform additional tasks or steps. Additionally, the use of "based on" means open-ended and inclusive in that a process, step, calculation, or other action that is "based on" one or more recited conditions or values ​​may in fact be based on additional conditions or values ​​beyond the recited conditions or values. The headings, lists, and numbering contained herein are for ease of description and are not intended to be limiting.

[0097] In this specification, terms such as "first", "second", etc. may be used to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first node can be referred to as a second node, and a second node can be referred to as a first node, without changing the meaning of the description, so long as all occurrences of a "first node" are consistently renamed, and all occurrences of a "second node" are consistently renamed. Although a first node and a second node are both nodes, they are not the same node.

[0098] The terms used herein are for the purpose of describing particular embodiments and are not intended to limit the scope of the claims. As used in the description of the illustrated implementations and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Also, as used herein, the term "and / or" is understood to refer to and include any and all possible combinations of one or more of the associated listed items. It is further understood that the terms "comprises" or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

[0099] As used herein, the term "if" may be interpreted, depending on the context, to mean "when" or "upon" or "upon determining" or "in accordance with determining" or "in response to detecting" that the aforementioned condition is true. Similarly, the phrases "if [the aforementioned condition is determined to be true]" or "if [the aforementioned condition is true]" or "when [the aforementioned condition is true]" may be interpreted to mean "upon determining" or "in response to determining" or "in accordance with determining" or "upon detecting" or "in response to detecting" that the aforementioned condition is true.

[0100] The foregoing description and summary of the invention should be understood in all respects as illustrative and exemplary, but not restrictive, and it should be understood that the scope of the invention disclosed herein should be determined not only from the detailed description of the exemplary implementations, but in accordance with the full breadth permitted by the patent laws. It should be understood that the implementations shown and described herein are merely illustrative of the principles of the invention, and that various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A non-transitory computer-readable storage medium storing program instructions that are computer-executable to perform operations, comprising: The operation, Obtaining two-dimensional (2D) content referenced by the content item for display via the application on a display of the first device; detecting that the content item references three-dimensional (3D) content associated with the 2D content; detecting that a second device is in communication with the first device, the second device being capable of displaying the 3D content referenced by the content item; identifying the 3D content referenced by the content item to the second device; A non-transitory computer readable storage medium comprising:

2. 2. The non-transitory computer-readable storage medium of claim 1, wherein detecting that the content item is a web page and that the web page references the 3D content includes identifying markup language statements within the web page that correspond to the 3D content.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the 3D content is a virtual object having stereoscopic display information.

4. The operation includes:

10. The non-transitory computer-readable storage medium of claim 1, further comprising: in response to detecting that the second device is in communication with the first device, displaying an affordance indicating availability of the 3D content.

5. Detecting that the second device is in communication with the first device includes: Detecting that the first device and the second device are both associated with the same account; detecting that the second device is within a threshold proximity of the first device; or Detecting that the first device is within a camera field of view of the second device; 2. The non-transitory computer-readable storage medium of claim 1 , comprising:

6. The operation includes: determining a first spatial relationship between the first device and the second device; displaying the 3D content on the display of the detected second device based on the first spatial relationship; and The non-transitory computer-readable storage medium of claim 1 , further comprising:

7. The operation includes: Identifying a change in position of the first device relative to the second device; determining a second spatial relationship between the first device and the second device; updating the 3D content on the display of the detected second device based on the second spatial relationship; and The non-transitory computer-readable storage medium of claim 6 , further comprising:

8. The operation includes: Detecting the type of the second device; or detecting an application on the second device capable of displaying the 3D content; 2. The non-transitory computer-readable storage medium of claim 1, further comprising detecting that the second device is capable of displaying the 3D content referenced by the content item based on:

9. The operation includes: sharing data associated with changes to the 2D content on the display of the first device; identifying the 3D content referenced by the content item of the detected second device based on the changes to the 2D content; The non-transitory computer-readable storage medium of claim 1 , further comprising:

10. A first device comprising a processor, a computer readable storage medium, an image sensor and a display, acquiring image data representative of a physical environment using the image sensor; detecting a second device depicted in the image data, the second device displaying two-dimensional (2D) content referenced by the content item without three-dimensional (3D) content referenced by the content item; obtaining the 3D content referenced by the content item based on the detection; and providing a view of a 3D environment including the 3D content referenced by the content item for display on the display; A method comprising:

11. The method of claim 10 , wherein the content item is a web page.

12. The method of claim 10 , further comprising detecting, in the image data, a 3D content indicator displayed as part of the content item displayed by an application on the first device.

13. Based on detecting the 3D content indicator, determining to obtain the 3D content from the first device; or providing a user interface option for displaying the 3D content via the first device; The method of claim 12 further comprising:

14. The method of claim 10 , further comprising detecting that the second device is in communication with the first device.

15. The method of claim 10 , wherein the 3D content is obtained from the first device.

16. The view of the 3D environment may include: a representation of the second device displaying the 2D content referenced by the content item; and the 3D content referenced by the content item adjacent the representation of the second device; and The method of claim 10, comprising:

17. 17. The method of claim 16, wherein an association indicator in the view of the 3D environment indicates an association between the 3D content referenced by the content item and the representation of the second device displaying the 2D content referenced by the content item.

18. an electronic device having a display and an image sensor; A processor; A system comprising: The computer-readable storage medium includes instructions that, when executed by the processor, cause the system to perform operations; The operation, Obtaining two-dimensional (2D) content referenced by the web page for display on a display of a first device; Detecting that the web page references three-dimensional (3D) content associated with the 2D content; Detecting that a second device is in communication with the first device, the second device being capable of displaying the 3D content referenced by the web page; and Identifying the 3D content referenced by the web page on the second device; Including, the system.

19. The system of claim 18 , wherein the 3D content is a virtual object having stereoscopic display information.

20. The operation, Detecting that the second device is in communication with the first device; In response to detecting that the second device is in communication with the first device, displaying an affordance indicating availability of the 3D content; The system of claim 18 further comprising:

Images