Virtual anchoring system and method for augmented reality

Abstract

The invention relates to a virtual anchoring system and method for augmented reality. Particular implementations of the subject technology provide virtual anchoring for augmented reality (XR) display devices. A device may generate an XR environment that includes computer generated (CG) content for display relative to various physical objects in a physical environment. To locate CG content, an XR application may request a physical anchoring object to which CG content may be anchored. Where a physical anchor object is unavailable in a physical environment, a virtual anchor to which CG content may be anchored and / or a virtual anchor object corresponding to the physical anchor object may be provided.

Description

[0001] Division Statement

[0002] This application is a divisional application of the Chinese invention patent application with an application date of April 15, 2021, an invention name of “Virtual Anchoring System and Method for Extended Reality”, and an application number of 202110407376.4.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 011,973, filed on April 17, 2020, entitled “Virtual Anchoring Systems And Methods For Computer-Generated Reality,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0005] This specification generally relates to extended reality environments. Background Art

[0006] Augmented reality technology aims to bridge the gap between virtual and physical environments by providing an augmented physical environment that is enhanced with electronic information. Thus, the electronic information appears to be part of the user's perceived physical environment. However, determining where to place the electronic information within the physical environment can be challenging. Summary of the invention

[0007] Some embodiments of the present application provide a device, including a processor and a computer-readable medium, the computer-readable medium having instructions stored thereon, which, when executed, cause the processor to: obtain environmental information of a physical environment of the electronic device through a system program of the electronic device; receive a request for a physical anchor object in the physical environment through the system program and from an application running on the electronic device; determine through the system program and based on the environmental information that the physical anchor object is unavailable in the physical environment; generate a virtual anchor point having a position in the physical environment through the system program; and render virtual content through the system program of the electronic device to display it at the position in the physical environment using the virtual anchor point.

[0008] Some embodiments of the present application provide a method, including: obtaining environmental information of a physical environment of an electronic device through a system program of the electronic device; receiving a request for a physical anchor object in the physical environment through the system program and from an application running on the electronic device; determining through the system program and based on the environmental information that the physical anchor object is unavailable in the physical environment; generating through the system program a virtual anchor point having a position in the physical environment; and rendering through the system program of the electronic device virtual content to display at the position in the physical environment using the virtual anchor point.

[0009] Some embodiments of the present application provide a non-transitory computer-readable medium comprising instructions, which, when executed by a computing device, cause the computing device to: obtain environmental information of a physical environment of the electronic device through a system program of the electronic device; receive a request for a physical anchor object in the physical environment through the system program and from an application running on the electronic device; determine through the system program and based on the environmental information that the physical anchor object is unavailable in the physical environment; generate a virtual anchor point having a location in the physical environment through the system program; and render virtual content through the system program of the electronic device to display it at the location in the physical environment using the virtual anchor point. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0011] Figure 1 An exemplary system architecture including various electronic devices that may implement the subject system is shown according to one or more implementations.

[0012] Figure 2 An exemplary software architecture is shown in accordance with one or more implementations of the subject technology.

[0013] Figure 3 An example of a physical environment of an electronic device according to an implementation of the subject technology is shown.

[0014] Figure 4 An example of anchoring virtual content to physical object anchors in a physical environment is shown in accordance with an implementation of the subject technology.

[0015] Figure 5 Examples of anchoring virtual content to physical object anchors and virtual anchors in an XR environment are shown in accordance with specific implementations of the subject technology.

[0016] Figure 6 Examples of anchoring virtual content to physical object anchors and virtual object anchors in an XR environment are shown in accordance with specific implementations of the subject technology.

[0017] Figure 7 An example of a physical environment of an electronic device without sufficient space for a virtual anchor object is shown in accordance with an implementation of the subject technology.

[0018] Figure 8 An example of a virtual portal in a physical environment of an electronic device generating sufficient virtual space for a virtual anchor object in accordance with an implementation of the subject technology is shown.

[0019] Fig. 9 A flow chart illustrating exemplary operations that may be performed to provide a virtual anchor in accordance with an implementation of the subject technology.

[0020] Fig.10 An electronic system is shown with which one or more implementations of the subject technology may be implemented. DETAILED DESCRIPTION

[0021] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0022] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner that complies with at least one physical law. For example, the XR system may detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) that presents an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust features of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0023] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0024] Specific implementations of the subject technology described herein provide an XR system that provides a virtual anchor at a location in a physical environment when a requested physical anchor object is not available. For example, an XR application may request from a device a physical table on which to display a virtual object. The XR system may determine that the requested physical table is not available in the physical environment, and may generate a virtual anchor and / or a virtual version of the requested physical anchor object (e.g., a virtual anchor object) at a fixed location in the physical environment. If the physical anchor object is available, the virtual content may then be rendered and displayed at the fixed location, in space at the location of the virtual anchor, or on the virtual anchor object, as if displayed on the requested physical anchor object.

[0025] Figure 1An exemplary system architecture 100 including various electronic devices that may implement the subject system according to one or more implementations is shown. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0026] System architecture 100 includes electronic device 105, electronic device 110, electronic device 115, and server 120. For the purpose of explanation, system architecture 100 is shown in FIG. Figure 1 1 is shown as including electronic device 105, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0027] The electronic device 105 may be a smart phone, a tablet computer, or a head-mounted portable system (e.g., a head-mounted display device that can be worn by a user) that includes a display system capable of presenting a visualization of an extended reality environment to a user. The electronic device 105 may be powered by a battery and / or any other power source. In one example, the display system of the electronic device 105 provides a stereoscopic presentation of the extended reality environment to the user, enabling a three-dimensional visual display of a particular scene rendering to be provided. In one or more specific implementations, instead of or in addition to using the electronic device 105 to access the extended reality environment, the user may use a handheld electronic device 104, such as a tablet computer, a watch, a mobile device, etc.

[0028] The electronic device 105 may include one or more cameras, such as a camera 150 (e.g., a visible light camera, an infrared camera, etc.). In addition, the electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near field communication sensors, etc. In addition, the electronic device 105 may include hardware elements that can receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for initiating recording within a given extended reality environment. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc. The electronic device 105 may also detect and / or classify physical objects in the physical environment of the electronic device 105.

[0029] The electronic device 105 may be communicatively coupled to a base device, such as the electronic device 110 and / or the electronic device 115. Generally, such base devices may include more computing resources and / or available power than the electronic device 105. In one example, the electronic device 105 may operate in various modes. For example, the electronic device 105 may operate in a standalone mode independent of any base device. When the electronic device 105 operates in a standalone mode, the number of input modalities may be constrained by the power limitations of the electronic device 105, such as the available battery power of the device. In response to the power limitations, the electronic device 105 may deactivate certain sensors within the device itself to preserve battery power.

[0030] The electronic device 105 may also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection) to work in conjunction with a given base device. The electronic device 105 may also operate in a connection mode in which the electronic device 105 is physically connected to the base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device charges the electronic device 105 when physically connected).

[0031] When the electronic device 105 is operated in a wireless connection mode or a connection mode, at least a portion of processing user input and / or rendering an extended reality environment can be offloaded to the base device, thereby reducing the processing burden on the electronic device 105. For example, in one specific implementation, the electronic device 105 works in conjunction with the electronic device 110 or the electronic device 115 to generate an extended reality environment, which includes physical objects and / or virtual objects that implement different forms of interaction (e.g., visual, auditory and / or physical or tactile interaction) between the user and the extended reality environment in real time. In one example, the electronic device 105 provides a rendering of a scene corresponding to the extended reality environment, which can be perceived by the user and interacted in real time. In addition, as part of presenting the rendered scene, the electronic device 105 may provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and current system workload.

[0032] The electronic device 105 may also detect an event that has occurred within the scene of the extended reality environment. Examples of such events include detecting the presence of a particular person, entity, or object in the scene. The detected physical objects may be classified by the electronic device 105, the electronic device 110, and / or the electronic device 115, and the location, position, size, dimensions, shape, and / or other characteristics of the physical objects may be used to provide physical anchor objects to the XR application to generate virtual content for display within the XR environment.

[0033] It should also be understood that electronic device 110 and / or electronic device 115 may also work in conjunction with electronic device 105 or independently of electronic device 105 to generate such an extended reality environment.

[0034] Network 106 may communicatively (directly or indirectly) couple, for example, electronic device 105, electronic device 110, and / or electronic device 115 with server 120. In one or more implementations, network 106 may be an interconnected network that may include the Internet or devices communicatively coupled to the Internet.

[0035] The electronic device 110 may include a touch screen and may be, for example, a smart phone including a touch screen, a portable computing device such as a laptop computer including a touch screen, a peripheral device including a touch screen (e.g., a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (such as a watch, wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device with a touchpad. In one or more specific implementations, the electronic device 110 may not include a touch screen, but may support touch screen-like gestures, such as in an extended reality environment. In one or more specific implementations, the electronic device 110 may include a touchpad. In Figure 1In the embodiment, by way of example, the electronic device 110 is depicted as a mobile smart phone device having a touch screen. In one or more specific implementations, the electronic device 110, the handheld electronic device 104 and / or the electronic device 105 may be and / or may include the following with respect to Fig.10 All or part of the electronic system discussed. In one or more specific implementations, the electronic device 110 can be another device, such as an Internet Protocol (IP) camera, a tablet computer, or a peripheral device such as an electronic stylus.

[0036] The electronic device 115 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera, a headset), a tablet device, a wearable device such as a watch, a wristband, etc. Figure 1 In the embodiment, the electronic device 115 is depicted as a desktop computer by way of example. The electronic device 115 may be and / or may include the following with respect to Fig.10 All or part of the electronic system in question.

[0037] The server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors and other dedicated or custom processors) for rendering and generating content such as graphics, images, video, audio, and multimedia files of an extended reality environment. In one specific implementation, the server 120 may be used as a cloud storage server that stores any of the aforementioned extended reality content generated by the above-mentioned devices and / or server 120.

[0038] Figure 2 1 shows an exemplary architecture that may be implemented on electronic device 105, handheld electronic device 114, electronic device 115, and / or server 120 in accordance with one or more specific implementations of the subject technology. For the purpose of explanation, Figure 2 Part of the architecture is described by Figure 1 The electronic device 105 of the present invention is implemented, such as by a processor and / or memory of the electronic device 105; however, appropriate parts of the architecture may be implemented by any other electronic device, including the electronic device 110, the electronic device 115 and / or the server 120. However, not all of the components depicted may be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figure. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0039] Figure 2Various portions of the architecture may be implemented in software or hardware, including by one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. Figure 3 In the example of FIG. 1 , sensor 152 provides environmental information (e.g., depth information from one or more depth sensors) to mapping and object detection engine 200. Camera 150 may also provide images, such as a video stream, to mapping and object detection engine 200. Mapping and object detection engine 200 may use environmental information (e.g., depth information and / or images) from sensor 152 and camera 150 to generate three-dimensional scene information, such as a three-dimensional map, of some or all of the physical environment of electronic device 105.

[0040] like Figure 2 As shown, the XR application 202 may request an anchor, such as a physical object anchor, in an anchor request from the mapping and object detection engine 200. The XR application 202 may be a gaming application, a media player application, a content editor application, a training application, a simulator application, or generally any application that displays CG or virtual content at a location that depends on the physical environment, such as by anchoring the virtual content to a physical object anchor.

[0041] The physical object anchor may be a general physical object, such as a horizontal flat surface (e.g., the surface of a floor or a tabletop), a vertical flat surface (e.g., the surface of a wall), or a specific physical object (e.g., a table, a wall, a TV cabinet, a sofa, a refrigerator, a table, a chair, etc.). The XR application 202 may include code that, when executed by one or more processors of the electronic device 105, generates CG content (e.g., a virtual cup, a virtual document, a virtual TV screen, a virtual cinema screen, a virtual keyboard, or other virtual content) for display on, near, attached to, or otherwise associated with the physical object anchor. The XR application 202 generates the virtual content when the mapping and object detection engine 200 detects the physical object anchor in the environmental information from the sensor 152 and the camera 150, and provides the relevant anchor information to the XR application 202 and / or the CG rendering engine 223.

[0042] Once the CG content (e.g., a virtual cup, a virtual document, a virtual TV screen, a virtual cinema screen, a virtual keyboard, etc.) is generated, the CG content may be provided to a CG rendering engine 223, such as Figure 2As shown. Environmental information such as a depth map of the physical environment and / or object information of objects detected in the physical environment may also be provided to the CG rendering engine 223. The CG rendering engine 223 may then render CG content from the XR application 202 for display by the display 225 of the electronic device 105. The CG content is rendered for display at an appropriate location on the display 225 (e.g., using scene information and / or anchor information provided by the mapping and object detection engine 200) to appear in association with a physical anchor object provided by the mapping and object detection engine 200. The display 225 may be, for example, an opaque display, and the camera 150 may be configured to provide a pass-through video feed to the opaque display. The CG content may be rendered for display at a location on the display corresponding to the display location of the physical anchor object in the pass-through video. For another example, the display 225 may be a transparent or translucent display. The CG content may be rendered for display at a location on the display corresponding to the direct field of view of the physical anchor object through the transparent or translucent display.

[0043] As shown, the electronic device 105 may also include a video rendering engine 227 that renders a video image of the physical environment based on the image from the camera 150 for display together with the rendered CG content from the CG rendering engine 223. For example, the video rendering engine 227 may be provided in an electronic device 105 including an opaque display to provide a pass-through video to the display. In an electronic device 105 implemented with a transparent or semi-transparent display that allows a user to directly view the physical environment, the video rendering engine 227 may be omitted or not used in some cases. Although Figure 2 The example shows a CG rendering engine 223 separate from the XR application 202, but it should be understood that in some specific implementations, the XR application 202 can render CG content for display on the display 225 without using a separate CG rendering engine 223.

[0044] The electronic device 105 may allow the XR application 202 to request and obtain anchor information from the mapping and object detection engine 200 (eg, via an application programming interface or API), such as Figure 2As shown, this can facilitate efficient development, implementation, and / or runtime execution of the XR applications 202 (e.g., because each XR application 202 does not have to perform its own object detection, scene mapping, transformations, etc.). However, this can be problematic in situations where the physical anchor object requested by the XR application 202 does not exist or is otherwise unavailable in the physical environment. For example, the XR application 202 may request a table on which to anchor a virtual cup when: there is no table in the physical environment of the electronic device 105 (e.g., within the mapping distance of the electronic device 105), or there is no table within the field of view of the user of the electronic device 105. That is, the electronic device 105 (e.g., the mapping and object detection engine 200) may determine that the physical anchor object (the table in this example) is not available in the physical environment based on the environmental information from the sensor 152 and / or the camera 150. The sensor 152 may include at least one depth sensor, and the environmental information may include environmental information based on at least one image from the camera and depth information from the depth sensor.

[0045] In the event that the requested physical anchor object is not available, the electronic device 105 (e.g., the mapping and object detection engine 200) generates a virtual anchor having a fixed position in the physical environment, and provides the virtual anchor to the XR application 202 and / or the CG rendering engine 223 in response to the request for the physical anchor object. The XR application and / or the CG rendering engine 223 may then use the virtual anchor to render virtual content for display at a fixed position in the physical environment. The electronic device 105 (e.g., the mapping and object detection engine 200) may also generate a virtual anchor object corresponding to the physical anchor object. For example, if a physical wall is requested as a physical anchor object, and the physical wall is not available, the electronic device 105 (e.g., the CG rendering engine 223) may generate and render a virtual wall (e.g., a virtual anchor object) for display on the display of the electronic device 105 at a position that makes the virtual anchor object appear to the user to be located at a fixed position in the physical environment. The XR application 202 and / or the CG rendering engine 223 may then use the virtual anchor to render the virtual content to also appear to the user to be located at or near a fixed location in the physical environment. Such anchoring of the virtual content to the virtual anchor object includes positioning the rendered virtual content relative to the rendered virtual anchor object (e.g., by positioning the rendered virtual content to appear to be on a virtual wall).

[0046] In one or more specific implementations, during operation of, for example, an XR application such as the XR application 202, the transformation of the physical anchor and / or virtual anchor may be updated at the system level of the electronic device 105 (e.g., by the mapping and object detection engine 200 and / or the CG rendering engine 223, without round-trip communication to the requesting application). For example, the CG rendering engine 223 may update the position and / or orientation of the CG content provided by the XR application 202 based on the physical anchor and / or virtual anchor (e.g., based on the physical anchor and / or virtual anchor identified for the XR application 202, such as in response to an anchor request from the XR application). In this way, an electronic device such as the electronic device 105 can render CG content anchored to or relative to a physical location within a physical environment at a lower latency than in a scenario where the position and orientation of the virtual content is updated at the XR application 202. Reducing rendering latency in this manner can help the electronic device 105 provide an experience that is free of potential XR anchoring artifacts, such as application content that appears to be unintentionally moving around relative to the physical environment and / or other XR content.

[0047] Figure 3 , Figure 4 and Figure 5 Examples are shown in which virtual content is anchored to a physical anchor object, a virtual anchor, and a virtual anchor object, respectively, according to aspects of the present disclosure.

[0048] exist Figure 3 In the example of FIG. 1 , user 101 wears electronic device 105 in physical environment 300. Figure 3 In the example of , physical environment 300 includes physical wall 302, physical table 304, and physical floor 306, which can be detected, mapped, and / or classified using environmental information based on images from one or more cameras such as camera 150 and one or more sensors (e.g., depth sensors) such as sensor 152. Figure 3 In the example of , virtual content including a virtual cup 320 and a virtual television 322 are displayed by a display (e.g., display 225) of the electronic device 105 to appear to the user 101 to be located at a fixed location in the physical environment. In this example, the virtual cup 320 is anchored to the physical table 304, and the virtual television 322 is anchored to the physical wall 302. In this way, the virtual content can be displayed to appear to be located at a fixed location in the physical environment 300 even as the user 101 moves and / or looks around in the physical environment.

[0049] To anchor the virtual cup 320 to the physical table 304 or the virtual TV 322 to the physical wall 302, the XR application may request (e.g., from a mapping and object recognition engine) a physical table (or horizontal surface) or a physical wall (or vertical surface) in the physical environment to which the XR application may anchor the relevant virtual content. In order to use a physical table or a physical wall as a physical anchoring object, the XR application may obtain the location of one or more points on the physical object that describe the physical location, shape, position, and / or orientation of the physical object. For example, the location of three points on the surface of the physical table 304 may be provided, allowing the XR application to identify a plane corresponding to the surface of the table. The XR application may then determine where the virtual cup 320 is positioned and how it is oriented to appear to rest on the surface of the table. In other examples, more information for the physical anchoring object (e.g., a three-dimensional mesh of the object) may be provided to facilitate physical anchoring.

[0050] In some cases, when an XR application requests a physical anchor object, the electronic device 105 may determine (e.g., based on environmental information from the sensor 152 and / or the camera 150) that the physical anchor object is not available in the physical environment.

[0051] Figure 4 An example is shown in which the physical environment 300 includes a physical wall 302 but does not include a physical table. In response to a request for a physical anchor object in the form of a physical table and determining that a physical table is not available, the electronic device 105 may generate a virtual anchor having a fixed position in the physical environment. Figure 4 In the example of , a virtual anchor 401 has been provided to the XR application, to which the virtual cup 320 is to be anchored, and the virtual cup 320 has been rendered and displayed to appear to be located at a fixed position of the virtual anchor 401 .

[0052] In this example, the virtual anchor 401 is a single point at a fixed location in the physical environment. The fixed location of the virtual anchor 401 can be defined relative to physical objects in the physical environment 300 so that the virtual anchor (and any virtual content anchored thereto) remains at a fixed physical location even as the user 101 moves and / or looks around the XR environment including the physical environment 300 and the virtual content. In this example, the fixed location of the virtual anchor 401 is defined by a first distance 404 from the physical floor 306, a second distance 402 from the physical wall 302, and a third distance 400 from the end of the physical wall. However, these distances are merely illustrative, and the fixed location of the virtual anchor 401 can be defined relative to any three points in the physical environment.

[0053] exist Figure 4In the example of , the virtual anchor 401 is located at a fixed position in the air in the physical environment 300, resulting in the virtual cup 320 appearing to float in space when displayed by the electronic device 105 at the fixed position of the virtual anchor. The fixed position of the virtual anchor 401 may be automatically determined by the electronic device 105 (e.g., based on heuristics) and / or may be selected and / or modified by the user 101. For example, the user 101 may use a handheld control to move the virtual anchor 401 by virtually grabbing or pinching the virtual anchor (e.g., as detected using the camera 150 and / or sensor 152), using voice commands, or other input modalities of the electronic device 105.

[0054] To facilitate placement and / or movement of the virtual anchor 401, the electronic device 105 may highlight potential locations in the physical environment 300 for the virtual anchor (e.g., to help the user select a fixed location). For example, potential locations in the physical environment 300 may be dynamically highlighted based on the user's field of view, gaze location, and / or movement speed (e.g., to allow the user to place or project the virtual anchor 401 to a recommended location). The potential location may be a location in space that has enough space for specific virtual content, or a location at or near other physical objects that may be suitable for displaying virtual content.

[0055] Figure 4 Examples of may result in an XR environment that is different from the expected XR environment from the XR application. For example, in order for the XR environment to look and feel real to the user, a virtual anchor that causes a virtual cup to float in space may be undesirable.

[0056] In some cases, the disclosed systems and methods may provide an improved XR experience by providing a virtual anchor object corresponding to a requested physical anchor object when the physical anchor object is not available in the physical environment. For example, if the physical environment is suitable (e.g., if there is space on the floor of a room) and if the requested anchor is a physical object (e.g., a horizontal surface, a table, or a wall), a virtual version of the requested physical object may be rendered and displayed.

[0057] Figure 5 An example is shown in which the physical environment 300 includes a physical wall 302 but does not include a physical table, and in which a virtual table 504 is provided as a virtual anchor object corresponding to an unavailable physical anchor object. Figure 5As shown, a virtual anchor object such as a virtual table 504 may be rendered for display such that the virtual table 504 appears to the user 101 to be located or anchored to a fixed position of the virtual anchor 401 in the physical environment. In this example, the base of one of the legs of the virtual table is located at the virtual anchor 401 (e.g., as defined by a distance from the physical floor 306 (zero in this example), a distance 502 from the physical wall 302, and a distance 500 from the end of the physical wall 302), and the remainder of the virtual table 504 (e.g., the tabletop and other legs) is generated relative to this position based on the desired size and shape of the table and based on the available space for the virtual table in the physical environment 300. In various examples, the size, orientation, shape, etc. of the virtual anchor object associated with the requested physical object depends on the detected environmental features.

[0058] exist Figure 5 In the example of FIG. 5 , the virtual cup 320 is rendered for display at a fixed position determined based on the size, orientation, position, and / or shape of the virtual anchor object (e.g., the virtual table 504). In this example, the virtual cup 320 is positioned at a different position than the position of the virtual anchor 401, but is fixed at a position determined by the characteristics of the table relative to the virtual anchor and the physical environment. It should also be understood that the virtual anchor 401 is fixed relative to the physical environment. Figure 5 The positions of the virtual anchor object and virtual cup 320 in are merely illustrative, and other arrangements in which virtual content is anchored to a virtual anchor object that is itself anchored to a fixed position relative to the physical environment are possible. For example, the virtual anchor may be located at another location on the virtual anchor object (e.g., the center of the desktop), separate from the virtual anchor object, or the virtual anchor object itself may be a virtual anchor. In various examples, rendering virtual content for display at a fixed position in the physical environment using a virtual anchor may include positioning the rendered virtual content relative to the rendered virtual anchor object.

[0059] When generating a virtual anchor object (e.g. Figure 5 Before the user 101 places the virtual table in the example of FIG. 300 , the electronic device 105 may identify an available space for the virtual anchor object based on the environmental information of the physical environment 300. The electronic device 105 may determine the size of the virtual anchor object based on the identified available space for the virtual anchor object. The electronic device 105 may provide the user 101 with the ability to adjust the size of the virtual anchor object (e.g., using gesture-based input, controller-based input, or other input).

[0060] In some cases, the electronic device 105 may determine that the physical environment is not suitable for providing a virtual anchor object corresponding to the requested physical anchor object (e.g., if there is no room for a virtual table in the room, or if there is no clear view of a wall). In cases where the physical environment is not suitable for providing a virtual anchor object, a virtual anchor that is a point in the space that has no associated virtual anchor object may be provided (e.g., Figure 4 ), or a virtual portal may be generated to create a space in the XR environment in which a virtual version of the requested physical anchor object is rendered. This may be particularly useful in crowded spaces such as on a bus or airplane, where a virtual portal may create a partially immersive environment that replaces nearby objects (such as a seat and tray table in front of the user) with a space in which virtual versions of physical objects (e.g., a virtual table, a virtual movie screen, a virtual wall) may be displayed.

[0061] Figure 6 The illustration does not include a method for rendering virtual anchor objects such as Figure 5 An example of a physical environment 600 with enough space for a virtual table, virtual walls, etc. Figure 6 In the example of , a user 101 (whose wearable electronic device 105) views a portion of a physical environment 600 that includes the backs of three seats 602 (e.g., airplane seats or bus seats) in a row in front of the row in which the user is sitting, and includes the armrests 604 of the user's own seat and a portion of the user's legs 601.

[0062] In a scenario where an XR application of the electronic device 105 requests a physical anchor object in the form of a physical table while the user is in the physical environment 600, the physical environment 600 does not include the requested physical anchor object, and the electronic device 105 may determine that there is insufficient space in the physical environment 600 for a virtual anchor object corresponding to the requested physical table.

[0063] Figure 7 An example is shown in which the electronic device 105 generates and renders a virtual portal 700 for display over a portion of the physical environment 600. In this example, the electronic device 105 renders a virtual anchor object such as a virtual table 504 for display at a fixed position relative to the physical environment by rendering a virtual anchor object for display in the virtual portal 700. In this example, the virtual portal and / or virtual table may be anchored to the physical environment (e.g., anchored to a position on one seat 602).

[0064] like Figure 7As shown, providing a virtual portal 700 may include blocking a portion of the user's view of the physical environment. This may include blocking the view of the physical environment using a display of the electronic device 105 and displaying the rendered virtual environment using the display. In one example, blocking the view of the physical environment using the display may include terminating the display of a transparent video feed corresponding to the blocked portion of the physical environment from a camera of the electronic device. In another example, blocking the view of the physical environment using the display may include directing display light into the user's eyes using a transparent or translucent display of an electronic device disposed between the physical environment and the user's eyes. The virtual anchor object and the virtual content anchored to the object may then be rendered and displayed in the virtual portal 700 (e.g., overlaid on the virtual portal).

[0065] As indicated by arrow 702, virtual portal 700 may be adjusted (e.g., resized) by the user. For example, the user may reach out their hand to what appears to the user to be a corner or edge of virtual portal 700, and pinch or otherwise virtually grasp the corner or edge, and pull or push the virtual edge to increase or decrease the size of the virtual portal. Increasing or decreasing the size of virtual portal 700 in this manner causes virtual portal 700 to be displayed above an expanded or reduced portion of physical environment 600. In some cases, the size of the virtual portal may be limited by characteristics of the physical environment. For example, if the user adjusts the size of virtual portal 700 until the edge of the virtual portal encounters the physical wall 704 of an airplane or bus, the user may be prevented from further increasing the size of virtual portal 700, or the virtual portal may be replaced with a virtual environment.

[0066] Figure 8 An example is shown, where Figure 6 and Figure 7 The user's view of the physical environment 600 is blocked by the displayed virtual environment 800, except for the user's view of the user's own legs 601 and the armrests 604 of the user's own chair. Figure 8 In the example of , the user's legs and armrests remain visible to help the user orient themselves in the virtual environment. However, this is merely illustrative, and in other examples, more or fewer (e.g., no) physical objects and / or environments may be visible to the user.

[0067] When the user resizes the virtual portal to exceed a predetermined size (e.g., beyond the user's 180 degree field of view) or when it is initially determined that there is insufficient space in the physical environment for a virtual anchor object (e.g., if the user sets a preference for the virtual environment rather than the virtual portal), the virtual environment 800 may be provided. Figure 8In the example of FIG. 1 , the electronic device 105 has rendered and displayed the virtual environment 800 above the physical environment 600, and has rendered and displayed a virtual anchor object (e.g., a virtual wall 804) for display at a fixed position relative to the physical environment 600 (e.g., by rendering the virtual anchor object at a fixed position in the virtual environment 800). Although not visible to the user, at least one position, such as a point on the virtual wall 804, is anchored to a position in the physical environment 600, such that the virtual content (e.g., Figure 6 The virtual movie screen 802 in the example of can be anchored to a virtual anchor. In this way, even if the user moves and / or looks around the physical environment 600, the virtual content can be displayed at a fixed position in the virtual environment 800.

[0068] In various operational scenarios, the virtual environment 800 may be an artificial environment (e.g., a virtual movie theater, a virtual beach, etc.), or may be a virtual version of a known physical environment that is different from the current physical environment 600. For example, the known physical environment may be the user's home, the user's living room, the user's bedroom, the user's office, the user's car, or any other physical environment that has been previously mapped (e.g., by the electronic device 105 or another device with depth sensing and / or imaging capabilities). The known physical environment may include one or more known physical anchor objects (e.g., the user's TV stand, the user's bedroom wall, the user's kitchen table, the user's desk, etc.). The known physical anchor objects may be requested independently of the known physical environment and / or as part of the known physical environment.

[0069] exist Figure 8 In the example of , the virtual environment may include other virtual objects, such as a virtual coffee table 806 corresponding to a physical coffee table in the user's living room. In this example, the requested physical anchor object may be a known physical wall in the user's living room, and the virtual anchor object may be a virtual wall 804 that corresponds to the wall of the user's living room (e.g., in terms of size, shape, color, position, etc.). In this way, the user may be provided with a virtual environment that is fixed (e.g., anchored) to the current physical environment and makes the user feel as if they are in a different known physical environment (e.g., so that even on an airplane, on a bus, or in a hotel room, the user can feel as if they are watching a movie in their own living room or working at their own desk).

[0070] As in Figure 8In the example of , displaying the virtual environment as in the example of may include blocking the view of the physical environment 600 using the display of the electronic device 105 and displaying the rendered virtual environment 800 using the display. Blocking the view of the physical environment using the display may include terminating the display of some or all of the pass-through video feeds from the camera of the electronic device, or may include directing display light into the user's eyes using a transparent or semi-transparent display of the electronic device disposed between the physical environment and the user's eyes.

[0071] Fig. 9 A flowchart of an exemplary process 900 for providing virtual anchoring for an extended reality display device according to a specific implementation of the subject technology is shown. Figure 1 and Figure 2 The process 900 is described with reference to the electronic device 105. However, the process 900 is not limited to Figure 1 and Figure 2 The electronic device 105 of the embodiment of the present invention, and one or more frames (or operations) of process 900 may be performed by one or more other components of other suitable devices (including electronic device 110, handheld electronic device 114, electronic device 115 and / or server 120). Further for the purpose of explanation, some frames of process 900 are described herein as occurring sequentially or linearly. However, multiple frames of process 900 may occur in parallel. In addition, the frames of process 900 do not have to be performed in the order shown, and / or one or more frames of process 900 do not have to be performed and / or may be replaced by other operations.

[0072] like Fig. 9 As shown, at block 902, an electronic device such as electronic device 105 obtains environmental information of a physical environment of the electronic device. The environmental information may include depth information, image information, object information, and / or other three-dimensional information about the physical environment (e.g., as determined using one or more depth sensors such as sensor 152 and / or one or more cameras such as camera 150 of the electronic device).

[0073] At block 904, the electronic device (e.g., from a mapping and object detection engine such as Figure 2 An XR application such as XR application 202 at the mapping and object detection engine 200 receives a request for a physical anchor object in the physical environment (e.g., a physical table, a horizontal plane, a vertical plane, a physical wall, a physical floor, a sofa, a chair, a TV stand, a window, a tree, a flower, a cloud, etc.).

[0074] At block 906, the electronic device determines, based on the environmental information, that the physical anchor object is not available in the physical environment.

[0075] At block 908, the electronic device generates a virtual anchor having a fixed position in the physical environment. The fixed position may be defined relative to one or more objects or locations in the physical environment. The virtual anchor may be a point in space and / or may be associated with a virtual anchor object corresponding to the requested physical anchor object. For example, the electronic device may generate a virtual anchor object corresponding to a physical anchor object.

[0076] At block 910 , in response to a request for a physical anchor object, the electronic device provides a virtual anchor (eg, from a mapping and object detection engine such as mapping and object detection engine 200 to an XR application such as XR application 202 ).

[0077] At block 912, the electronic device renders virtual content for display at a fixed location in the physical environment using the virtual anchor. The electronic device may also render a virtual anchor object for display at the fixed location or at another fixed location defined relative to the fixed location in the physical environment. Rendering virtual content for display at a fixed location in the physical environment using the virtual anchor may include positioning the rendered virtual content relative to the rendered virtual anchor object (e.g., by positioning the rendered virtual content relative to the rendered virtual anchor object). Figure 5 or Figure 7 The virtual cup 320 is positioned on the virtual table 504 as in the example of Figure 8 804 as in the example of ).

[0078] Fig.10 An electronic system 1000 is shown that can be used to implement one or more implementations of the subject technology. The electronic system 1000 can be Figure 1 The electronic device 105, handheld electronic device 104, electronic device 110, electronic device 115 and / or server 120 are shown and / or may be a part thereof. The electronic system 1000 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1000 includes a bus 1008, one or more processing units 1012, a system memory 1004 (and / or cache), a ROM 1010, a permanent storage device 1002, an input device interface 1014, an output device interface 1006, and one or more network interfaces 1016, or a subset and variant thereof.

[0079] The bus 1008 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 1000. In one or more specific implementations, the bus 1008 communicatively connects one or more processing units 1012 with the ROM 1010, the system memory 1004, and the permanent storage device 1002. The one or more processing units 1012 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different specific implementations, the one or more processing units 1012 can be a single processor or a multi-core processor.

[0080] ROM 1010 stores static data and instructions required by one or more processing units 1012 and other modules of electronic system 1000. On the other hand, permanent storage device 1002 can be a read-write memory device. Permanent storage device 1002 can be a non-volatile memory unit that stores instructions and data even when electronic system 1000 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as permanent storage device 1002.

[0081] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 1002. Like the permanent storage device 1002, the system memory 1004 may be a read-write memory device. However, unlike the permanent storage device 1002, the system memory 1004 may be a volatile read-write memory, such as a random access memory. The system memory 1004 may store any of the instructions and data that one or more processing units 1012 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 1004, the permanent storage device 1002, and / or the ROM 1010. The one or more processing units 1012 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0082] The bus 1008 is also connected to an input device interface 1014 and an output device interface 1006. The input device interface 1014 enables a user to transmit information and select commands to the electronic system 1000. Input devices that can be used with the input device interface 1014 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 1006 may, for example, enable the display of images generated by the electronic system 1000. Output devices that can be used with the output device interface 1006 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0083] Finally, if Fig.10 As shown, bus 1008 also couples electronic system 1000 to one or more networks and / or to one or more network nodes, such as via one or more network interfaces 1016. Figure 1 . In this manner, electronic system 1000 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of electronic system 1000 may be used with the subject disclosure.

[0084] The above functions can be implemented in computer software, firmware or hardware. The technology can be implemented using one or more computer program products. The programmable processor and computer can be included in a mobile device or packaged as a mobile device. The process and logic flow can be performed by one or more programmable processors and one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through a communication network.

[0085] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files including higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0086] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0087] As used in this specification and any claims of this patent application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, tangible objects that store information in a form that can be read by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0088] To provide interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0089] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with a specific implementation of the subject matter described in this specification, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0090] A computing system may include a client and a server. The client and the server are generally remote from each other and may interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some specific implementations, the server transmits data (e.g., an HTML page) to a client device (e.g., in order to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., a result of a user interaction) may be received at the server from the client device.

[0091] According to various aspects of the present subject disclosure, a method is provided, the method comprising: utilizing an electronic device to obtain environmental information of a physical environment of the electronic device; receiving, by the electronic device, a request for a physical anchor object in the physical environment; determining, based on the environmental information, that the physical anchor object is unavailable in the physical environment; generating a virtual anchor having a fixed position in the physical environment; providing the virtual anchor in response to the request for the physical anchor object; and rendering, by the electronic device, virtual content for display at a fixed position in the physical environment using the virtual anchor.

[0092] According to various aspects of the subject disclosure, a system is provided, the system including a processor and a memory device, the memory device containing instructions, which when executed by the processor causes the processor to perform operations including: using an electronic device to obtain environmental information of a physical environment of the electronic device; receiving, by the electronic device, a request for a physical anchor object in the physical environment; determining, based on the environmental information, that the physical anchor object is unavailable in the physical environment; generating a virtual anchor having a fixed position in the physical environment; providing the virtual anchor in response to the request for the physical anchor object; and using the virtual anchor by the electronic device to render virtual content for display at a fixed position in the physical environment.

[0093] According to various aspects of the subject disclosure, a non-transitory computer-readable medium is provided that includes instructions that, when executed by a computing device, cause the computing device to perform operations including: obtaining environmental information of a physical environment of the electronic device using an electronic device; receiving, by the electronic device, a request for a physical anchor object in the physical environment; determining, based on the environmental information, that the physical anchor object is unavailable in the physical environment; generating a virtual anchor having a fixed position in the physical environment; providing the virtual anchor in response to the request for the physical anchor object; and rendering, by the electronic device, virtual content for display at a fixed position in the physical environment using the virtual anchor.

[0094] Those skilled in the art will recognize that various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The functions can be implemented in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of the subject technology.

[0095] It should be understood that the specific order or hierarchy of steps in the process disclosed by the present invention is an illustration of an exemplary method. Based on design preference requirements, it should be understood that the specific order or hierarchy of steps in the process can be rearranged. Some of the steps can be performed simultaneously. The attached method claims present elements of various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0096] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications of these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically indicated. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are only for convenience and do not limit the present invention described herein.

[0097] As used herein, the term website may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, and the like. Thus, the term website may be used interchangeably with the terms web page and server. The predicate words "configured to," "capable of operating to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is capable of operating to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code or is capable of operating to execute code.

[0098] As used herein, the term automatically may include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0099] Phrases such as "aspects" do not mean that this aspect is essential to the subject technology or that this aspect applies to all configurations of the subject technology. The disclosure related to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "implementation" do not mean that this implementation is essential to the subject technology or that this implementation applies to all configurations of the subject technology. The disclosure related to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, and vice versa. Phrases such as "configuration" do not mean that this configuration is essential to the subject technology or that this configuration applies to all configurations of the subject technology. The disclosure related to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

Claims

1. A device comprising: processor; and A computer readable medium having stored thereon instructions which, when executed, cause the processor to: Obtaining environmental information of a physical environment of the electronic device through a system program of the electronic device; receiving, by the system program and from an application running on the electronic device, a request for a physical anchor object in the physical environment; determining, by the system program and based on the environment information, that the physical anchor object is not available in the physical environment; generating, by the system program, a virtual anchor point having a location in the physical environment; as well as Virtual content is rendered by the system program of the electronic device for display at the location in the physical environment using the virtual anchor point.

2. The apparatus of claim 1 , wherein the instructions further cause the processor to: The position or orientation of the virtual content is updated by a system program based on information from the application program. 3 . The device of claim 1 , wherein the system program comprises a mapping engine, wherein the environmental information of the physical environment is at least partially obtained by the mapping engine based on one or more sensors of the electronic device. 4 . The apparatus of claim 1 , wherein the physical anchor object comprises a general physical object or a specific physical object, wherein the general physical object comprises a flat surface.

5. The apparatus of claim 1 , wherein the instructions further cause the processor to: A virtual object generated by the application is received, the virtual object corresponding to the physical anchor object, wherein the rendering of the virtual content includes rendering the virtual object relative to the virtual anchor point for display.

6. The apparatus of claim 5, wherein the virtual object is a first virtual object, wherein the instructions further cause the processor to: generating a second virtual object; and The second virtual object is rendered relative to the virtual anchor point, wherein a portion of the first virtual object separates a position of the second virtual object from the virtual anchor point.

7. The apparatus of claim 1 , wherein the instructions further cause the processor to: The virtual anchor point is defined relative to at least one physical object in the physical environment.

8. The apparatus of claim 1, wherein the instructions further cause the processor to: One or more highlighted locations in the physical environment are provided for display, the one or more highlighted locations corresponding to potential locations in the physical environment having sufficient space to display the virtual content.

9. The device of claim 8, wherein the one or more highlighted locations are automatically displayed based at least in part on a detected field of view, gaze location, or motion speed.

10. The apparatus of claim 1, wherein the instructions further cause the processor to: The virtual content is resized based on available space in the physical environment for the virtual content.

11. The apparatus of claim 1 , wherein the instructions further cause the processor to: A virtual environment is rendered for display over a portion of the physical environment, wherein rendering the virtual content includes rendering the virtual content for display in the virtual environment.

12. The apparatus of claim 11, wherein the virtual environment is displayed so as to block a view of the physical environment.

13. The apparatus of claim 1 , wherein the instructions further cause the processor to: A virtual portal is rendered for display on a portion of the physical environment, wherein rendering the virtual content includes rendering the virtual content for display in the virtual portal.

14. A method comprising: Obtaining environmental information of a physical environment of the electronic device through a system program of the electronic device; receiving, by the system program and from an application running on the electronic device, a request for a physical anchor object in the physical environment; determining, by the system program and based on the environment information, that the physical anchor object is not available in the physical environment; generating, by the system program, a virtual anchor point having a location in the physical environment; as well as Virtual content is rendered by the system program of the electronic device to be displayed at the location in the physical environment using the virtual anchor point.

15. The method according to claim 14, further comprising: The position or orientation of the virtual content is updated by the system program based on the information from the application program.

16. The method according to claim 14, further comprising: A virtual object generated by the application is received, the virtual object corresponding to the physical anchor object, wherein the rendering of the virtual content includes rendering the virtual object relative to the virtual anchor point for display.

17. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to: Obtaining environmental information of a physical environment of the electronic device through a system program of the electronic device; receiving, by the system program and from an application running on the electronic device, a request for a physical anchor object in the physical environment; determining, by the system program and based on the environment information, that the physical anchor object is not available in the physical environment; generating, by the system program, a virtual anchor point having a location in the physical environment; and Virtual content is rendered by the system program of the electronic device for display at the location in the physical environment using the virtual anchor point. 18 . The non-transitory computer-readable medium of claim 17 , wherein the system program comprises a mapping engine, wherein the environmental information of the physical environment is obtained at least in part by the mapping engine based on one or more sensors of the electronic device.

19. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the computing device to: The position or orientation of the virtual content is updated by the system program based on the information from the application program.

20. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the computing device to: A virtual object generated by the application is received, the virtual object corresponding to the physical anchor object, wherein the rendering of the virtual content includes rendering the virtual object relative to the virtual anchor point for display.

Images