Anchoring Virtual Content to Physical Surfaces

Abstract

Systems and techniques are provided for rendering virtual content within a scene. An exemplary method includes determining a surface plane within a scene of a physical environment, rendering the virtual content in perspective to a location within the scene, rendering a content placement indicator associated with the virtual content on a first space of the surface plane, the content placement indicator indicating a first space for placement of the virtual content, determining one or more locations of one or more objects on a surface associated with the surface plane, and rendering the content placement indicator on a second space of the surface plane in response to an input, the second space being determined based on the one or more locations and the input, and the content placement indicator indicating the second space for placement of the virtual content.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to organizing and rendering virtual content for extended reality. For example, aspects of this disclosure relate to techniques and systems for placing virtual content on physical surfaces. [Background technology]

[0002] Extended reality (e.g., augmented reality, virtual reality, mixed reality, etc.) devices such as smart glasses and head-mounted displays (HMDs) typically implement cameras and various sensors to track the position of the extended reality (XR) device and other objects in a physical environment. The XR device can use such tracking information to provide a realistic XR experience to a user of the XR device. For example, the XR device may enable a user to experience or interact with an immersive virtual environment or content. To provide a realistic XR experience, the XR technology may integrate virtual content with the physical world. In some cases, the XR technology may match the relative pose and motion of objects and devices in the physical world. For example, the XR device may use the tracking information to calculate the relative pose of a map of the device, objects, and / or real-world environment in order to match the relative positions and motions of the device, objects, and / or real-world environment. Using the pose and motion of one or more devices, objects, and / or real-world environments, the XR device can convincingly render the content relative to the real-world environment. The relative pose information can be used to match virtual content with the user's perceived motion and the spatiotemporal state of devices, objects, and the real-world environment. Summary of the Invention

[0003] In some examples, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively "systems and techniques") are described for anchoring virtual content to a physical surface. According to at least one illustrative example, a method for rendering virtual content within a scene of a physical environment is provided. In some examples, the method may include determining at least one surface plane within the scene of the physical environment, rendering a virtual content item at a first location within the scene with a first perspective via a computing device, rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item, determining one or more locations of one or more objects on a surface associated with the at least one surface plane, and rendering the content placement indicator on a second space of the at least one surface plane in response to a user input, the second space of the at least one surface plane being determined based on the one or more locations and the user input, and the content placement indicator indicating at least a portion of the second space for placement of the virtual content item.

[0004] According to at least one illustrative example, a non-transitory computer-readable medium is provided for rendering virtual content within a scene of a physical environment. In some aspects, the non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to determine at least one surface plane within the scene of the physical environment, render a virtual content item at a first location within the scene with a first perspective, render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item, determine one or more locations of one or more objects on a surface associated with the at least one surface plane, and render a content placement indicator on a second space of the at least one surface plane in response to user input, the second space of the at least one surface plane being determined based on the one or more locations and the user input, and the content placement indicator indicating at least a portion of the second space for placement of the virtual content item.

[0005] According to at least one illustrative example, an apparatus is provided for rendering virtual content within a scene of a physical environment. In some aspects, the apparatus may include a memory having computer-readable instructions stored therein and one or more processors configured to: determine at least one surface plane within the scene of the physical environment; render a virtual content item at a first location within the scene with a first perspective; render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; determine one or more locations of one or more objects on a surface associated with the at least one surface plane; and render a content placement indicator on a second space of the at least one surface plane in response to a user input, the second space of the at least one surface plane being determined based on the one or more locations and the user input, and the content placement indicator indicating at least a portion of the second space for placement of the virtual content item.

[0006] According to another illustrative example, another apparatus for rendering virtual content within a scene of a physical environment may include means for determining at least one surface plane within the scene of the physical environment; means for rendering a virtual content item at a first location within the scene with a first perspective via a computing device; means for rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; and means for determining one or more locations of one or more objects on a surface associated with the at least one surface plane and rendering the content placement indicator on a second space of the at least one surface plane in response to user input, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, and the content placement indicator indicates at least a portion of the second space for placement of the virtual content item.

[0007] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include rendering the virtual content item in a second perspective at a second location in the scene based on the content placement indicator in response to an additional user input requesting placement of the virtual content item on the second space. In some examples, the second location may be in the second space. In some cases, the methods, non-transitory computer-readable media, and devices described above may include determining the second perspective based on the perspective of the second space.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining a first surface plane and a second surface plane of at least one surface plane, where a first space is associated with the first surface plane and a second space is associated with the second surface plane, and determining a second perspective based on the perspective of the second surface plane.

[0009] In some examples, rendering the content placement indicator on the second space may include determining an overlap between the virtual content item and the second surface plane, and switching to rendering the content placement indicator on the second space associated with the second surface plane upon determining that the overlap exceeds a first threshold.

[0010] In some examples, the first space and the second space are associated with a first surface plane of the at least one surface plane. In some aspects, the methods, non-transitory computer-readable media, and devices described above may include detecting one or more objects occluding at least a portion of the first surface plane, and segmenting the first surface plane into at least a first segment and a second segment based at least on the occluded portion of the first surface plane. In some examples, the first space is associated with the first segment, and the second space is associated with the second segment.

[0011] In some cases, segmenting the first surface plane into at least a first segment and a second segment may include determining the fit of a virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

[0012] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include changing an aspect ratio of the virtual content item between the first segment and the second segment.

[0013] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include rendering the virtual content item in a third perspective at a third location within the scene based on the content placement indicator prior to rendering the content placement indicator on the second space. In some examples, the third location may be within the first space.

[0014] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include moving a content placement indicator within a first space in response to user input before rendering the content placement indicator on a second space.

[0015] In some cases, determining at least one surface plane in the scene may include applying an offset to the at least one surface plane perpendicular to the at least one surface and toward a viewpoint of a user associated with the computing device.

[0016] In some cases, determining at least one surface plane in the scene may include determining one or more edges of one or more objects in the scene, and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

[0017] In some cases, rendering the content placement indicator may include rendering a pattern on at least one surface plane, in some examples, the pattern may include at least one of a highlight, an outline, a color, a shading, a shadow, a hatching, and a gradient.

[0018] In some cases, determining at least one surface plane in the scene may include determining one or more surfaces of one or more objects in the scene, and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0019] In some aspects, the device may be or be part of a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a multi-camera system, or other device. In some aspects, the device includes a camera or cameras that capture one or more images. In some aspects, the device further includes a display that displays one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors.

[0020] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0021] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0022] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a simplified block diagram illustrating an exemplary extended reality system, according to some examples of the present disclosure. [Figure 2A] FIG. 1 illustrates an example scene having various surfaces available for placing virtual content for rendering, in accordance with some examples of the present disclosure. [Figure 2B] FIG. 1 illustrates an example scene with various physical objects on a surface available for placing virtual content for rendering, in accordance with some examples of the present disclosure. [Figure 3A] A diagram showing an example of selecting a surface plane to highlight when the location of a virtual content item overlaps different surfaces on a scene, according to some examples of the present disclosure. [Figure 3B] 1A-1C illustrate examples of switching between highlighted surface planes for placement of virtual content items, according to some examples of the present disclosure. [Figure 4] A diagram showing exemplary adjustments to the aspect ratio of a content placement indicator to fit within available space on a surface plane, according to some examples of the present disclosure. [Figure 5A] FIG. 2 illustrates an example surface plane and content placement indicators rendered at fixed positions relative to a complex surface, in accordance with certain examples of the present disclosure. [Figure 5B] FIG. 1 illustrates an exemplary rendering of virtual content at a fixed position relative to a complex surface, in accordance with some examples of the present disclosure. [Figure 5C] FIG. 13 illustrates another exemplary surface plane and content placement indicators rendered at fixed positions relative to a complex surface, in accordance with some examples of the present disclosure. [Figure 5D] FIG. 13 illustrates another exemplary rendering of virtual content at a fixed position relative to a complex surface, in accordance with some examples of the present disclosure. [Figure 6]A diagram showing an example rendering of a highlighted surface plane and content placement indicators in a scene having a surface with no available space, in accordance with some examples of the present disclosure. [Figure 7] 1 is a flowchart illustrating an example process for anchoring virtual content to a physical surface, according to some examples of the present disclosure. [Figure 8] FIG. 1 illustrates an example system architecture for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purposes of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0025] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application as set forth in the appended claims.

[0026] As previously described, an extended reality (XR) device may enable a user to experience or interact with an immersive virtual environment or content. To provide a realistic XR experience, the XR device may integrate virtual content with a physical world scene or image captured by the XR device (e.g., via one or more cameras of the XR device). In some cases, the XR device may perform three-dimensional registration of virtual and real objects. In some cases, the XR device may match the relative pose and movement of an object in the physical world and the XR device. For example, the XR device may use tracking information to calculate a relative pose of a map of the XR device, the object, and / or the real-world environment in order to match the relative positions and movements of the XR device, the object, and / or the real-world environment. Using the pose and movement of the XR device, the one or more objects, and / or the real-world environment, the XR device may convincingly render the content with respect to the real-world environment. The relative pose information may be used to match the virtual content with the user's perceived actions and the spatiotemporal state of the XR device, the object, and the real-world environment.

[0027] When presenting virtual content to a user within a physical world scene or image, the XR device can configure the virtual content so that the rendered virtual content appears to the user as if it were located within a certain space / region in the physical world. In some cases, it may be difficult for the XR device to determine which spaces / regions in the physical world are suitable (e.g., not occluded and / or occupied by objects in the physical world, not cluttered, etc.) and / or have suitable surfaces on which to depict / render the virtual content. In many cases, the XR device may have difficulty managing and / or organizing the virtual content rendered by the XR device within a particular environment and / or avoiding problems when presenting virtual content within a particular environment, such as, for example, occlusions caused by objects in the environment and / or aspects of the environment (e.g., the layout of the environment, the configuration of surfaces and / or spaces in the environment, the placement of any objects in the environment, etc.), clutter caused by the rendered virtual content and objects in the environment.

[0028] In some examples, the XR device may attempt to organize the virtual content rendered by the XR device by arranging the virtual content within a certain space on the physical world. For example, the XR device may attempt to organize the virtual content by attaching or anchoring the virtual content to a certain area / region in the physical world. In some cases, the XR device may receive a user input requesting a certain organization and / or arrangement of the virtual content. In response to the user input, the XR device may attach or anchor the virtual content to a certain area / region in the physical world. In other words, the virtual content may be rendered by the XR device to have a fixed spatial relationship to a particular space / region in the physical world scene. The particular space / region may correspond to one or more real objects in the physical world. As described in more detail below, the correspondence may be by association with a surface of one or more objects. In some examples, the three-dimensional orientation of a surface in the physical world scene may be used to render a perspective view of the virtual content such that the virtual content appears aligned with the surface in three dimensions. In some cases, virtual content may be anchored to a surface such that a fixed spatial relationship between the rendered virtual content and the surface is maintained under user and / or view movement. In some cases, the perspective view of the virtual content is adapted to changes in the user's view and / or position within the physical world.

[0029] As described further herein, in some examples, the XR device may depict content placement indicators that suggest and / or identify suitable and / or suggested areas or regions for placing virtual content within a physical world scene. In some cases, a user may manually move a content placement indicator of the virtual content (e.g., via user input provided to the XR device). In some examples, the content placement indicator may be moved in response to user input (e.g., a user dragging a virtual content item across a scene). The virtual content item may be initially rendered in a first perspective, such as a floor plan, at a first location within the scene. The first location may be predefined or may be manually determined by the user. The virtual content item may be rendered at an initial size, which may be predefined and / or user defined. The initial size may be adapted through user input and / or in response to user input (e.g., drag input, etc.). The content placement indicators may represent and / or identify areas or regions depicted within surfaces and / or surface planes within the physical environment where some virtual content may be placed and rendered by the XR device. A surface plane may be a virtual surface plane associated with an actual surface of an object in the physical environment. In some cases, the surface plane may be aligned with the surface of the object. In some cases, the surface plane may be offset from the surface of the object. The offset may be toward the user's viewpoint. In other cases, the surface plane may coincide with the surface of the object.

[0030] The content placement indicators may include visual elements such as visual contours, windows, hatches, shapes (e.g., bounding boxes or any other shapes) that indicate at least a portion of a particular space of a surface (and / or surface plane) of the physical environment where the virtual content item may be placed / rendered for the user. In some cases, the content placement indicators may include visual elements that depict the fit and placement of the virtual content item onto a particular space of a surface in the physical environment. The content placement indicators (e.g., visual elements) may have the same or similar shape and / or configuration as the virtual content. Thus, the content placement indicators (e.g., visual elements) may depict / render to the user how and / or where the virtual content appears within and / or relative to the physical environment.

[0031] When rendering virtual content in a physical environment, the XR device may position and render the virtual content at / in a content placement indicator. To a user, the virtual content may appear to be located in a space (e.g., an area or region) in the physical environment that is associated with, e.g., outlined or bounded by, the content placement indicator. In some cases, the process of manually moving a content placement indicator of virtual content may be cumbersome and imprecise. For example, when a user wants to attach or anchor virtual content to a surface in a physical environment that is cluttered (e.g., crowded with objects, cluttered, has limited available space, etc.) and / or complex (e.g., uneven, undulating / rough / bumpy / textured, elevated, irregular, asymmetric, curved, three-dimensional, of fixed geometry, etc.), it may be difficult to estimate whether the virtual content will fit (e.g., at a minimum size for readability of text content) within the surface and / or within any suitable space within the surface (e.g., space not occupied or blocked by objects, space that can fit at least a threshold amount of virtual content, etc.). In some cases, it may be difficult to determine whether virtual content may better fit on a particular surface or surface plane, or a different surface or surface plane, or whether placing virtual content within a particular surface may occlude any objects within that surface that the user may not want to be occluded by the virtual content (e.g., the user may want the virtual content to remain visible to the user after it has been placed and rendered on the surface).

[0032] Systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively "systems and techniques") for placing virtual content on and / or with respect to surfaces in a physical environment are described herein. In some examples, an XR device may identify surface planes (planes on / of or with respect to respective surfaces of one or more objects) in the physical environment on which a user may anchor (e.g., place / attach) virtual content for rendering by the XR device. To this end, the XR device may identify one or more objects in the physical environment, determine a relative pose of the one or more objects with respect to the XR device, and identify one or more surfaces of the one or more objects. The XR device may provide an indication and / or suggestion of the identified surface plane to the user to assist the user in selecting and / or accepting a particular surface plane on which to place / draw the virtual content. The indication may be in the form of a content placement indicator that may highlight all or only a portion of the surface plane, or may indicate all or only a portion of the surface plane for placement of the virtual content, e.g., by a visual outline. The XR device can render the virtual content in such a manner as to depict the virtual content on any of such surface planes selected and / or accepted by the user for placement and rendering of the virtual content.

[0033] For example, in some cases, as a user of an XR device moves a rendering of the content placement indicator such that the user sees as if the content placement indicator is being moved through / across a scene in the physical environment, the XR device can detect and highlight any suitable surface planes in the physical environment to which the virtual content may be anchored for rendering. The content placement indicator can represent a space on a surface plane for placement of the virtual content in the physical environment. In some examples, the content placement indicator can represent and / or identify the contour, shape, and / or configuration of the virtual content to indicate the corresponding space of the surface plane in the physical environment that is occupied by the virtual content (e.g., used to render the virtual content) when the virtual content is snapped (e.g., placed at / in, anchored at, etc.) to the content placement indicator at the depicted / identified location of the content placement indicator. The user can use the content placement indicator to determine where to place the virtual content within the surface plane in the physical environment.

[0034] In some examples, a user may select and / or accept to place virtual content within the content placement indicator at a corresponding location (e.g., a depicted / identified location) of the content placement indicator within a surface plane that may be highlighted by the XR device. For example, a user may select a current location (e.g., a depicted / identified location) of the content placement indicator to anchor the virtual content to the content placement indicator at the selected location of the content placement indicator within a scene or image of the physical environment (e.g., within the surface plane). The XR device may render the virtual content within the content placement indicator at the selected location of the content placement indicator within the scene of the physical environment (e.g., within the surface plane). For example, the XR device may render the virtual content within the content placement indicator at a particular location of the surface plane that corresponds to the selected location of the content placement indicator within the content placement indicator.

[0035] In some cases, the XR device may highlight (or in some cases visually depict / identify) a surface plane(s) in the physical environment. In some examples, the XR device may highlight all or a portion of the surface plane(s) to provide a content placement indicator. Alternatively or additionally, (further) content placement indicators, e.g., in the form of visual contours, may be rendered in / on the surface plane(s) to indicate a particular space of the surface plane for placement of a virtual content item. As the XR device moves the content placement indicator through / across the scene of the physical environment (e.g., based on user input, device settings, user preferences, and / or dynamic adjustment determinations), the XR device may change the highlighted surface plane (and / or the highlighted portion of the surface plane). In some examples, the XR device may identify several candidate surface planes in the physical environment and highlight a particular surface plane in the physical environment. As the XR device moves the content placement indicator through / across the scene of the physical environment, the XR device may highlight a surface plane that is different from the candidate surface plane.

[0036] The XR device may highlight the entire surface plane or a portion of the surface plane. In some examples, the XR device may highlight the entire surface plane and present the user with content placement indicators within the highlighted surface plane. In some cases, the presented content placement indicators may include a smaller highlight (e.g., a highlighted portion / region) within the highlighted surface plane, a visual indicator within the highlighted surface plane, a projected slot / area within the highlighted surface plane, a visual outline (and / or boundary) of the virtual content within the highlighted surface plane, and / or a representation of the virtual content placement within the highlighted surface plane. The content placement indicators (e.g., a smaller highlight, a visual indicator, a projected slot / area, a visual outline, a representation of the virtual content placement, etc.) may indicate / identify where the virtual content may be placed within the highlighted surface plane. In some cases, the content placement indicators may depict / identify where and / or how the virtual content is placed and / or fits within the highlighted surface plane.

[0037] In some cases, when the user accepts (e.g., via user input) the placement of the content placement indicator at the indicated location (e.g., proposed and / or depicted location) in the highlighted surface plane and / or releases the virtual content item (e.g., during / after placing / re-positioning the content placement indicator) at the indicated location, the XR device can snap (e.g., anchor / attach / position) the content placement indicator to the indicated location in the highlighted surface plane. In some examples, the XR device can match (e.g., align, size / resize, fit, place / re-position, fit, etc.) the virtual content to the content placement indicator in the indicated location in the highlighted surface plane. For example, the XR device can place (e.g., orient / rotate and / or location) the virtual content along various dimensions in space (e.g., along the X-axis, Y-axis, and / or Z-axis in space) according to the position of the content placement indicator at the indicated location. To illustrate, the XR device may place the virtual content in space to match the position of the virtual content to the position of the content placement indicator (and / or associated plane) at the indicated location. In some cases, the XR device may size / scale the virtual content according to the size / scale of the content placement indicator. For example, the XR device may adjust the size / scale of the virtual content according to the size / scale of the content placement indicator such that the virtual content item fits within the content placement indicator.

[0038] In some cases, prior to sizing / scaling, positioning, and / or positioning the virtual content as described above, the XR device may configure (e.g., place, size / scale, shape, etc.) the content placement indicators to fit within the space of the highlighted surface plane associated with the indicated location and / or to match the spatial configuration of the highlighted surface plane (e.g., location, orientation / rotation, perspective, size / scale, shape / geometry, etc.). For example, the XR device may configure or reconfigure (e.g., place, size / scale, shape / reshape, etc.) the content placement indicators to fit (and / or fit) within the space of the highlighted surface plane associated with the indicated location and / or to match the spatial configuration of the highlighted surface plane (e.g., location, orientation / rotation, perspective, size / scale, shape / geometry, etc.). The XR device can then configure or reconfigure (e.g., place / relocate, size / scale, shape / reshape, etc.) the virtual content according to the configuration (e.g., location, orientation / rotation, perspective, size / scale, shape / geometry, etc.) of the content placement indicators. The XR device can configure or reconfigure the virtual content such that it fits (and / or fits) within the content placement indicators, and the virtual content may also be anchored to the space of a highlighted surface plane associated with the indicated location and / or configured according to the configuration of the space of a highlighted surface plane associated with the indicated location.

[0039] In some cases, if the surface associated with the highlighted surface plane is cluttered (e.g., crowded with objects, cluttered, has limited available space, etc.), the XR device can occlude (e.g., cover / hide, remove, etc.) at least a portion of the clutter from the highlight (and / or the highlighted surface plane) for easier placement and / or viewing of virtual content within the highlighted surface plane. Additionally or alternatively, if the surface associated with the highlighted surface plane is complex (e.g., uneven, undulating / rough / bumpy / textured, elevated, irregular, asymmetric, curved, three-dimensional, rare shape, etc.), the XR device can adjust the highlighted surface plane based on a reconstructed (e.g., smoothed / flattened, reshaped, etc.) surface associated with the highlighted surface plane for easier placement and / or viewing of virtual content within the highlighted surface plane. The XR device can place and render the virtual content within the adjusted surface plane. In some cases, if the surface associated with the highlighted surface plane is complex, the XR device may place and / or render the virtual content (and / or content placement indicators) on a raised surface plane that is located / located above that surface (e.g., at a threshold distance / height above that surface) and parallel to the located (reconstructed) surface for easier placement and / or viewing of the virtual content (and / or content placement indicators) within the plane.

[0040] In some examples, the XR device may render the content placement indicator in the highlighted surface plane with the same aspect ratio as the virtual content item associated with the content placement indicator. In some cases, the XR device may resize the content placement indicator and the virtual content item associated with the content placement indicator to fit within one or more available spaces in the highlighted surface plane. In some examples, as the user moves the virtual content item across the highlighted surface plane, the XR device may move the content placement indicator to available spaces in the highlighted surface plane (and possibly other highlighted surface planes). In some cases, as the user moves the virtual content item across the highlighted surface plane, the XR device may only move the content placement indicator to available spaces that can fit the content placement indicator (e.g., at least a threshold amount of the content placement indicator fits) when moving the content placement indicator. When moving the content placement indicator, the XR device may skip cluttered or unavailable areas in the highlighted surface plane and move the content placement indicator toward the next available space or adjacent available space in the highlighted surface plane (or another available surface plane). In some examples, the XR device may move the content placement indicator toward the next / adjacent available space in the same direction that the user (and / or the XR device) is moving the virtual content item. In some cases, the user may provide input to the XR device to release and / or anchor the content placement indicator to an available space in the highlighted surface plane. For example, when the user releases the virtual content item (e.g., through user input), the XR device may snap the content placement indicator to the space / area of ​​the surface plane that corresponds to the location where the user released the virtual content item.When snapping a content placement indicator to a location, the XR device may match the configuration of the content placement indicator (e.g., orientation / rotation, perspective, size / scale, shape, etc.) to the configuration of the space associated with that location.

[0041] In some cases, if the surface is too cluttered or uneven (e.g., above a threshold), the XR device may render the highlighted surface plane a distance from (e.g., above, in front of, etc.) a higher point on the surface (and / or portions thereof). When the user releases virtual content on the surface, the XR device may place the virtual content at the same level as the highlighted surface plane. If there are items on the surface that are a certain distance / above (e.g., a threshold amount higher) than the average height of the rest of the surface and / or the height of other parts of the surface, the XR device may occlude or ignore those items so that the highlighted surface plane is not placed too far above or in front of the rest of the surface.

[0042] In some cases, a physical environment may have multiple surfaces on which the content placement indicators and / or virtual content items may be rendered. The XR device may highlight a particular surface plane as the user or the XR device moves the content placement indicator or virtual content item. In some cases, to prevent flickering of the highlighted surface plane when the user or the XR device moves the content placement indicator or virtual content item between surface planes (and / or when the content placement indicator or virtual content item overlaps multiple surface planes), the XR device may wait to highlight the surface plane (or may not highlight the surface plane) until a threshold amount of the content placement indicator or virtual content item crosses the surface plane (e.g., from the user's perspective).

[0043] In some cases, if the XR device determines that there is no space available for the virtual content item in that aspect ratio or that the available space is not optimal for placing the virtual content, the XR device can suggest a different aspect ratio that maximizes the available space. For example, the XR device can adjust the aspect ratio of the content placement indicator or the virtual content item to fit or match the available space. The XR device can render the content placement indicator or the virtual content item on the available space according to the adjusted aspect ratio. For example, if the virtual content is configured according to a wide aspect ratio and the available space does not match or cannot fit into the wide aspect ratio, the XR device can render the content placement indicator in the available space according to a narrower aspect ratio. The content placement indicator rendered with the narrower aspect ratio can indicate to the user that the virtual content may be rendered on the available space according to the narrower aspect ratio. When the user releases the virtual content or places the virtual content at the location of the content placement indicator, the XR device can change the virtual content to the narrower aspect ratio and render the virtual content at that location according to the narrower aspect ratio.

[0044] In some cases, an XR device may implement the techniques described herein in the context of a virtual reality (VR) session. For example, an XR device may implement content placement indicators in a video pass-through VR application in which the XR device captures an image(s) of a physical environment (e.g., including one or more surfaces in the physical environment) and uses the captured image(s) to render the physical environment (e.g., including one or more surfaces) in a VR display for the purpose of providing visibility of the physical environment around the user. Here, the XR device can use the captured image(s) of the physical environment to render content placement indicators in the physical environment rendered / depicted by the XR device.

[0045] Examples of systems and techniques described herein for processing data are shown in Figures 1-8 and described below. Figure 1 illustrates an exemplary extended reality (XR) system 100, according to some examples of the present disclosure. The XR system 100 may implement the systems and techniques disclosed herein. The XR system 100 may perform various tasks and operations, such as, for example, extended reality tasks and operations (e.g., tracking, mapping, localization, content rendering, pose estimation, object detection / recognition, etc.), image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, machine vision, object modeling and alignment, multimedia rendering and / or compositing, and / or any other data processing tasks, effects, and / or calculations.

[0046] As used herein, placing a virtual content item on or in a surface or space in a scene of a physical environment means placing or anchoring a virtual content item on or in a location or area in a three-dimensional (3D) map of the scene (location or area) corresponding to the surface or space such that the virtual content item appears to a user of the XR system 100 as if the virtual content item were located on or in the surface or space in the scene. Similarly, as used herein, placing a content placement indicator on or in a surface or space in a scene of a physical environment means placing or anchoring a content placement indicator on or in a location or area in a 3D map of the scene (location or area) corresponding to the surface or space such that the virtual content item appears to a user of the XR system 100 as if the virtual content item were located on or in the surface or space in the scene.

[0047] In some examples, the XR system 100 can perform tracking and localization, pose estimation, mapping of the physical world / environment (e.g., scene) around the XR system 100 (e.g., in which the XR system 100 is located), placement and rendering of virtual content on a screen, display, and / or viewable plane / area as part of the XR experience. For example, the XR system 100 can generate a map (e.g., a three-dimensional (3D) map) of a scene in the physical world, track the pose (e.g., location and orientation) of the XR system 100 relative to the scene (e.g., relative to the 3D map of the scene), place and / or anchor virtual content in specific location(s) on the map of the scene (e.g., corresponding to physical locations in the scene, such as a surface, a surface plane, a space within a surface or surface plane), and render the virtual content item and / or virtual content placement indicator on the display / screen such that the virtual content item and / or virtual content placement indicator appears to be at a physical location in the scene that corresponds to the specific location on the map of the scene where the virtual content item is placed and / or anchored.

[0048] In the example shown in FIG. 1, the XR system 100 includes one or more image sensors 102, one or more inertial sensors 104 (e.g., one or more inertial measurement units), one or more other sensors 106 (e.g., one or more radio detection and ranging (radar) sensors, light detection and ranging (LIDAR) sensors, acoustic / sound sensors, infrared (IR) sensors, magnetometers, touch sensors, laser range finders, light sensors, proximity sensors, motion sensors, active pixel sensors, machine vision sensors, ultrasonic sensors, etc.), storage 108, compute components 110, an XR engine 120, an interface management engine 122 (e.g., a user interface management engine), an image processing engine 124, and a rendering engine 126. It should be noted that the components 102-126 shown in FIG. 1 are non-limiting examples provided for purposes of illustration and explanation, and other examples may include more, fewer, and / or different components than those shown in FIG. 1. For example, in some cases, the XR system 100 may include one or more display devices, one or more other processing engines, one or more receivers (e.g., global positioning systems, global navigation satellite systems, etc.), one or more communication devices (e.g., radio frequency (RF) interfaces and / or any other wireless / wired communication receivers / transmitters), one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 1. An example architecture and example hardware components that may be implemented by the XR system 100 are further described below in conjunction with FIG. 8.

[0049] Further, for simplicity and explanation, one or more image sensors 102 will be referred to herein (e.g., in the singular) as image sensor(s) 102. However, as previously mentioned, the XR system 100 may include a single image sensor or multiple image sensors. Additionally, references to any of the components of the XR system 100 in the singular or plural should not be construed as limiting the number of such components implemented by the XR system 100 to one or more than one. For example, references to a processor in the singular should not be construed as limiting the number of processors implemented by the XR system 100 to one. Those skilled in the art will recognize that for any of the components shown in FIG. 1, the XR system 100 may include only one of such component(s) or more than two of such component(s).

[0050] The XR system 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the XR system 100 may be part of an electronic device (or devices), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a game console, a video streaming device, an Internet-of-Things (IoT) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device(s).

[0051] In some implementations, one or more of the image sensor 102, the inertial sensor(s) 104, the other sensor(s) 106, the storage 108, the compute component 110, the XR engine 120, the interface management engine 122, the image processing engine 124, and the rendering engine 126 may be part of the same computing device. For example, in some cases, one or more of the image sensor 102, the inertial sensor(s) 104, the other sensor(s) 106, the storage 108, the compute component 110, the XR engine 120, the interface management engine 122, the image processing engine 124, and the rendering engine 126 may be integrated into a smartphone, a laptop, a tablet computer, a smart wearable device, a gaming system, and / or any other computing device. In other implementations, one or more of the image sensor 102, the inertial sensor(s) 104, the other sensor(s) 106, the storage 108, the compute component 110, the XR engine 120, the interface management engine 122, the image processing engine 124, and the rendering engine 126 may be part of two or more separate computing devices. For example, in some cases, some of the components 102-126 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0052] The image sensor(s) 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, the image sensor(s) 102 may be part of a camera or computing device, such as an XR device (e.g., HMD, smart glasses, etc.), a digital camera, a smartphone, a smart television, a gaming system, etc. In some examples, the image sensor(s) 102 may be part of a multiple camera assembly, such as a dual camera assembly. The image sensor(s) 102 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by the compute component 110, the XR engine 120, the interface management engine 122, the image processing engine 124, and / or the rendering engine 126 described herein.

[0053] In some examples, the image sensor(s) 102 can capture image data, generate frames based on the image data, and / or provide image data or frames to the XR engine 120, the interface management engine 122, the image processing engine 124, and / or the rendering engine 126 for processing. A frame can include a video frame of a video sequence or a still image. A frame can include a pixel array representing a scene. For example, a frame can be a red-green-blue (RGB) frame with red, green, and blue color components per pixel, a luma, red-difference, blue-difference (luma, chroma-red, chroma-blue (YCbCr) frame with one luma component and two chrominance (color) components (red chrominance and blue chrominance) per pixel, or any other suitable type of color or monochrome image.

[0054] The inertial sensor(s) 104 can detect certain forces and angular velocities of the XR system 100. In some cases, the inertial sensor(s) 104 can detect the orientation of the XR system 100. The inertial sensor(s) can generate linear acceleration measurements, rotational velocity measurements, and / or heading measurements. In some examples, the inertial sensor(s) 104 can be used to measure the pitch, roll, and yaw of the XR system 100. The other sensor(s) 106 can detect and generate other measurements used by the XR system 100. In some cases, the XR engine 120 can use data and / or measurements from the image sensor(s) 102, the inertial sensor(s) 104, and / or the other sensor(s) 106 to track the attitude of the XR system 100. As mentioned above, in other examples, the XR system 100 may include other sensors such as magnetometers, acoustic / sound sensors, IR sensors, machine vision sensors, smart scene sensors, radar sensors, LIDAR sensors, light sensors, etc.

[0055] The storage device 108 may be any storage device or devices that store data. Moreover, the storage device 108 may store data from any of the components of the XR system 100. For example, the storage device 108 may store data (e.g., image data or video data) from the image sensor(s) 102, data (e.g., measurements) from the inertial sensor(s) 104, data (e.g., measurements) from the other sensor(s) 106, data (e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, configurations, XR application data, recognition data, output, etc.) from the compute component 110, data from the XR engine 120, data from the interface management engine 122, data from the image processing engine 124, and / or data (e.g., output frames) from the rendering engine 126. In some examples, the storage device 108 may include a buffer that stores frames for processing by the compute component 110.

[0056] The one or more compute components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The compute components 110 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, object detection, and any of the various operations described herein. In the example shown in FIG. 1, the compute components 110 may implement an XR engine 120, an interface management engine 122, an image processing engine 124, and a rendering engine 126. In other examples, the compute component 110 may also implement one or more other processing engines.

[0057] The operations for the XR engine 120, the interface management engine 122, the image processing engine 124, and the rendering engine 126 (as well as any other processing engines) may be implemented by any of the compute components 110. In one illustrative example, the operations of the rendering engine 126 may be performed by the GPU 114, and the operations of the XR engine 120, the interface management engine 122, and the image processing engine 124 may be performed by the CPU 112, the DSP 116, and / or the ISP 118. In some cases, the compute components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof that performs any of the various operations described herein.

[0058] In some examples, the XR engine 120 can perform XR operations based on data from one or more sensors on the XR system 100, such as the image sensor(s) 102, the inertial sensor(s) 104, the other sensor(s) 106, and / or one or more LIDAR, radar, ultrasonic, IR sensors, etc. In some examples, the XR engine 120 can perform tracking, localization, pose estimation, mapping, content anchoring operations, and / or any other XR operations / functions.

[0059] The interface management engine 122 can manage and / or generate interface content presented / rendered by the XR system 100, such as XR interfaces and / or interface objects. For example, the interface management engine 122 can manage and / or generate graphical user interfaces, control objects, visual indicators such as content placement indicators described herein, surface and / or surface plane highlighting described herein, interface elements, and the like. In some examples, the interface management engine 122 can perform various operations to determine and / or manage how, where, and / or when to render user interfaces, visual indicators, control objects, and the like during an XR experience. An XR experience can include using the XR system 100 to present XR content (e.g., virtual reality content, augmented reality content, mixed reality content, and the like) to a user associated with the XR system 100 (e.g., a user wearing the XR system 100 and / or, in some cases, a user using the XR system 100 for an XR experience).

[0060] In some examples, XR content and experiences may be provided by XR system 100 through XR applications that provide specific XR experiences, such as, for example, XR gaming experiences, XR classroom experiences, XR shopping experiences, XR entertainment experiences, XR activities (e.g., motion, troubleshooting activities, etc.), among others. During an XR experience, a user may view and / or interact with virtual content using XR system 100. In some cases, a user may view and / or interact with virtual content while also viewing and / or interacting with the physical environment around the user, allowing the user to have an immersive experience between the physical environment and the virtual content mixed or integrated with the physical environment.

[0061] In some examples, the interface management engine 122 can use data from the image sensor(s) 102, the inertial sensor(s) 104, the other sensor(s) 106, and / or any other sensors to detect objects in the scene (e.g., edges, surfaces, items on surfaces, windows, doors, walls, tables, books, devices, chairs, etc.), planes in the scene, identify characteristics of the scene and / or objects in the scene, identify occlusions in the scene, etc. In some examples, the interface management engine 122 can also use data from other devices or applications, such as data from or reported by other user devices and / or data provided by a user of the XR system 100.

[0062] In some examples, the interface management engine 122 can coordinate with the XR engine 120 and / or the rendering engine 126 to render user interfaces, control objects, content placement indicators, surface highlighting, surface plane highlighting, and the like. As described further herein, in some examples, the interface management engine 122 can detect surfaces in a scene on which virtual content may be placed for rendering, and the XR system 100 can visually identify such surfaces to the user. For example, the interface management engine 122 can determine the configuration (e.g., size, shape, orientation, perspective, location, slope, etc.) of a surface in a scene, detect any objects located on the surface, and determine whether there is any available space in a surface plane associated with, e.g., aligned with, and / or offset from, the surface on which a particular virtual content item may be placed for rendering. The available space in the surface plane can be determined as such space that is not occluded by any of the detected objects. For surface planes that are offset from the surface, depending on the value of the offset, some or all of the object may be occluded by the surface plane, such that by changing the value of the offset, the available (unoccluded) space may be modified.

[0063] In some cases, the XR system 100 (e.g., using the XR engine 120, the interface management engine 122, the image processing engine 124, and / or the rendering engine 126) may visually identify a surface plane to a user (e.g., via highlighting, hatching, outlining, changing visual attributes of the depicted surface to identify the surface as available, etc.), and the interface management engine 122 may depict a content placement indicator that appears to the user to be located within the available / unoccluded space on the surface plane. The content placement indicator may visually identify to the user the space available as a candidate location for placing and rendering a particular virtual content item. The interface management engine 122 may configure the content placement indicator based on the particular virtual content item. For example, the interface management engine 122 may determine the size and shape of a particular virtual content item, e.g., based on prior renderings of the virtual content item and / or user input, and configure the content placement indicator to have the size and shape of the particular virtual content item. In some cases, interface management engine 122 can determine a configuration of the content placement indicators and associated virtual content items to enable them to fit within the available space and / or maintain a constant position, orientation, etc. relative to the available space and / or surface plane. For example, interface management engine 122 can change the shape and / or aspect ratio of the content placement indicators and associated virtual content items if / as needed to fit within the available space.

[0064] When the XR system 100 renders the content placement indicator to the user, the content placement indicator can indicate how the virtual content item would look if it were placed in the available space depicted to the user (e.g., anchored to the available space in the mapped scene and rendered to appear as if the virtual content item were located in the available space). After the content placement indicator is rendered to the user, the user can accept or reject the available space as a location for placement of the virtual content (e.g., via user input, such as an input gesture, a gaze interpreted as an input, an input using a controller, or any other input means, such as releasing a dragged virtual content item). If the user accepts the available space as a location for placement of the virtual content, the XR system 100 can anchor the virtual content item to the available space and render the virtual content item to appear as if it were located in the available space. If the user rejects the available space, the interface management engine 122 can coordinate with the XR system 100 to render another content placement indicator in a different available space (either in the surface plane or in a different surface plane).

[0065] In some examples, when a user attempts to move a content placement indicator to a different location / space, for example, by moving or dragging a virtual content item across a scene, the interface management engine 122 can coordinate with the XR system 100 to render the content placement indicator in the different available space. In some cases, the different available space may be an adjacent space. In other cases, if the adjacent space is unavailable (e.g., there are one or more objects and / or occlusions in the space), the interface management engine 122 can skip the adjacent space (e.g., refrain from placing and rendering the content placement indicator in the adjacent space) and coordinate with the XR system 100 to render the content placement indicator in a different, non-adjacent, available space.

[0066] The image processing engine 124 can perform one or more image processing operations. In some examples, the image processing engine 124 can perform image processing operations based on data from the image sensor(s) 102. In some cases, the image processing engine 124 can perform image processing operations such as, for example, filtering, demosaicing, scaling, color correction, color conversion, segmentation, noise reduction filtering, spatial filtering, artifact correction, etc. The rendering engine 126 can take image data generated and / or processed by the compute component 110, the image sensor(s) 102, the XR engine 120, the interface management engine 122, and / or the image processing engine 124, and render content (e.g., virtual content, videos, images, etc.) for presentation on a display device.

[0067] Although the XR system 100 is shown to include certain components, one skilled in the art will appreciate that the XR system 100 may include more or less components than those shown in Figure 1. For example, the XR system 100 may optionally include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Illustrative examples of computing devices and hardware components that may be implemented using the XR system 100 are described below with respect to Figure 8.

[0068] 2A illustrates an example scene 200 with various surfaces available for placing virtual content for rendering by the XR system 100. In this example, the scene 200 is a composite physical environment with a virtual content item 202 being rendered by the XR system 100 within the scene. In particular, the virtual content item 202 is placed on a physical surface 214 (in this example, a wall) within the scene 200 and is rendered to appear to the user as if the virtual content item 202 were actually on the physical surface 214. The scene 200 also includes other surfaces, such as surface 212, which in this example is the top surface of a table 210, and surface 216, which in this example is the side of a bookshelf.

[0069] Furthermore, since the user of the XR system 100 is rendering another virtual content item, i.e., virtual content item 204, in its original configuration / perspective at a first location and in a first perspective somewhere in the foreground on the scene 200, the virtual content item 204 appears to actually be on the scene 200. To assist the user in identifying the placement of the virtual content item 204 within the scene 200, the XR system 100 may identify surfaces and / or spaces within the scene 200 that are available for placement of the virtual content item 204. For example, as the user moves the virtual content item 204 via the input element 220, the XR system 100 may identify to the user whether a particular surface and / or space within a surface is available for placement of the virtual content item 204. The input element 220 may include any means of providing input to the XR system 100, such as, for example, hand gestures (e.g., pointing a finger, moving a hand, and / or placing a hand in a certain way, and / or any other hand motion or posture) that can be interpreted as input by the XR system 100, head gestures (e.g., certain motions and / or postures of the user's head that are interpreted as input by the XR system 100, etc.), gaze (e.g., eye movement and / or gaze to a location that the XR system 100 may interpret as input), input via an input device (e.g., a controller, trackpad, mouse, etc.), and / or any other input technique and / or device. In this example, the input element 220 includes a ray projected by ray casting.

[0070] As shown, the XR system 100 has identified a surface plane 218 as an available surface plane for placing the virtual content item 204. The user can then use input elements 220 to place (e.g., snap, anchor, move) the virtual content item 204 within the surface plane 218 identified by the XR system 100. The surface plane 218 can include the actual surface 212 of the table 210 or a plane relative to (e.g., parallel to) the surface 212 of the table 210.

[0071] 2A, the XR system 100 renders a content placement indicator on the surface plane 218 by highlighting the surface plane 218 (e.g., rendering a highlighted surface plane) to indicate that the surface plane 218 is available for placement and rendering of the virtual content item 204. In other examples, the XR system 100 may additionally or alternatively indicate the surface plane 218 as available for placement in other ways. For example, in some cases, the XR system 100 may render an outline around the surface plane 218 as a content placement indicator, render the surface plane 218 with a visual pattern (e.g., color, gradient, shading, transparency, fill, line, text, shadow, reflection, glow, soft edges, virtual object, etc.) as a content placement indicator, render a visual indicator (e.g., arrow, text, video, image, visual effect, etc.) as a content placement indicator, and / or provide any other rendering or visualization that may indicate the surface plane 218 as available.

[0072] FIG. 2B illustrates an example scene 200 from FIG. 2A with various physical objects on the surface 212 of the table 210. In some examples, the XR system 100 can help the user avoid placing (or refrain from suggesting placing) the virtual content item 204 in a space in the surface plane 218 that is occupied (occluded) by one or more objects on the surface 212, and / or assist in finding available space in the surface plane 218. For example, the XR system 100 can detect any available space in the surface plane 218, such as any space that is not occupied (occluded) by objects on the surface 212 of the table 210 and / or any space that is not cluttered. As described further below, based on the detection of any space that is not occluded by objects, the XR system 100 can detect any available space in the surface plane that is offset from the associated surface in a direction perpendicular to that surface. Such objects can be objects on the surface 212, or any other objects in the scene.

[0073] In some examples, the XR system 100 can perform surface detection to detect one or more surfaces in the physical environment. The XR system 100 can also perform edge detection to determine any edges of any surfaces and / or objects in the physical environment. The XR system 100 can determine the pose of the XR system 100 in the physical environment. For example, the XR system 100 can determine the pose, orientation, and / or perspective of the XR system 100 based on any detected edges, surface planes, and / or objects in the physical environment. In some examples, the XR system 100 can perform object detection to detect objects in the scene 200. For example, the XR system 100 can perform object detection to detect objects on the surface 212 of the table 210 in the scene 200. The XR system 100 can locate and map any detected objects, including any detected objects in the surface 212, to track and / or understand their location in the scene 200 (e.g., in a three-dimensional (3D) map of the scene 200 used by the XR system 100 to provide the XR functionality / experience). The XR system 100 can then determine any space in the surface plane 218 available for placement, for example, by determining the portions of the surface plane 218 that are occluded by the objects, segmenting the surface plane into segments, e.g., rectangular spaces, based on the determined occluded portions such that the segments are not occluded by the objects, and rendering the virtual content item 204 (e.g., any space in the surface plane 218 that is not occupied / occluded by objects on the surface 212 or elsewhere). For example, the XR system 100 can analyze the 3D map of the scene to identify any spaces / locations within the surface plane 218 that are not associated with a mapped object (e.g., to identify those spaces / locations that correspond to the surface plane 218 and that do not have a mapped object associated with any space / location in the 3D map).

[0074] After detecting any space in the surface plane 218 available for placing and rendering the virtual content item 204, the XR system 100 may render one or more content placement indicators indicating one or more available spaces on the surface plane 218. In some cases, the XR system 100 may render only one content placement indicator indicating the available spaces on the surface plane 218 at any one time. The location of this single content placement indicator may be determined based on the proximity to the (anchor point / trip of) the input element 220, the proximity to the representation of the virtual content item in the first perspective that may be moved / dragged by the user input, the overlap between such representation and the respective available space, etc. For example, the XR system 100 may place a content placement indicator 206 on the available space 208 detected by the XR system 100. The content placement indicator 206 may indicate the available space 208 as a candidate location for placing the virtual content item 204. In this way, the content placement indicator 206 indicates to the user that the available space 208 is an available location for placing and rendering the virtual content item 204. In some cases, the content placement indicator 206 indicates to the user a location where the virtual content item 204 will be automatically anchored by the XR system 100 in response to a user input, such as releasing a dragged virtual content item. When the user selects the content placement indicator 206 (e.g., the user accepts placement of the virtual content item 204 on the space 208 of the surface plane 218 identified by the content placement indicator 206), the XR system 100 can place the virtual content item 204 on the space 208 associated with the content placement indicator 206. The XR system 100 can render the virtual content item 204 to appear as if it were on the space 208 associated with the content placement indicator 206.

[0075] In the example shown in FIG. 2B, the XR system 100 renders one content placement indicator. However, in some cases, the XR system 100 may render more than one content placement indicator. For example, the XR system 100 may render the content placement indicator 206 and one or more additional content placement indicators associated with one or more additional spaces available for placing and rendering the virtual content item 204. The one or more additional spaces may include one or more spaces on the surface plane 218 highlighted by the XR system 100. In some cases, the one or more additional spaces may include spaces available on different surfaces / surface planes in the scene. For example, the one or more additional spaces may include spaces available on the surface 214 (which the XR system may or may not highlight for the user) and / or spaces available on the surface 216 (which the XR system may or may not highlight for the user).

[0076] In some cases, when a user moves a virtual content item 204 (e.g., via an input element 220) within the scene 200, the XR system 100 may move the content placement indicator 206 to a particular available space on the surface plane 218. Additionally, when moving the content placement indicator 206, the XR system 100 may skip (e.g., refrain from placing / rendering) an object(s) on the surface 212. For example, when the XR system 100 moves the content placement indicator 206 from an available space 209 on the surface plane 218, rather than placing the content placement indicator 206 on an adjacent space on the surface plane 218 that is currently occupied / occluded by the object 230, the XR system 100 may skip the object 230 and place the content placement indicator 206 on the available space 208 in the surface plane 218. In this manner, the XR system 100 may refrain from placing the content placement indicator 206 on the object 230 (or the space occupied by the object 230). In other words, the XR system 100 can switch from rendering a content placement indicator in a first space 209 of the surface plane 218 to rendering a content placement indicator 206 in a second space 208 of the surface plane 218. By switching from one rendering location to the next, the XR system can ensure that the user is always presented with suggestions for placing virtual content items. As explained above, the suggestion space indicated by the currently rendered content placement indicator can be determined based on proximity to and / or overlap with the virtual content item 204.

[0077] In some examples, the XR system 100 can occlude clutter from the surface plane 218. For example, the XR system 100 can render an occlusion 240 over the cluttered space of the surface 212 to occlude the clutter from the surface plane 218.

[0078] 3A illustrates an example of selecting a surface plane to highlight when the location of a virtual content item 310 overlaps a different surface (or portion thereof) on the scene 200. As described above, the highlighting or graduated highlighting of the surfaces may be used as a respective content placement indicator. As shown in this example, a portion 312 of a virtual content item 310 rendered in a first location and with a first perspective and possibly moved via user input overlaps with a portion of the surface 212, a portion 314 of the virtual content item 310 overlaps with a portion of the surface 216, and a portion 316 of the virtual content item 310 overlaps with a portion of the surface 214. In some examples, rather than highlighting surface planes corresponding to all of the surfaces 212, 214, and 216, the XR system 100 may select a particular surface plane to highlight from the surfaces 212, 214, and 216.

[0079] In some cases, the XR system 100 can determine which surface has the greatest amount of overlap with the virtual content item 310 and select that surface for highlighting (and / or rendering a highlighted surface plane). For example, the XR system 100 can determine which of the portions 312, 314, and 316 of the virtual content item 310 that overlap with the surfaces 212, 214, and 216 is the largest and select the surface plane associated with the largest portion of the virtual content item 204. In this example, the portion 312 is larger than the portions 314 and 316. Thus, the XR system 100 can highlight the surface plane 218 that corresponds to the surface 212 associated with the portion 312.

[0080] In some cases, the XR system 100 can determine whether any of the surfaces 212, 214, and 216 contain a threshold amount or percentage of the virtual content item 310. If a threshold amount or percentage of the virtual content item 310 is on / above the surface, the XR system 100 can select that surface for highlighting (and / or for highlighting a surface plane). For example, the XR system 100 can determine whether any of the portions 312, 314, and 316 of the virtual content item 310 that overlap with the surfaces 212, 214, and 216 meet or exceed a threshold. The threshold can be chosen, for example, 50% or more, such that only a single surface plane is highlighted at a time. The XR system 100 can select a surface associated with a portion of the virtual content item 310 (e.g., a portion from the portions 312, 314, and 316) that meets or exceeds the threshold.

[0081] In some examples, the XR system 100 may toggle the highlighted surface plane, e.g., as a rendered content placement indicator, as the user moves the virtual content item 310 (e.g., via the input element 220). For example, as shown in FIG. 3B, the user is moving the virtual content item 310 such that the portion 314 of the virtual content item 310 that overlaps the surface 216 exceeds a threshold and / or is larger than the portion 312 of the virtual content item 310 that overlaps the surface 212 and the portion 316 of the virtual content item 310 that overlaps the surface 214. The XR system 100 may determine that the portion 314 of the virtual content item 310 that overlaps with the surface 216 exceeds a threshold and / or is larger than the portion 312 of the virtual content item 310 that overlaps with the surface 212 and the portion 314 of the virtual content item 310 that overlaps with the surface 214, and may highlight the surface plane 320 that corresponds to the surface 216 associated with the portion 314 of the virtual content item 310 that exceeds the threshold and / or is larger than the portions 312 and 314 of the virtual content item 310. In this manner, the XR system 100 may switch from rendering a content placement indicator by highlighting the surface plane 218 associated with the surface 212, as shown in FIG. 3A, to rendering a content placement indicator by highlighting the surface plane 320 associated with the surface 216, as shown in FIG. 3B.

[0082] In some cases, to prevent flickering between surface plane highlights, the XR system 100 can set a certain threshold to determine which surface to select as previously described. In some examples, the XR system 100 can set the threshold to at least 60% to prevent flickering between surface plane highlights. For example, the XR system 100 can select a surface when at least 60% of the virtual content item 310 overlaps with the surface / surface plane. To explain, if the portion 314 of the virtual content item 310 includes at least 60% of the virtual content item 310, the XR system 100 can select the surface plane 320 corresponding to the surface 216 that overlaps with the portion 314 of the virtual content item 310.

[0083] In some cases, the XR system 100 may switch between available spaces on the highlighted surface plane as well. For example, when the XR system 100 moves a content placement indicator (e.g., content placement indicator 206) within the highlighted surface plane, if the content placement indicator overlaps with some available spaces on the highlighted surface plane, the XR system 100 may select an available space that has the greatest amount of overlap with the content placement indicator and place / render the content placement indicator within that available space. As another example, when the XR system 100 moves a content placement indicator (e.g., content placement indicator 206) within the highlighted surface plane, if the content placement indicator overlaps with some available spaces on the highlighted surface plane, the XR system 100 may select an available space that has an overlap with the content placement indicator that meets or exceeds a certain threshold and place / render the content placement indicator within that available space. The described movement / switching of the content placement indicator between available spaces may be performed without highlighting the surface plane. The content placement indicator may also be in the form of brightness / color grading of the highlighting of the surface plane indicating the respective available space.

[0084] In some cases, if the XR system 100 determines that there is no space available in the surface plane for the virtual content item of that aspect ratio or that the available space is not optimal for placing the virtual content, the XR system 100 may suggest a different aspect ratio for the virtual content item that maximizes the available space and / or fits within the available space. The suggestion may be in the form of changing the content placement indicator from a first aspect ratio to a second aspect ratio when switching between the available spaces. For example, the XR system 100 may adjust the aspect ratio of the content placement indicator and / or the virtual content item to fit or match the available space. The XR system 100 may render a resized content placement indicator and / or a resized virtual content item on the available space according to the adjusted aspect ratio.

[0085] 4 is a diagram illustrating an example adjustment of the aspect ratio of the content placement indicator 404 to fit within the available space on the surface plane 218. In this example, the surface 212 of the desk 210 in the scene 200 is covered by an object. The XR system 100 has identified the space 420 between the objects 410, 412, 414, 416, and 418 as available space, e.g., via object detection, occlusion detection, and segmentation. However, the virtual content item 402 does not fit within the space 420 with the current aspect ratio of the virtual content item 402 (as rendered in the first perspective). Thus, the XR system 100 can render the content placement indicator 404 within the space 420 according to an aspect ratio narrower than the aspect ratio of the virtual content item 402.

[0086] For example, the XR system 100 can adjust the aspect ratio of the content placement indicator 404 from a wide aspect ratio corresponding to the aspect ratio of the virtual content item 402 to a narrower aspect ratio that allows the content placement indicator 404 to fit within the space 420. The XR system 100 can position the content placement indicator 404 within the space 420 according to the narrower aspect ratio and render the content placement indicator 404 to appear to be located within the space 420 with the narrower aspect ratio. When the user releases the virtual content item 402 and / or snaps (e.g., positions / anchores) the virtual content item 402 to the content placement indicator 404 (and / or the space 420 associated with the content placement indicator 404), the XR system 100 can change the aspect ratio of the virtual content item 402 to match the aspect ratio of the content placement indicator 404 and position the virtual content item 402 within the space 420 according to the adjusted aspect ratio. The XR system 100 may render the virtual content items 402 with an adjusted aspect ratio so that they appear to be located on the space 420. The adjustment of the size and / or aspect ratio of the content placement indicators 404 may be conditional on a minimum height and / or width of the virtual content items after being placed / rendered in the user's current view. In other words, spaces on the surface plane 218 may be determined to be unavailable / unsuitable for rendering virtual content items based on their size (height and / or width) in the current view being smaller than a minimum (height and / or width) threshold. Such spaces are not explicitly indicated by the XR system 100 by rendering a dedicated content placement indicator (such as a gradient highlight and / or visual outline) on the respective space.

[0087] In some examples, the ability to adjust the aspect ratio to position a virtual content item 402 may be enabled by user settings or may be enabled by default. In some cases, the adjusted aspect ratio and / or configuration / parameters of the adjusted virtual content item 402 (and / or the adjusted content positioning indicator) may be defined by user settings or may be specified by the virtual content item 402. For example, in some cases, the adjusted aspect ratio of the virtual content item 402 (and / or whether the aspect ratio of the virtual content item 402 may be adjusted) may depend on the type of virtual content of the virtual content item 402 and / or whether the virtual content item 402 is suitable for or capable of being resized. For example, a virtual content item that includes textual content may represent a type of virtual content that is not suitable for aspect ratio adjustment.

[0088] In some cases, if a surface does not have available space, e.g., unoccluded space having at least a minimum size (and / or has a threshold amount of clutter), or if the surface is a complex surface (e.g., uneven, undulating / rough / bumpy / textured, elevated, irregular, asymmetric, curved, three-dimensional, rare shape, etc.), the XR system 100 may determine a surface plane associated with the surface that is generally aligned with the surface in that it reflects the overall perspective (orientation) of the complex surface. The surface plane may be considered a simplification of the complex surface, e.g., by performing edge detection on the complex surface to determine the perspective of the complex surface, detecting and removing objects on the surface, etc., such as smoothing / flattening, reshaping the surface. The surface plane may be determined as a planar representation of the complex surface for easier placement and / or viewing of virtual content items within the highlighted surface plane. The surface plane may be located at the average surface level of the complex surface, or may be offset by an offset value in a direction perpendicular to the surface plane / simplified complex surface. The offset may be toward the user's line of sight. As a result, the adjusted (offset) surface plane occludes at least a portion of the complex surface, thus making (more) space available for rendering the virtual content item. The XR system 100 may position and render the virtual content item within the adjusted surface plane. In some cases, if the surface is complex, the XR system 100 may position and / or raise / elevate / offset the highlighted surface plane and content placement indicators and render them on a raised / elevated plane relative to (e.g., generally parallel to) the complex surface. The highlighted surface plane / content placement indicators may be located / positioned above the surface (e.g., a threshold distance / height above the surface) for easier placement and / or viewing of the virtual content item (and / or content placement indicator) within the surface plane.

[0089] FIG. 5A illustrates an exemplary surface plane 512 and content placement indicators 504 rendered at a fixed position relative to a complex surface 510. As shown, the surface 510 is created by books on a bookshelf that are of different sizes, so the surface 510 is non-uniform. Thus, the surface 510 is not flat / uniform and does not have available space that is flat or uniform. In some cases, the perspective or orientation of the surface plane 512 may be determined based on edge detection (of books on a shelf) and identifying parallel edges and their common orientation. The XR system 100 may move the surface plane 512 and content placement indicators 504 to be rendered at a fixed distance (offset) from the surface 510 toward the user's viewpoint. For example, the XR system 100 may move the surface plane 512 and content placement indicators 504 to be a threshold distance away (e.g., from the viewpoint of the user of the XR system 100) from the surface 510 (e.g., above it, in front of it, etc.). The XR system 100 can render the surface plane 512 and the content placement indicator 504 at the moved position, which is a threshold distance from the surface 510 .

[0090] 5B , when a user releases the virtual content item 502 on the content placement indicator 504 and / or accepts the placement of the virtual content item 502 in the content placement indicator 504, the XR system 100 may place the virtual content item 502 within the content placement indicator 504 and render the virtual content item 502 at a location associated with the content placement indicator 504 at the time of releasing the virtual content item 502, which is a threshold distance away from the surface 510 as previously described. As previously described, the content placement indicator 504 may be moved across the surface plane 512 based on the movement of the virtual content item 502 in response to user input.

[0091] 5C illustrates another exemplary surface plane 522 and content placement indicators 530 rendered at a fixed position relative to (offset from) a complex surface 520. The surface 520 in this example is a wall constructed of irregularly shaped (e.g., curved) cement blocks creating an irregular / uneven surface. Thus, the surface 520 is not flat, but rather irregular / uneven. As shown, since the surface 520 is irregular / uneven, the XR system 100 can determine the perspective (orientation) of the surface plane 522 based on a simplified (smoothed, flattened, reshaped, etc.) surface. Additionally or alternatively, edge detection can be performed to determine the perspective (orientation) of the surface plane 522. The XR system can move / offset the rendered surface plane 522 and content placement indicators 530 to be / appear a fixed distance / offset in front of the surface 520, i.e., towards the user's viewpoint. For example, the XR system 100 may move the surface plane 522 and the content placement indicators 530 to be a threshold distance in front of the surface 520 (e.g., from the perspective of a user of the XR system 100). The XR system 100 may render the surface plane 522 and the content placement indicators 530 at a moved / offset position that is a threshold distance in front of the surface 520.

[0092] Referring to FIG. 5D , when a user releases the virtual content item 502 on the content placement indicator 530 and / or accepts the placement of the virtual content item 502 within the content placement indicator 530, the XR system 100 may place the virtual content item 502 within the content placement indicator 530 and render the virtual content item 502 at a position associated with the content placement indicator 530 at the time of the release of the virtual content item, which is a threshold distance in front of the surface 520 as previously described.

[0093] FIG. 6 illustrates an example rendering of a highlighted surface plane and content placement indicators in a scene with a surface that has no available space (meeting a minimum size requirement). In this example, the surface 602 of a table is cluttered with items and has no available space (e.g., not occupied / occluded by any items) for placing a virtual content item 610. Because the surface 602 has no available space, the XR system 100 may elevate / raise the highlighted surface plane 604 a certain distance / height above the surface 602 and the content placement indicator 606 rendered on the highlighted surface plane 604. As in the previous example, a table top cluttered with items may be considered a complex surface that may be simplified as described above to determine the associated surface plane. The surface plane 604 may be offset from the average surface level of the composite surface by a certain offset value, as described below.

[0094] In some examples, if there are one or more objects that extend beyond the surface 602 by a threshold distance / amount that exceeds other items on the surface 602, the XR system 100 may position the highlighted surface plane 604 at a certain distance / offset away from the surface 602 that allows the highlighted surface plane 604 to partially occlude the one or more objects. In other words, the distance from the surface 602 may not be sufficient to completely occlude those one or more objects that extend beyond the surface 602 by a threshold distance / amount that exceeds other items on the surface 602.

[0095] 6, the candles 620 and 622 and the water bottle 624 on the surface 602 are significantly higher than the remaining items on the surface 602. As a result, the candles 620 and 622 and the water bottle 624 extend a significant amount above the remaining items relative to the surface 602. Given the height of the candles 620 and 622 and the water bottle 624, if the highlighted surface plane 604 were to be raised / elevated above the candles 620 and 622 and the water bottle 624, the raised / elevated position of the highlighted surface plane 604 may cause the highlighted surface plane 604 to appear too high above the surface 602. In other words, at such a height, the highlighted surface plane 604 may not appear to a user to be on or near the surface 602, but rather may appear to be floating in the air.

[0096] To prevent this, the XR system 100 can implement rules that define thresholds and / or ranges that cause the highlighted surface plane 604 to move (e.g., raise / promote / offset) relative to the surface 602 and / or other items on the surface 602. If the furthest point of the object(s) (e.g., candles 620 and 622 and water bottle 624) relative to surface 602 (and / or the top or average height of the remaining items on surface 602) exceeds a defined threshold and / or range, then XR system 100 can position / render the highlighted surface plane 604 at a distance within the threshold and / or range from surface 602 and / or the remaining items on surface 602, even if part of the object(s) (e.g., candles 620 and 622 and water bottle 624) is occluded by the highlighted surface plane 604 and another part of the object(s) extends beyond the highlighted surface plane 604 (e.g., even if the highlighted surface plane 604 only partially occludes the object(s)).

[0097] 6, the farthest points of the candles 620 and 622 and the water bottle 624 relative to the surface 602 (and / or the top or average height of the other items on the surface 602) exceed a threshold height 640 above the surface 602. Thus, the highlighted surface plane 604 is positioned and rendered within the threshold height 640 above the surface 602. Thus, the highlighted surface plane 604 partially occludes the candles 620 and 622 and the water bottle 624, but does not rise / protrude up to or beyond the surface height 630 that includes the farthest points of the candles 620 and 622 and the water bottle 624 relative to the surface 602 and / or the other items on the surface 602. In this manner, the highlighted surface plane 604 is not moved too far away from the surface 602 and / or the other items on the surface 602.

[0098] Alternatively, the surface plane 604 may be moved by increasing the offset until at least one space on the moved / offset surface plane is available for rendering the virtual content item 610. To this end, the initial location of the surface plane may be determined, for example, at an average height above the surface 602. As previously described, the portion of the initial surface plane that is occluded by objects such as the candles 620 and 622 and the water bottle 624 may be determined, and the available space(s) (meeting a minimum size requirement) may be determined based on the unoccluded portion of the initial surface plane. Because the initial surface plane is offset / raised relative to the surface 602, the initial surface plane, when highlighted, partially occludes most, if not all, of the candles 620 and 622 and the water bottle 624, as well as other items on the surface 602. If it is determined that the initial surface plane does not have available space (meeting the minimum size requirement), the offset of the surface plane may be (iteratively) increased, for example, by a predetermined amount, and the process described above is repeated until at least one space (meeting the minimum size requirement) is available for rendering the virtual content item. In this manner, even complex surfaces may be processed to determine a suitable surface plane for rendering the virtual content item.

[0099] 7 is a flowchart illustrating an example process 700 for placing a virtual content item on a surface of a physical scene. At block 702, process 700 may include determining at least one surface plane in the scene of the physical environment. In some cases, determining at least one surface plane in the scene may include determining one or more surfaces of one or more objects in the scene and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0100] At block 704, the process 700 may include rendering the virtual content item at a first location in the scene and with a first perspective.

[0101] At block 706, the process 700 may include rendering a content placement indicator associated with the virtual content item on the first space of the at least one surface plane. In some examples, the content placement indicator may indicate at least a portion of the first space for placement of the virtual content item.

[0102] At block 708, process 700 may include determining one or more locations of one or more objects on a surface associated with the at least one surface plane.

[0103] At block 710, the process 700 may include rendering a content placement indicator on a second space of the at least one surface plane in response to the user input. In some examples, the second space of the at least one surface plane is determined based on one or more locations and the user input. In some examples, the content placement indicator may indicate at least a portion of the second space for placement of the virtual content item.

[0104] In some aspects, process 700 may include rendering the virtual content item in a second perspective at a second location within the scene based on the content placement indicator in response to an additional user input requesting placement of the virtual content item on the second space. In some examples, the second location may be within the second space. In some cases, process 700 may include determining the second perspective based on the perspective of the second space.

[0105] In some aspects, process 700 may include determining a first surface plane and a second surface plane of at least one surface plane, where a first space is associated with the first surface plane and a second space is associated with the second surface plane, and determining a second perspective based on a perspective of the second surface plane.

[0106] In some examples, rendering the content placement indicator on the second space may include determining an overlap between the virtual content item and the second surface plane, and switching to rendering the content placement indicator on the second space associated with the second surface plane upon determining that the overlap exceeds a first threshold.

[0107] In some examples, the first space and the second space are associated with a first surface plane of the at least one surface plane. In some aspects, the process 700 may include detecting one or more objects occluding at least a portion of the first surface plane, and segmenting the first surface plane into at least a first segment and a second segment based at least on the occluded portion of the first surface plane. In some examples, the first space is associated with the first segment and the second space is associated with the second segment.

[0108] In some cases, segmenting the first surface plane into at least a first segment and a second segment may include determining the fit of a virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

[0109] In some aspects, the process 700 may include changing an aspect ratio of the virtual content item between the first segment and the second segment.

[0110] In some aspects, the process 700 may include rendering the virtual content item in a third perspective at a third location within the scene based on the content placement indicator prior to rendering the content placement indicator on the second space. In some examples, the third location may be within the first space.

[0111] In some aspects, the process 700 may include moving the content placement indicator within the first space prior to rendering the content placement indicator on the second space in response to user input.

[0112] In some cases, determining at least one surface plane in the scene may include applying an offset to the at least one surface plane perpendicular to the at least one surface and toward a viewpoint of a user associated with the computing device.

[0113] In some cases, determining at least one surface plane in the scene may include determining one or more edges of one or more objects in the scene, and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

[0114] In some cases, rendering the content placement indicator may include rendering a pattern on at least one surface plane, in some examples, the pattern may include at least one of a highlight, an outline, a color, a shading, a shadow, a hatching, and a gradient.

[0115] In some examples, the processes described herein (e.g., process 700, and / or other processes described herein) may be performed by a computing device or apparatus. In one example, process 700 may be performed by the XR system 100 of FIG. 1. In another example, process 700 may be performed by a computing system having a computing device architecture 800 shown in FIG. 8. For example, a computing device with the computing device architecture 800 shown in FIG. 8 may implement the operations of FIG. 7 and / or the components and / or operations described herein with respect to any of FIGS. 1-7.

[0116] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or autonomous vehicle computing device, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device with resource capabilities to perform the processes described herein, including process 700 and / or any other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0117] Components of a computing device may be implemented with circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuitry (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein.

[0118] Process 700 is illustrated as a logical flow diagram, whose operations represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0119] Additionally, process 700 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executing on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0120] 8 is a diagram illustrating an example of a system implementing some aspects of the present technology. In particular, FIG. 8 illustrates an example of a computing system 800, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device comprising any of the components thereof, in which the components of the system communicate with each other using a connection 805. The connection 805 may be a physical connection using a bus, or a direct connection to a processor 810, such as in a chipset architecture. The connection 805 may also be a virtual connection, a network connection, or a logical connection.

[0121] In some embodiments, computing system 800 is a distributed system in which the functionality described in this disclosure may be distributed across one data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each performing some or all of the functionality that is the subject of the component description. In some embodiments, the components may be physical or virtual devices.

[0122] The exemplary system 800 includes at least one processing unit (CPU or processor) 810 and connections 805 coupling various system components to the processor 810, including system memory 815, such as read-only memory (ROM) 820 and random access memory (RAM) 825. The computing system 800 may include a cache of high-speed memory 812, either directly connected to the processor 810, in close proximity to the processor 810, or integrated as part of the processor 810.

[0123] Processor 810 may include any general purpose processor, as well as hardware or software services, such as services 832, 834, and 836, stored in storage device 830 and configured to control processor 810, as well as special purpose processors whose software instructions are embedded in the actual processor design. Processor 810 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0124] To enable user interaction, computing system 800 includes input devices 845, which can represent any number of input mechanisms, such as a microphone for speaking, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 800 may also include output devices 835, which can be one or more of several output mechanisms. In some cases, a multi-modal system may enable a user to provide multiple types of input / output to communicate with computing system 800. Computing system 800 may generally include a communications interface 840, which can govern and manage user input and system output.The communications interface may be any of the following: audio jack / plug, microphone jack / plug, universal serial bus (USB) port / plug, Apple® Lightning® port / plug, Ethernet port / plug, fiber optic port / plug, proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WLAN), and Bluetooth® wireless signal transmission. The wireless communication device may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing WiMAX (Wireless Access), infrared (IR) communications wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.The communications interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 800 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite system (BDS), and the European Galileo GNSS. There is no constraint to operate with any particular hardware arrangement, and therefore the basic features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0125] The storage device 830 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (ICC), a circuit, IC chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM),The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a memory, such as RRAM / ReRAM, phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0126] Storage device 830 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by processor 810. In some embodiments, hardware services that perform a particular function may include software components stored in a computer readable medium with respect to necessary hardware components such as processor 810, connections 805, output devices 835, etc., to perform the function. The term "computer readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instruction(s) and / or data. Computer readable media may also include non-transitory media on which data may be stored and that do not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memories, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored on it, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0127] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when mentioned, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0128] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks comprising devices, device components, and steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0129] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagrams. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0130] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions, or in some cases configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network-attached storage devices, etc.

[0131] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor or processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing in a single device, as further examples.

[0132] The instructions, media for propagating such instructions, computing resources for executing such instructions, and other structures supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0133] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

[0134] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0135] When a component is described as being "configured to" perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0136] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0137] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) fulfills the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items in the set of A and B.

[0138] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0139] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be realized at least in part by a computer-readable data storage medium that includes program code that includes instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0140] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0141] Illustrative examples of the present disclosure include the following: Aspect 1. An apparatus for rendering virtual content within a scene of a physical environment, the apparatus comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to: determine at least one surface plane within the scene of the physical environment; render a virtual content item at a first location within the scene with a first perspective; render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; determine one or more locations of one or more objects on a surface associated with the at least one surface plane; and render the content placement indicator on a second space of the at least one surface plane in response to user input, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, and the content placement indicator indicates at least a portion of the second space for placement of the virtual content item.

[0142] Aspect 2. The device of aspect 1, wherein the one or more processors are further configured to render the virtual content item with a second perspective at a second location within the scene based on the content placement indicator in response to additional user input requesting placement of the virtual content item in the second space, the second location being within the second space.

[0143] Example 3. The apparatus of Example 2, wherein the one or more processors are further configured to determine a second perspective based on the perspective of the second space.

[0144] Aspect 4. The apparatus of aspect 2, wherein the one or more processors are further configured to determine a first surface plane and a second surface plane of at least one surface plane, where a first space is associated with the first surface plane and a second space is associated with the second surface plane, and to determine a second perspective based on the perspective of the second surface plane.

[0145] Aspect 5. The device of aspect 4, wherein to render the content placement indicator on the second space, the one or more processors are configured to determine an overlap between the virtual content item and the second surface plane, and upon determining that the overlap exceeds a first threshold, switch to rendering the content placement indicator on the second space associated with the second surface plane.

[0146] Aspect 6. The apparatus of any of Aspects 1-5, wherein the first space and the second space are associated with a first surface plane of at least one surface plane, and the one or more processors are further configured to detect one or more objects occluding at least a portion of the first surface plane and segment the first surface plane into at least a first segment and a second segment based at least on the occluded portion of the first surface plane, wherein the first space is associated with the first segment and the second space is associated with the second segment.

[0147] Aspect 7. The apparatus of aspect 6, wherein to segment the first surface plane into at least a first segment and a second segment, the one or more processors are configured to determine the fit of a virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

[0148]

[0023] Example 8. The apparatus of any of Examples 6-7, wherein the one or more processors are further configured to modify an aspect ratio of the virtual content item between the first segment and the second segment.

[0149] Aspect 9. The device of any of aspects 1 to 8, wherein the one or more processors are configured to render a virtual content item with a third perspective at a third location within the scene based on the content placement indicator before rendering the content placement indicator on the second space, the third location being within the first space.

[0150] Aspect 10. The device of any of aspects 1-9, wherein the one or more processors are further configured to move the content placement indicator within the first space prior to rendering the content placement indicator on the second space in response to user input.

[0151] Aspect 11. The apparatus of any of aspects 1 to 10, wherein, to determine at least one surface plane in the scene, the one or more processors are configured to determine one or more surfaces of one or more objects in the scene, and determine the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0152] Aspect 12. The device of aspect 11, wherein to determine at least one surface plane within the scene, the one or more processors are further configured to apply an offset to the at least one surface plane perpendicular to the at least one surface and toward a viewpoint of a user associated with the device.

[0153] Aspect 13. The apparatus of any of aspects 1 to 12, wherein to determine at least one surface plane in the scene, the one or more processors are configured to determine one or more edges of one or more objects in the scene, and to determine the at least one surface plane based on the one or more edges of the one or more objects in the scene.

[0154] Example 14. The apparatus of any of Examples 1-13, wherein the one or more processors are configured to render a pattern on at least one surface plane to render the content placement indicator.

[0155] Aspect 15. The apparatus of aspect 14, wherein the pattern includes at least one of highlighting, contouring, color, shading, shadow, hatching, and gradients.

[0156] Example 16. The apparatus of any of Examples 1-15, wherein the apparatus comprises a mobile device.

[0157] Example 17. The apparatus of example 16, wherein the mobile device includes an extended reality device.

[0158] Example 18. The apparatus of any of Examples 1-17, wherein the first space and the second space include a space on at least one surface plane that is at least one of: not occupied by any physical object; and not having any physical object visually blocking visibility of a respective portion of the at least one surface plane associated with the first space and the second space.

[0159] Example 19. The apparatus of any of Examples 1-18, wherein the second space includes available space on at least one surface plane that is occupied by one or more physical objects in the physical scene.

[0160] Aspect 20. The device of any of aspects 1-19, wherein one or more processors are configured to determine, based on an aspect ratio of the virtual content item, that at least a threshold amount of the virtual content item does not fit within the second space, determine that at least a threshold amount of the virtual content item fits within the second space when the virtual content item is configured according to a different aspect ratio, configure a content placement indicator according to the different aspect ratio, and render the content placement indicator on the second space according to the different aspect ratio.

[0161] Aspect 21. The device of aspect 20, wherein one or more processors are configured to adjust the aspect ratio of the virtual content item to a different aspect ratio and render the virtual content item in a different aspect ratio, rendered onto the second space and within the content placement indicator, in response to an additional input requesting placement of the virtual content item onto the second space.

[0162] Aspect 22. The device of any of aspects 1 to 21, wherein, to determine at least one surface plane in the scene, the one or more processors are configured to determine one or more surfaces of one or more objects in the scene, and determine the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0163] Aspect 23. The apparatus of aspect 22, wherein the one or more processors are further configured to apply smoothing to one or more surfaces of one or more objects in the scene before determining at least one surface plane based on one or more surfaces of the one or more objects in the scene.

[0164] Aspect 24. The apparatus of aspect 22, wherein to determine at least one surface plane in the scene, one or more processors are configured to align the at least one surface plane with at least one surface of one or more surfaces of one or more objects in the scene.

[0165] Example 25. The apparatus of any of Examples 1-24, wherein the one or more processors are configured to render at least one surface plane for at least one surface.

[0166] Aspect 26. The device of aspect 25, wherein to render the at least one surface plane, the one or more processors are configured to determine an amount of flatness of the surface that is less than a threshold and render the at least one surface plane at a distance from one or more points on the surface associated with the at least one surface plane.

[0167] Aspect 27. The apparatus of aspect 25, wherein, to render the at least one surface plane, the one or more processors are configured to determine that a highest point of the surface is a first threshold distance above one or more other points of the surface, and to render the at least one surface plane a second threshold distance above the highest point of the surface.

[0168] Aspect 28. The device of any of aspects 1 to 27, wherein to render the content placement indicator, the one or more processors are configured to render a visual contour on at least one surface plane.

[0169] Aspect 29. The apparatus of aspect 28, wherein the visual contour includes at least one of a frame or a bounding box.

[0170]

[0046] Example 30. The device of any of Examples 28-29, wherein the one or more processors are further configured to render visual contours in perspective of the respective surface planes.

[0171] Aspect 31. The device of any of aspects 1 to 30, wherein to render the first content placement indicator and / or the second content placement indicator, the one or more processors are configured to render a preview of the virtual content item on at least one surface plane.

[0172] Aspect 32. A method for rendering virtual content within a scene of a physical environment, comprising: determining at least one surface plane within the scene of the physical environment; rendering a virtual content item at a first location within the scene with a first perspective via a computing device; rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; and rendering the content placement indicator on a second space of the at least one surface plane in response to user input, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, and the content placement indicator indicates at least a portion of the second space for placement of the virtual content item.

[0173] Aspect 33. The method of aspect 32, further comprising, in response to additional user input requesting placement of the virtual content item on the second space, rendering the virtual content item with a second perspective at a second location within the scene based on the content placement indicator, the second location being within the second space.

[0174] Example 34. The method of example 33, further comprising determining a second perspective based on the perspective of the second space.

[0175] Example 35. The method of example 33, further comprising: determining a first surface plane and a second surface plane of at least one surface plane, where a first space is associated with the first surface plane and a second space is associated with the second surface plane; and determining a second perspective based on the perspective of the second surface plane.

[0176] Aspect 36. The method of aspect 35, wherein rendering the content placement indicator on the second space includes determining an overlap between the virtual content item and the second surface plane, and switching to rendering the content placement indicator on the second space associated with the second surface plane upon determining that the overlap exceeds a first threshold.

[0177] Aspect 37. The method of any of aspects 32-36, wherein the first space and the second space are associated with a first surface plane of at least one surface plane, and the method further includes detecting one or more objects occluding at least a portion of the first surface plane, and segmenting the first surface plane into at least a first segment and a second segment based at least on the occluded portion of the first surface plane, wherein the first space is associated with the first segment and the second space is associated with the second segment.

[0178] Aspect 38. The method of aspect 37, wherein segmenting the first surface plane into at least a first segment and a second segment includes determining fit of a virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

[0179] Example 39. The method of any of examples 37-38, further comprising changing an aspect ratio of the virtual content item between the first segment and the second segment.

[0180] Aspect 40. The method of any of aspects 32 to 39, further comprising rendering a virtual content item at a third location within the scene with a third perspective based on the content placement indicator before rendering the content placement indicator on the second space, the third location being within the first space.

[0181] Aspect 41. The method of any of aspects 32-40, further comprising, in response to user input, moving the content placement indicator within the first space prior to rendering the content placement indicator on the second space.

[0182] Aspect 42. The method of any of aspects 32 to 41, wherein determining at least one surface plane in the scene includes determining one or more surfaces of one or more objects in the scene, and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0183] Aspect 43. The method of aspect 42, wherein determining at least one surface plane in the scene includes applying an offset to the at least one surface plane perpendicular to the at least one surface and toward a viewpoint of a user associated with the computing device.

[0184] Aspect 44. The method of any of aspects 32 to 43, wherein determining at least one surface plane in the scene includes determining one or more edges of one or more objects in the scene, and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

[0185] Embodiment 45. The method of any of embodiments 32-44, wherein rendering the content placement indicator includes rendering a pattern on at least one surface plane.

[0186] Aspect 46. The method of aspect 45, wherein the pattern includes at least one of highlighting, contouring, color, shading, shadow, hatching, and gradients.

[0187] Embodiment 47. The method of any of embodiments 32-46, wherein the first space and the second space include a space on at least one surface plane that is at least one of: not occupied by any physical object; and not having any physical object visually blocking visibility of a respective portion of the at least one surface plane associated with the first space and the second space.

[0188] Example 48. The method of any of examples 32 to 47, wherein the second space includes available space on at least one surface plane that is occupied by one or more physical objects in the physical scene.

[0189] Aspect 49. The method of any of aspects 32-48, further including: determining, based on an aspect ratio of the virtual content items, that at least a threshold amount of the virtual content items will not fit within the second space; determining that at least the threshold amount of the virtual content items will fit within the second space when the virtual content items are configured according to a different aspect ratio; configuring a content placement indicator according to the different aspect ratio; and rendering the content placement indicator on the second space according to the different aspect ratio.

[0190] Aspect 50. The method of aspect 49, further comprising adjusting an aspect ratio of the virtual content item to a different aspect ratio, and rendering the virtual content item in the different aspect ratio, rendered onto the second space and within the content placement indicator, in response to additional input requesting placement of the virtual content item onto the second space.

[0191] Aspect 51. The method of any of aspects 32 to 50, wherein determining at least one surface plane in the scene includes determining one or more surfaces of one or more objects in the scene, and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

[0192] Aspect 52. The method of aspect 51, further comprising applying smoothing to one or more surfaces of one or more objects in the scene before determining at least one surface plane based on one or more surfaces of one or more objects in the scene.

[0193] Aspect 53. The method of aspect 51, wherein determining at least one surface plane in the scene includes aligning the at least one surface plane with at least one surface of one or more surfaces of one or more objects in the scene.

[0194] Embodiment 54. The method of any of embodiments 32 to 53, further comprising rendering at least one surface plane for the surface.

[0195] Aspect 55. The method of aspect 54, wherein rendering the at least one surface plane includes determining that an amount of flatness of the surface is less than a threshold, and rendering the at least one surface plane at a distance from one or more points on the surface associated with the at least one surface plane.

[0196] Aspect 56. The method of aspect 54, wherein rendering at least one surface plane includes determining that a highest point of the surface is a first threshold distance above one or more other points of the surface, and rendering the at least one surface plane a second threshold distance above the highest point of the surface.

[0197] Embodiment 57. The method of any of embodiments 32-56, wherein rendering the content placement indicator includes rendering a visual contour on at least one surface plane.

[0198] Aspect 58. The method of aspect 57, wherein the visual contour includes at least one of a frame or a bounding box.

[0199] Embodiment 59. The method of any of embodiments 57 to 58, further comprising rendering visual contours in perspective of the respective surface planes.

[0200] Aspect 60. The method of any of aspects 32-59, wherein rendering the content placement indicator includes rendering a preview of the virtual content item on at least one surface plane.

[0201] Embodiment 61. An apparatus comprising means for carrying out a method according to any of embodiments 32-59.

[0202] Aspect 62. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 32-59.

[0203] Aspect 63. A method for rendering virtual content within a scene of a physical environment, the method comprising: determining at least one surface plane within the scene of the physical environment; determining that an amount of surface flatness associated with the at least one surface is below a threshold; rendering the at least one surface plane at a distance from one or more points on a surface associated with the at least one surface plane; rendering a content placement indicator associated with a virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; and rendering, via a computing device, the virtual content item in a first perspective at a first location in the scene associated with the first space of the at least one surface.

[0204] Example 64. An apparatus comprising means for carrying out the method according to example 63.

[0205] Aspect 65. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method according to aspect 63.

Claims

1. 1. An apparatus for rendering virtual content within a scene of a physical environment, comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: determining at least one surface plane in the scene of the physical environment; Rendering a virtual content item at a first location within the scene and with a first perspective; rendering a content placement indicator associated with the virtual content item on the first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; Rendering the content placement indicator on a second space of the at least one surface plane in response to user input. a second space of the at least one surface plane is determined based on the one or more locations and the user input, and the content placement indicator indicates at least a portion of the second space for placement of the virtual content item.

2. the one or more processors:

2. The device of claim 1, further configured to, in response to further user input requesting placement of the virtual content item on the second space, render the virtual content item in a second perspective at a second location within the scene based on the content placement indicator, the second location being within the second space.

3. the one or more processors: The apparatus of claim 2 , further configured to determine the second perspective based on a perspective of the second space.

4. the one or more processors: determining a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; determining the second perspective based on the perspective of the second surface plane; The apparatus of claim 2 , further configured to:

5. To render the content placement indicator over the second space, the one or more processors: determining an overlap between the virtual content item and the second surface plane; upon determining that the overlap exceeds a first threshold, switching to rendering the content placement indicator on the second space associated with the second surface plane. The apparatus of claim 4 , configured as follows:

6. the first space and the second space are associated with a first surface plane of the at least one surface plane; the one or more processors: Detecting one or more objects occluding at least a portion of the first surface plane; segmenting the first surface plane into at least a first segment and a second segment based at least on the occluded portion of the first surface plane. It is further structured as follows: The apparatus of claim 1 , wherein the first space is associated with the first segment and the second space is associated with the second segment.

7. 7. The apparatus of claim 6, wherein to segment the first surface plane into at least the first segment and the second segment, the one or more processors are configured to determine a fit of the virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

8. 8. The apparatus of claim 7, wherein the one or more processors are further configured to vary an aspect ratio of the virtual content item between the first segment and the second segment.

9. the one or more processors:

2. The device of claim 1, configured to render the virtual content item at a third location within the scene with a third perspective based on the content placement indicator prior to rendering the content placement indicator onto the second space, the third location being within the first space.

10. the one or more processors: The apparatus of claim 1 , further configured to move the content placement indicator within the first space prior to rendering the content placement indicator over the second space in response to the user input.

11. To determine the at least one surface plane in the scene, the one or more processors: determining one or more surfaces of the one or more objects in the scene; determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene; The apparatus of claim 1 , configured as follows:

12. To determine the at least one surface plane in the scene, the one or more processors:

12. The device of claim 11, further configured to apply an offset to the at least one surface plane perpendicular to at least one surface and toward a viewpoint of a user associated with the device.

13. To determine the at least one surface plane in the scene, the one or more processors: determining one or more edges of the one or more objects in the scene; determining the at least one surface plane based on the one or more edges of the one or more objects in the scene; The apparatus of claim 1 , configured as follows:

14. The apparatus of claim 1 , wherein the one or more processors are configured to render a pattern on the at least one surface plane to render the content placement indicator.

15. The apparatus of claim 14 , wherein the pattern comprises at least one of a highlight, an outline, a color, a shade, a shadow, a hatch, and a gradient.

16. The apparatus of claim 1 , wherein the apparatus comprises a mobile device.

17. The apparatus of claim 16 , wherein the mobile device comprises an extended reality device.

18. 1. A method for rendering virtual content within a scene of a physical environment, comprising: determining at least one surface plane in the scene of the physical environment; rendering, via a computing device, a virtual content item at a first location within the scene and with a first perspective; rendering a content placement indicator associated with the virtual content item on the first space of the at least one surface plane, the content placement indicator indicating at least a portion of a first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; Rendering the content placement indicator on a second space of the at least one surface plane in response to user input; wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, and the content placement indicator indicates at least a portion of the second space for placement of the virtual content item.

19. 20. The method of claim 18, further comprising: in response to further user input requesting placement of the virtual content item on the second space, rendering the virtual content item in a second perspective at a second location within the scene based on the content placement indicator, the second location being within the second space.

20. 20. The method of claim 19, further comprising determining the second perspective based on a perspective of the second space.

21. determining a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; determining the second perspective based on the perspective of the second surface plane; 20. The method of claim 19, further comprising:

22. Rendering the content placement indicator over the second space includes: determining an overlap between the virtual content item and the second surface plane; upon determining that the overlap exceeds a first threshold, switching to rendering the content placement indicator on the second space associated with the second surface plane; 22. The method of claim 21 , comprising:

23. The first space and the second space are associated with a first surface plane of the at least one surface plane, and the method further comprises: Detecting one or more objects occluding at least a portion of the first surface plane; segmenting the first surface plane into at least a first segment and a second segment based at least on the occluded portions of the first surface plane; Further comprising: The method of claim 18 , wherein the first space is associated with the first segment and the second space is associated with the second segment.

24. 24. The method of claim 23, wherein segmenting the first surface plane into at least the first segment and the second segment comprises determining the fit of the virtual content item rendered in perspective of the first surface plane within the first segment and the second segment.

25. 25. The method of claim 24, further comprising changing an aspect ratio of the virtual content item between the first segment and the second segment.

26. 20. The method of claim 18, further comprising rendering the virtual content item at a third location within the scene and with a third perspective based on the content placement indicator prior to rendering the content placement indicator onto the second space, the third location being within the first space.

27. 20. The method of claim 18, further comprising: moving the content placement indicator within the first space prior to rendering the content placement indicator over the second space in response to the user input.

28. Determining the at least one surface plane in the scene comprises: determining one or more surfaces of the one or more objects in the scene; determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene; 20. The method of claim 18, comprising:

29. Determining the at least one surface plane in the scene comprises:

30. The method of claim 28, comprising applying an offset to the at least one surface plane perpendicular to at least one surface and toward a viewpoint of a user associated with the computing device.

30. Determining the at least one surface plane in the scene comprises: determining one or more edges of the one or more objects in the scene; determining the at least one surface plane based on the one or more edges of the one or more objects in the scene; 20. The method of claim 18, comprising:

31. The method of claim 18 , wherein rendering the content placement indicator comprises rendering a pattern on the at least one surface plane.

32. 32. The method of claim 31, wherein the pattern comprises at least one of a highlight, an outline, a color, a shade, a shadow, a hatch, and a gradient.

Images