Adaptation of space and content for augmented reality and composite reality

Abstract

To allow a user to experience immersiveness in a physical environment without relying on operation of the user for setting of AR / MR experience.SOLUTION: At the time of an AR simulation or MR simulation, a designation of at least one virtual object which can be arranged in a real world environment is obtained. On the basis of scanning of the real world environment, a first representation expressing the real world environment is generated. From the first representation, a second representation expressing at least one real world environment is generated. Using a machine learning model, the at least one virtual object and the at least one second representation are evaluated. Based at least in part on the evaluation, adaptation of the at least one virtual object and at least one usable space present in the real world environment is determined. On the basis of this adaptation, the at least one virtual object is rendered on a computing device.SELECTED DRAWING: Figure 4

Description

Cross-reference to related art

[0001] This application is a divisional application of Patent Application No. 2023-79217, filed on May 12, 2023. [Technical Field]

[0002] This disclosure relates to space and content adaptation for augmented and mixed reality. [Background technology]

[0003] Augmented reality (AR) and mixed reality (MR) technologies are being increasingly used in a wide variety of fields, including healthcare, social networking and engagement, communication, shopping, the entertainment industry, travel, navigation, and education. AR involves overlaying computer-generated imagery onto a user's real-world environment. AR systems can use a video device to display sequential images (e.g., a video feed) of a real-world environment to a user, with various virtual objects inserted in appropriate locations within the environment. For example, an AR system can identify a real-world object called a "table" and display a virtual "cup" on the video device so that the virtual "cup" appears to be sitting on the table (e.g., from the video device's perspective). Mixed reality (MR), on the other hand, is generally considered an extension of AR, allowing users to interact with real and virtual objects in the environment. For example, in a mixed reality experience, users can adapt and manipulate visual content (e.g., graphics for games or interactive stories) to overlay on the physical environment they are viewing.

[0004] However, currently, many AR / MR experiences rely on the user to position and size virtual objects on their display device in order to make them "fit" into the user's real-world environment. Summary of the Invention [Problem to be solved by the invention]

[0005] However, relying on user interaction to set up the AR / MR experience in this way could potentially hinder users from having an immersive experience in the physical environment. [Means for solving the problem]

[0006] One embodiment described herein is a computer-implemented method. The computer-implemented method includes obtaining an indication of at least one virtual object placeable in a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation. The computer-implemented method further includes generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of the computing device. The computer-implemented method further includes generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space in the real-world environment. The computer-implemented method further includes evaluating, using one or more machine learning models, the at least one virtual object and the at least one second representation of the real-world environment, and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation. The computer-implemented method further includes rendering at least one virtual object at a location on a display of the computing device, the location associated with at least one available space in the real-world environment, based on the adaptation.

[0007] Another embodiment described herein is a computing device. The computing device includes a display, one or more sensors, one or more processors, and a memory that stores instructions, where execution of the instructions on the one or more processors performs a process. The process includes obtaining an indication of at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation. The process further includes generating a first representation of the real-world environment based on a scan of the real-world environment by the one or more sensors. The process further includes generating at least one second representation of the real-world environment from the first representation of the real-world environment, the second representation showing at least one available space in the real-world environment. The process further includes evaluating the at least one virtual object and the at least one second representation of the real-world environment using one or more machine learning models, and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation. The process further includes rendering at least one virtual object at a location on the display that is associated with at least one available space in the real-world environment based on the match.

[0008] Other embodiments described herein include a non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs a process. The process includes obtaining an indication of at least one virtual object placeable in a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation. The process further includes generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of the computing device. The process further includes generating at least one second representation of the real-world environment from the first representation of the real-world environment, the second representation showing at least one available space in the real-world environment. The process further includes evaluating the at least one virtual object and the at least one second representation of the real-world environment using one or more machine learning models, and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation. The process further includes rendering at least one virtual object at a location on a display of the computing device, the location associated with at least one available space in the real-world environment, based on the matching.

[0009] So that the above aspects can be realized and understood in detail, the embodiments briefly summarized hereinabove will now be described in more detail with reference to the accompanying drawings.

[0010] It should be noted, however, that the attached drawings illustrate typical embodiments and are therefore not to be considered limiting, as other embodiments of similar effect are contemplated. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 is a block diagram illustrating an exemplary augmented reality device, according to one embodiment. [Figure 2] A block diagram illustrating an exemplary workflow for automatically placing and manipulating virtual object(s) as part of an AR / MR experience, according to one embodiment. [Figure 3] FIG. 10 is a block diagram illustrating another exemplary workflow for automatically placing and manipulating virtual object(s) as part of an AR / MR experience, according to one embodiment. [Figure 4] 1 is a flowchart illustrating a method for automatically placing and manipulating virtual object(s), according to one embodiment. [Figure 5] 1 is a flowchart illustrating another method for automatically placing and manipulating virtual object(s), according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Many AR / MR experiences allow users to view and virtually interact with virtual objects within a real-world environment. These virtual objects can be (or include) digital representations of people, fictional characters, places, items, and identifiers such as brands and logos that make up a virtual reality (VR), AR, or MR environment. Furthermore, virtual objects can describe a virtual world that can be experienced synchronously and persistently by any number of users, and can also provide continuity of data such as personal identity, user history, entitlements, ownership, and payments. For example, but not by way of limitation, AR / MR experiences may allow users to "preview" various virtual furniture items in a particular real-world environment (e.g., a physical space such as a living room, office, or bedroom), view and interact with virtual characters within the user's real-world environment as part of an interactive narrative experience, view and interact with rendered representations while playing an interactive game, or view and interact with other virtual users (e.g., avatars) in the virtual world.

[0013] However, a problem with traditional AR / MR experiences is that they largely rely on the user to position and size virtual objects on their computing device so that the virtual objects blend into the user's real-world environment. For example, consider a case where a user begins a new interactive narrative experience on a computing device. As part of the interactive narrative experience, the computing device renders virtual objects (e.g., representations of characters or items in the interactive narrative, representations of virtual scenery such as natural features or landscapes in the interactive narrative, etc.) on the computing device, allowing the user to visualize the virtual objects (via the computing device) within the user's real-world environment.

[0014] However, a computing device may render a virtual object without the context of one or more attributes of the real-world environment (e.g., the size of the real-world environment, the size / position / orientation of real items in the real-world environment, etc.). For example, consider a real-world environment that includes a "real" physical table and a virtual object that is a virtual character. In this case, a computing device may render the virtual character on the computing device in a way that makes it appear as if the virtual character is standing on the table, rather than standing on the floor beside the table. Additionally or alternatively, a computing device may render the virtual character at an incorrect size on the screen, causing the virtual character to appear to be an incorrect size relative to the table (e.g., too small or too large relative to the table from the perspective of the computing device).

[0015] Conventional AR / MR experiences typically rely on an initial calibration performed by the user at the start of the AR / MR experience to position and size virtual objects. During the initial calibration, the user is prompted to place a virtual character on a table and adjust the size appropriately (e.g., by adjusting the size of the virtual character on the screen of the computing device) so that subsequent virtual objects (e.g., of the same or different type as the first virtual object) can be placed on the same table at the same size. After the initial calibration, subsequent virtual objects can be automatically positioned under the assumption that the environmental information has not changed since the initial calibration. However, relying on the user to manually perform this initial calibration can confuse the user, disrupt the illusion of immersion, or lead to a negative user experience, such as the user accidentally bumping into real objects while trying to manipulate virtual objects.

[0016] To address this, embodiments herein describe techniques for automatically adjusting attributes (e.g., size, placement, etc.) of virtual object(s) rendered on a computing device so that the virtual object(s) fit into a real-world environment. That is, embodiments automatically determine how well a user's real-world environment fits or does not fit into a proposed virtual object(s), thereby enabling automatic adjustment and customization of the virtual object(s) for a given real-world environment without manual user intervention.

[0017] In one embodiment described below, a computing device includes an augmentation component configured to generate a three-dimensional (3D) mesh representing available space (e.g., empty space) within a user's real-world environment. The real-world environment may be an internal physical environment (e.g., an indoor space such as a room) or an external physical environment (e.g., an outdoor space such as a backyard or park). The augmentation component may also retrieve (e.g., from an AR / MR application running on the computing device) one or more virtual objects being planned for the real-world environment. For example, the virtual object(s) may be (or include) digital representations of people, fictional characters, places, items, characters, items, etc. associated with the AR / MR experience, and rendering the virtual object(s) on the computing device allows the user to visualize the virtual object(s) within the real-world environment. The virtual object(s) may include static virtual object(s) and / or dynamic virtual object(s).

[0018] The augmentation component can use one or more machine learning techniques to evaluate (i) a 3D mesh representing available space in the real-world environment and (ii) one or more virtual objects to identify a set of correspondences between the virtual object(s) and available space in the real-world environment. For example, the augmentation component can do so to match a "talking teapot" virtual object in an interactive storytelling AR / MR experience to the flat surface of a real-world table as seen on a user's computing device. The augmentation component can then automatically render the "talking teapot" virtual object on the computing device so that, from the perspective of the computing device, the virtual object appears to lie on the flat surface of the table.

[0019] Additionally, in some embodiments, the augmentation component can identify multiple correspondences between available space in the real-world environment and one or more dynamic (e.g., changing) virtual objects. For example, the augmentation component can determine a set of user paths in the real-world environment (e.g., a room) and, for each user path, identify a 3D mesh that represents available space in the real-world environment.

[0020] For example, a first path may represent a user walking in a first direction (e.g., right) around a real object (e.g., a table) in a real-world environment, and a second path may represent a user walking in a second direction (e.g., left) around the real object in a real-world environment. In such an example, the augmentation component may generate a first 3D mesh representing the usable space in the real-world environment associated with the first path, and a second 3D mesh representing the usable space in the real-world environment associated with the second path. The augmentation component may then use machine learning techniques to identify, for each user path, a correspondence between the usable space in the real-world environment represented by the respective 3D mesh and the virtual object(s).

[0021] As the user moves through the real-world environment, the augmentation component can determine the user's position within the resulting path (e.g., based on one or more sensors of the computing device) and automatically render virtual object(s) in the available space based on the user's position, the user's attributes (e.g., the user's height or accessibility limitations), and the correspondence information. As an example, consider a "superhero character" virtual object in an interactive gaming experience; when the user is on a first path, the virtual object can be rendered to appear on a first side of the user, and when the user is on a second path, the virtual object can be rendered to appear on the other, second side of the user.

[0022] Thus, embodiments may enable automated placement and manipulation of one or more virtual objects on a user's computing device as part of an AR / MR experience, without relying on the user to manually manipulate the virtual objects. As a result, embodiments may enable a computing device to generate a dynamic, personalized AR / MR experience that automatically adapts to multiple users and real-world environments.

[0023] It should be noted that many of the following embodiments use interactive narrative experiences as a reference example of an AR / MR experience that can utilize the automated virtual object placement and manipulation techniques described herein. However, it should be noted that the techniques described herein can be used in a wide variety of AR / MR experiences, such as shopping, travel, social interaction (e.g., "multiverse"), navigation, gaming, and the like.

[0024] 1 is a block diagram illustrating a reality augmentation device 110 configured with an augmentation component 134, according to one embodiment. The reality augmentation device 110 generally refers to various computing devices, such as, for example, a smartphone, a tablet, a laptop, a headset, a visor, a head-mounted display (HMD), or glasses. The reality augmentation device 110 can be an AR-enabled computing device, an MR-enabled computing device, or an AR / MR-enabled computing device. The reality augmentation device 110 can implement one or more techniques described herein for automated placement and manipulation of virtual objects as part of an AR / MR experience.

[0025] The reality augmented device 110 includes a processor 120, memory 130, storage 140, one or more cameras 150, one or more display devices 160, one or more sensors 170, and a network interface 180. The processor 120 refers to any number of processing elements and may include any number of processing cores. The memory 130 may include volatile memory, non-volatile memory, and combinations thereof. The memory 130 generally includes program code for performing various functions associated with applications (e.g., application 132) hosted on the reality augmented device 110.

[0026] The application 132 refers to a client-server application (or other distributed application) component. The application 132 may be a "thin" client, where the majority of the instructions for execution are provided by the application 132, but the execution of the operations may occur on a back-end server (not shown) computing system or in a traditional software application installed on the augmented reality device 110. The application 132 includes an extension component 134, which will be described in more detail below.

[0027] The storage device 140 may be a disk drive storage device. Note that while the storage device 140 is shown as a single unit, the storage device 140 may be a combination of fixed and / or removable storage devices. Such storage devices may include fixed disk drives, removable memory cards, or optical storage devices, network attached storage (NAS), or storage area networks (SAN). The network interface 180 may be any type of network communication interface that allows the augmented reality device 110 to communicate with other computers and / or components in a computing environment (e.g., external unit 190) over a data communications network.

[0028] The display device(s) 160 and the camera(s) 150 allow a user to view the real-world environment in which the reality augmentation device 110 is located from the perspective of the reality augmentation device 110. For example, the camera(s) 150 can capture a visual scene. As used herein, a visual scene refers to a view(s) of the real-world environment in which the reality augmentation device 110 is being used. For example, the visual scene can be a continuous image (e.g., a video feed) of the real-world environment. The visual scene can be displayed on the display device(s) 160. For example, activating the camera(s) 150 can project the visual scene (on the display device(s) 160) and allow virtual objects to be overlaid on top of the visual scene. The display device(s) 160 may include any suitable display technology that provides a physical conversion of signals to light, such as liquid crystal display(s) (LCD), light emitting diode(s) (LED), organic light emitting diode display(s) (OLED), or quantum dot (QD) display(s). The camera(s) 150 may be provided in conjunction with image recognition software (e.g., stored in memory 130) for identifying real objects within the field of view of the camera(s) 150.

[0029] The sensor(s) 170 are configured to sense information from the real-world environment. In one embodiment, the sensor(s) 170 include an accelerometer, a gyroscope, or a combination thereof. The accelerometer can measure acceleration forces acting on the reality augmented device 110, providing information about whether and in what direction the reality augmented device 110 is moving. The accelerometer can also be used to determine the tilt of the reality augmented device 110. The gyroscope can measure the orientation of the reality augmented device 110, providing information about whether the reality augmented device 110 is level or how many degrees the reality augmented device 110 is tilted in one or more planes. In one embodiment, the combination of the accelerometer and the gyroscope can also provide information about the sense of direction of the reality augmented device 110, in terms of pitch and roll relative to gravity.

[0030] In general, the reality augmented device 110 may include any number of sensors and / or utilize any technology or combination of technologies suitable for the functions described herein to determine the orientation (e.g., tilt) of the reality augmented device 110. Similarly, the sensor(s) 170 may include various types of sensors, including but not limited to accelerometers and gyroscopes. Other types of sensors 170 include, but are not limited to, Light Detection and Ranging (LiDAR) sensors, Global Positioning System (GPS) receivers, or inertial motion units (IMUs), any type of sensor that provides information about the orientation or position of the reality augmented device 110 in a real-world environment.

[0031] The augmentation component 134 generally allows a user to visualize and virtually interact with one or more virtual objects within a real-world environment. In the embodiments described herein, the augmentation component 134 is capable of automatically placing and manipulating virtual objects that are overlaid on visual scenes of the real-world environment. The augmentation component 134 includes a visualization tool 136 and a path planning tool 138, either of which may include software components, hardware components, or a combination thereof. The visualization tool 136 and the path planning tool 138 are described in more detail below.

[0032] In some embodiments, the reality augmented device 110 can also use and / or interact with the external unit 190 to provide an AR / XR experience to the user. In such embodiments, one or more components of the reality augmented device 110 can be included in the external unit 190. For example, the external unit 190 can include an AR headset, which can include display device(s) 160, camera(s) 150, and / or sensor(s) 170. The AR headset can be a user-worn headset. Such user-worn headsets can include HMDs, glasses, AR / MR glasses, AR / MR visors, helmets, etc. In embodiments in which the AR headset is separate from the reality augmented device 110, the AR headset can be communicatively connected to the reality augmented device 110 and can interact with the augmentation component 134 to implement the techniques described herein.

[0033] 2 is a block diagram illustrating an exemplary workflow 200 for automatically placing and manipulating one or more virtual objects as part of an AR / MR experience, according to one embodiment. Workflow 200 can be performed by augmentation component 134, one or more components of augmented reality device 110, or any combination thereof.

[0034] In one embodiment, the augmentation component 134 can generate an environment representation 202. The environment representation 202 is typically a 3D point cloud or a 3D mesh that represents the real-world environment in which the reality augmentation device 110 is located. The augmentation component 134 can generate this environment representation 202 as part of an initial scan of the real-world environment using the camera(s) 150 and / or the sensor(s) 170 (e.g., LiDAR sensors). The augmentation component 134 can prompt the user to scan the real-world environment. For example, if scanning an indoor environment, the user can scan floors, walls, and / or one or more real-world objects within the indoor environment. If scanning an outdoor environment, the user can scan natural features (e.g., trees, bushes, etc.) within the outdoor environment. In one example, the augmentation component 134 can detect multiple surfaces within the real-world environment using the camera(s) 150 during the scan. One or more images of the real-world environment can be captured by the camera(s) 150, and the augmentation component 134 can determine the 3D geometry of the real-world environment based on the captured image(s). In another example, the augmentation component 134 can determine the 3D geometry of the real-world environment based on LiDAR scan results from scanning the real-world environment with one or more LiDAR sensors.

[0035] The augmentation component 134 can detect the planar geometry of various surfaces in a real-world environment using various computer vision techniques (e.g., scale-invariant feature transform (SIFT)) and / or software development kits (SDKs) (e.g., ARKit, ARCore, Wikitude, etc.) based on the camera(s) 150 and / or sensor(s) 170. Using one or more of these tools (e.g., using SIFT), the augmentation component 134 can process each image and extract a set of feature points (e.g., object edges, object corners, object centers, etc.) from each image. The augmentation component 134 can track the feature points across multiple images (i.e., multiple frames) as the reality augmentation device 110 moves during scanning.

[0036] The augmentation component 134 can then perform plane fitting on these feature points to find the plane that best matches in terms of scale, orientation, and position. This plane can then be continuously updated by the augmentation component 134 (e.g., based on feature extraction and plane fitting) as the reality augmentation device 110 moves during scanning. In this manner, the augmentation component 134 can determine planes for various surfaces in the real-world environment to generate the environment representation 202.

[0037] In some cases, the environment representation 202 may contain missing regions (e.g., holes, gaps, etc.). Such missing regions may be due to improper or incomplete scanning of the real-world environment. For example, the user may not have scanned all areas of the real-world environment (e.g., the ceiling in an indoor environment or the sky in an outdoor environment). In such cases, the augmentation component 134 may use computer image recognition techniques (e.g., Poisson reconstruction, open source libraries such as MeshFix) and / or deep learning techniques (e.g., Point2Mesh, 3D model inpainting based on a generative adversarial network (GAN)), to detect and fill the missing regions so that the environment representation 202 forms a watertight mesh. An example of an algorithm that can be used to detect and fill the missing regions is described in Attene, M., "A lightweight approach to repairing digitized polygon meshes," Visual Computer 26, 1393-1406 (2010).

[0038] In some cases, the augmentation component 134 may also perform detection of missing regions (e.g., ceiling in an indoor environment, sky in an outdoor environment, etc.) by inspecting the "y-up" range to see if there is a closed mesh within some predefined threshold (e.g., a predefined gradient (%)) in the "y-up" range. If there is no closed mesh within the "y-up" range, the augmentation component 134 may then use any of the computer image recognition and / or deep learning techniques described above to fill in the mesh within the "y-up" range so that the environment representation 202 forms a watertight mesh.

[0039] 2, the augmentation component 134 includes a visualization tool 136. The visualization tool 136 is generally configured to generate visualizations of virtual object(s) for an AR / MR experience. In one embodiment, the visualization tool 136 can determine how to automatically place and manipulate (e.g., adjust size and / or orientation) the virtual object(s) 208 based on an evaluation of the virtual object(s) 208 and the environment representation 202. The visualization tool 136 includes a transformation tool 210, an adaptation component 220, and an insertion tool 230, any of which can each include a software component, a hardware component, or a combination thereof.

[0040] Transformation tool 210 receives environment representation 202 and generates transformed environment representation 204 by performing one or more computer image recognition techniques on environment representation 202. In one embodiment, transformed environment representation 204 is a 3D point cloud or a 3D mesh that represents usable space (e.g., empty space) within the real-world environment. For example, transformation tool 210 can compute an inverse representation of environment representation 202 to generate a 3D point cloud or a 3D mesh that represents usable space within the real-world environment.

[0041] The adaptation component 220 receives the transformed environment representation 204 and the virtual object(s) 208. In one embodiment, the virtual object(s) 208 include one or more static virtual objects. Each of the static virtual objects may take the form of a 3D mesh. The adaptation component 220 may evaluate the transformed environment representation 204 using machine learning techniques (e.g., deep learning machine learning models) and adapt the 3D meshes representing the static virtual object(s) to the available space within the transformed environment representation 204.

[0042] In some embodiments, (as part of the adaptation step) the adaptation component 220 may automatically adjust the size and / or orientation of the virtual object(s) 208 so that the virtual object(s) 208 blend into the available space in the real-world environment. For example, consider if the virtual object(s) 208 include a virtual "globe" object. The adaptation component 220 may then adjust the size of the "globe" so that it occupies the maximum available space in the real-world environment. The maximum available space may vary depending on the type of real-world environment. As an example, consider if the real-world environment is a living room, the maximum available space may be the space above the coffee table in the living room. In another example, the maximum available space may be the empty space in a corner of the living room.

[0043] In some embodiments, the adaptation component 220 may consider the context (e.g., semantic meaning) of the virtual object(s) 208 and the context of the available space in the real-world environment when determining how to fit the virtual object(s) 208 into the available space in the transformed environment representation 204. For example, the adaptation component 220 may perform semantic segmentation to identify the “class” of each real-world item in the real-world environment and the “class” of the virtual object(s) 208.

[0044] For example, consider a case where the virtual object(s) 208 includes a "dog" virtual object and the real-world environment includes a first available space that is a space on a floor and a second available space that is a space on a table within the real-world environment. In this particular example, the matching component 220 can determine, based on performing semantic segmentation, that the class label of the first available space is "floor," the class label of the second available space is "table," and the class label of the virtual object is "dog." Then, based on these class labels, the matching component 220 can match the "dog" virtual object to the first available space (whose class label is "floor") rather than the second available space (whose class label is "table").

[0045] In some embodiments, the matching component 220 can automatically identify the semantic meaning of the various available spaces (e.g., based on performing semantic segmentation using a semantic deep learning neural network). In other embodiments, the matching component 220 can also identify the semantic meaning of the various available spaces based on user input in the form of criteria 214. For example, criteria 214 can specify a user's preferences regarding matching various types of virtual object(s) with various types of available spaces. Continuing with the example above, criteria 214 can specify that a "dog" virtual object(s) should be matched with available space on a "floor," as opposed to other types of available space, such as on a "table" or a "chair."

[0046] 2, the matching component 220 outputs correspondence information 206 based on an evaluation of the transformed environment representation 204 and the virtual object(s) 208. The correspondence information 206 may include a mapping between each of the virtual object(s) 208 and the available space within the transformed environment representation 204.

[0047] The insertion tool 230 receives the correspondence information 206 and generates rendering information 212 for rendering the virtual object(s) 208 on the display device(s) 160 of the reality augmentation device 110 so that the virtual object(s) 208 appear in the available space (of the real-world environment) to which they are matched as indicated in the correspondence information 206. In some embodiments, the insertion tool 230 can determine the rendering information 212 based at least in part on the transformed environment representation 204 and / or the virtual object(s) 208. The rendering information 212 may include, for example, a position in screen space (e.g., a two-dimensional (2D) position on the display device(s) 160) of each of the virtual object(s) 208.

[0048] 3 is a block diagram illustrating an exemplary workflow 300 for automatically placing and manipulating one or more virtual objects as part of an AR / MR experience, according to one embodiment. Workflow 300 can be implemented by augmentation component 134, one or more components of augmented reality device 110, or any combination thereof. In one embodiment, workflow 300 can be used to automatically place and manipulate one or more dynamic virtual objects as part of an AR / MR experience.

[0049] As discussed above in the discussion of FIG. 2 , the augmentation component 134 can generate an environment representation 202 that represents the real-world environment in which the reality augmentation device 110 is located. The environment representation 202 can be input to a path planning tool 138 that includes a path prediction tool 370. The path prediction tool 370 is generally configured to determine a set of predefined paths 380 that the user may take through the real-world environment. Each path 380 can include a 3D volume that represents the amount of space the user will occupy as they move through the real-world environment. The path prediction tool 370 can implement various path planning algorithms, such as, for example, heuristic search methods, intelligent algorithms (e.g., particle swarm algorithms, genetic algorithms, etc.), etc.

[0050] In some embodiments, the set of paths 380 may include various paths through a real-world environment for a single user having a set of user attributes 360. For example, consider the case where the user is an adult. In that case, the path prediction tool 370 may generate multiple paths 380 based on the user's height as specified in the user attributes 360. That is, each path 380 may include a 3D volume based in part on the user's height (e.g., vertical distance from the floor). In other embodiments, the set of paths 380 may include various paths through a real-world environment for multiple users having different sets of user attributes 360. For example, a first set of paths 380 may include various paths for an "adult" user having a particular range of heights (e.g., multiple 3D volumes based on a particular height range). Similarly, a second set of paths 380 may include various paths for a "child" user having another particular range of heights (e.g., multiple 3D volumes based on another particular height range).

[0051] The path prediction tool 370 may determine a set of paths 380 and provide them to a transformation tool 310 of the visualization tool 136. The transformation tool 310 may operate similarly to the transformation tool 210 described in the discussion of Figure 2. For example, the transformation tool 310 may compute an inverse representation of the environment representation 202 to generate multiple transformed environment representations 304 corresponding to the set of paths 380. Each transformed environment representation 304 may be a 3D point cloud or a 3D mesh representing the available space along a different path 380 through the real-world environment.

[0052] The matching component 320 can perform operations similar to the matching component 220. For example, the matching component 320 can receive the transformed environment representation 304 (corresponding to the paths 380) along with the virtual object(s) 208 and evaluate the information using one or more machine learning techniques (e.g., a (semantic) deep learning neural network model) to determine multiple correspondence information 306. Each correspondence information 306 can include, for each resulting path 380, a match between the virtual object(s) 208 and the available space in the transformed environment representation 304 that corresponds to the path 380.

[0053] In some embodiments, similar to the adaptation component 220, the adaptation component 320 can be configured to automatically consider the context (e.g., semantic meaning) of the virtual object(s) 208 along with the context of the available space of the transformed environment representation 304 when performing the adaptation process, where the context is determined based on performing semantic segmentation. In other embodiments, the adaptation component 320 can also determine the context from the criteria 214.

[0054] The insertion tool 330 may perform operations similar to the insertion tool 230. For example, the insertion tool 330 may receive the plurality of correspondence information 306 and generate rendering information 212 for rendering the virtual object(s) 208 on the display device(s) 160 of the reality augmentation device 110. The rendering of the virtual object(s) 208 may be performed such that the virtual object(s) 208 appear in the available space to which the path(s) correspond as indicated in the correspondence information 306 corresponding to the path(s). In some embodiments, the insertion tool 330 may also determine the rendering information 212 based at least in part on the plurality of transformed environment representations 304 and / or the virtual object(s) 208.

[0055] In some embodiments, the insertion tool 330 can output a different set of rendering information 212 based on the user's current location within the real-world environment. For example, the insertion tool 330 can receive sensor data 340 indicating the user's current location. Based on the user's current location, the insertion tool 330 can then determine from the plurality of correspondence information 306 whether to update the rendering of the virtual object(s) 208 and include the updated set of rendering information 212. In this manner, the augmentation component 134 can automatically place and manipulate the virtual object(s) based on the user's current location within the real-world environment as part of an AR / MR experience.

[0056] 4 is a flow chart illustrating a method 400 for automatically placing and manipulating one or more virtual objects, according to one embodiment. Method 400 can be performed by one or more components of a reality augmentation device (e.g., reality augmentation device 110). In one embodiment, method 400 is performed by an augmentation component of the reality augmentation device (e.g., augmentation component 134).

[0057] Method 400 begins at block 402, where the augmentation component begins scanning the real-world environment using one or more cameras (e.g., camera 150) and / or one or more sensors (e.g., sensor 170). At block 404, the augmentation component generates a first representation (e.g., environment representation 202) of the real-world environment based on the scan. The first representation of the real-world environment may include a 3D point cloud or a 3D mesh representing the real-world environment (e.g., a 3D representation of boundaries (e.g., floors, walls, etc.) and real objects (e.g., tables, chairs, trees, etc.) in the real-world environment).

[0058] At block 406, the augmentation component generates a second representation representing the real-world environment (e.g., transformed environment representation 204) based on the first representation representing the real-world environment. The second representation representing the real-world environment may include a 3D point cloud or a 3D mesh representing available space (e.g., empty space) within the real-world environment (e.g., a 3D representation representing available space within a boundary (e.g., a floor) or available space above or below a particular real object (e.g., a table surface) within the real-world environment). In one embodiment, the second representation representing the real-world environment may be generated by computing an inverse representation of the first representation representing the real-world environment.

[0059] At block 408, the augmentation component obtains an indication of one or more virtual objects (e.g., virtual object(s) 208) that can be placed in the real-world environment (e.g., as part of the AR / MR experience). The virtual object(s) can include at least one static or dynamic virtual object that can take the form of a 3D mesh.

[0060] At block 410, the augmentation component adapts the virtual object(s) to the available space(s) in the second representation of the real-world environment based on the evaluation of the virtual object(s) and the second representation of the real-world environment using one or more machine learning techniques. In one embodiment, the adapting step of block 410 may include determining and / or adjusting the size and / or orientation of the virtual object(s) to match the available space in the real-world environment to which they are adapted. The augmentation component may generate a set of information (e.g., correspondence information 206) that includes information indicative of such adaptation. In one embodiment, the processing of block 410 may be performed by the adaptation component 220, as detailed in the description of FIG. 2 above.

[0061] At block 412, the augmentation component generates information (e.g., rendering information 212) for rendering the virtual object(s) on a display device (e.g., display device(s) 160) of the augmented reality device based on the adaptation. For example, the rendering information may include at least one of: (i) where in screen space to render the virtual object(s), (ii) at what orientation to render the virtual object(s), or (iii) at what size to render the virtual object(s). Such rendering information may cause the virtual object(s) to appear on the display device at the appropriate size, position, and / or orientation according to the adaptation performed at block 410, as if they were located in the available space in the real-world environment.

[0062] At block 414, the augmentation component renders the virtual object(s) on a display device of the augmented reality device according to the rendering information. Note that in embodiments where the display device is part of an external unit (e.g., external unit 190), the augmentation component (at block 414) may render the virtual object(s) on the display device of the external unit.

[0063] 5 is a flow chart illustrating a method 500 for automatically placing and manipulating one or more virtual objects, according to one embodiment. Method 500 can be performed by one or more components of a reality augmentation device (e.g., reality augmentation device 110). In one embodiment, method 500 is performed by an augmentation component of a reality augmentation device (e.g., augmentation component 134).

[0064] Method 500 is entered at block 502, where the augmentation component begins scanning the real-world environment using one or more cameras (e.g., camera 150) and / or one or more sensors (e.g., sensor 170). At block 504, the augmentation component generates a first representation (e.g., environment representation 202) of the real-world environment based on the scan. The first representation of the real-world environment may include a 3D point cloud or a 3D mesh representing the real-world environment (e.g., 3D representations of boundaries (floors, walls, etc.) and real objects (tables, chairs, etc.) in the real-world environment).

[0065] At block 506, the augmentation component generates a set of user paths (e.g., paths 380) through the real-world environment based in part on the first representation of the real-world environment. In some embodiments, the augmentation component may generate the set of user paths based on one or more user attributes (e.g., user attributes 360) in addition to the first representation. In one embodiment, the processing of block 506 may be performed by path planning tool 138, as detailed in the description of FIG. 3 above.

[0066] At block 508, the augmentation component generates a plurality of second representations (e.g., transformed environment representations 304) representing the real-world environment based on the first representation representing the real-world environment and the set of user paths. Each of the second representations representing the real-world environment may include a 3D point cloud or a 3D mesh representing available space along different paths in the real-world environment. In one embodiment, each of the second representations representing the real-world environment may be generated by computing an inverse representation of each path in the environment representation 202.

[0067] At block 510, the augmentation component obtains an indication of one or more virtual objects (e.g., virtual object(s) 208) that can be placed in a real-world environment (e.g., as part of an AR / MR experience). The virtual object(s) can include at least one dynamic (e.g., changing) virtual object that can take the form of a 3D mesh.

[0068] At block 512, the augmentation component generates, for each second representation of a real-world environment, a set of information (e.g., correspondence information 306) for fitting the virtual object(s) to the available space(s) in the second representation of the real-world environment. The generation of such information is based on an evaluation of the virtual object(s) and the second representation of the real-world environment using one or more machine learning techniques. In one embodiment, the fitting step of block 512 can include determining and / or adjusting the size and / or orientation of the virtual object(s) to fit the available space in each user path in the real-world environment to which they are being fitted. In one embodiment, the processing of block 512 can be performed by the fitting component 320, as detailed in the description of FIG. 3 above.

[0069] At block 514, the augmentation component generates information (e.g., rendering information 212) for rendering the virtual object(s) on a display device (e.g., display device(s) 160) of the reality augmented device based on the above-described matching information and the current position of the reality augmented device. For example, the rendering information may include at least one of: (i) where in screen space to render the virtual object(s), (ii) at what orientation to render the virtual object(s), or (iii) at what size to render the virtual object(s). Such rendering information may cause the virtual object(s) to appear on the display device at an appropriate size, position, and / or orientation, as if they were located in the available space associated with the user's current position in the real-world environment.

[0070] At block 516, the augmentation component renders the virtual object(s) on a display device of the augmented reality device according to the rendering information. Note that in embodiments where the display device is part of an external unit (e.g., external unit 190), the augmentation component (at block 516) may render the virtual object(s) on the display device of the external unit.

[0071] Although reference is made in this disclosure to various embodiments, it should be understood that the disclosure is not limited to the specific embodiments described. Rather, it is contemplated that the following features and elements can be combined in any combination to implement and practice the teachings herein, regardless of whether the embodiments to which they are associated are different. Furthermore, when elements of an embodiment are described in the form of "at least one of A and B," it will be understood that embodiments including only element A, embodiments including only element B, and embodiments including elements A and B are contemplated. Furthermore, while some embodiments may provide advantages over other possible solutions or over the prior art, whether or not a given embodiment provides a particular advantage is not a limitation of the disclosure. Accordingly, the aspects, features, embodiments, and advantages disclosed herein are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claim(s). Similarly, any reference to "the invention" should not be construed as a generalization of any inventive subject matter disclosed herein, and should not be considered an element or limitation of the appended claims unless expressly recited in one or more claims.

[0072] As will be appreciated by those skilled in the art, the embodiments described herein may be embodied as a system, a method, or a computer program product. Accordingly, the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be collectively referred to herein as a "circuit," "module," or "system." Furthermore, the embodiments described herein may also take the form of a computer program product, and may be embodied as one or more computer-readable medium(s) and computer-readable program code on the computer-readable medium(s).

[0073] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0074] Computer program code for carrying out processes according to embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language, or similar programming languages. The program code may run entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0075] Aspects of the present disclosure are described herein with reference to flowchart diagrams and block diagrams that illustrate methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart diagrams or block diagrams, and combinations of blocks in the flowchart diagrams or block diagrams, can be embodied by computer program instructions. These computer program instructions can then be provided to a processor of a programmable data processing device, such as a general-purpose computer or a special-purpose computer, to form a machine, such that the instructions, executed by the processor of the programmable data processing device, such as a computer, can implement the function(s) / act(s) identified in the block(s) of the flowchart diagrams or block diagrams.

[0076] These computer program instructions may be stored on a computer-readable medium that can instruct a device, such as a computer or other programmable data processing apparatus, to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture that includes instructions that implement the function(s) / act(s) identified in the block(s) of the flowchart or block diagram.

[0077] Computer program instructions may also be loaded into a device such as a computer or other programmable data processing apparatus to cause a series of processing steps to be performed on the device, such as a computer or other programmable data processing apparatus, to generate a computer-implemented process, whereby the instructions executing on the device, such as a computer or other programmable data processing apparatus, provide a process for implementing the functions / acts identified in one or more blocks of the flowchart or block diagrams.

[0078] The flowchart diagrams and block diagrams depicted in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart diagrams or block diagrams can be considered to represent a module, segment, or portion of code, including one or more executable instructions for implementing the logical function(s) identified therein. It should also be noted that in some alternative implementations, the functions noted in the blocks can be executed in a different order than that depicted in the figures. For example, two blocks shown as consecutive blocks can, in fact, be executed substantially simultaneously, or the blocks can be executed in the reverse order or in a different order, depending on the functionality involved. It should also be noted that each block of the block diagrams or flowchart diagrams, and combinations of blocks in the block diagrams or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the functions or operations identified therein, or by a combination of dedicated hardware and computer instructions.

[0079] It should be noted that while the foregoing description is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is defined by the following claims.

[0080] The present invention can be implemented in the following embodiments.

[0081] Embodiment 1 1. A computer-implemented method comprising: obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment with one or more sensors of a computing device; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on a display of the computing device, the location associated with the at least one available space in the real-world environment, based on the matching; A method comprising:

[0082] Embodiment 2 2. The computer-implemented method of claim 1, wherein determining a fit between the at least one virtual object and the at least one available space in the real-world environment includes determining a size of the at least one virtual object that blends into the at least one available space.

[0083] Embodiment 3 3. The computer-implemented method of claim 2, wherein rendering the at least one virtual object comprises rendering the at least one virtual object on the display of the computing device at the determined size.

[0084] Embodiment 4 determining a plurality of paths for the computing device through the real-world environment; 2. The computer-implemented method of claim 1, wherein the at least one second representation is indicative of the at least one available space along a first path of the plurality of paths.

[0085] Embodiment 5 5. The computer-implemented method of claim 4, wherein the plurality of paths comprises a first set of paths through the real-world environment based on a first height range and a second set of paths through the real-world environment based on a second height range.

[0086] Embodiment 6 determining a plurality of paths for the computing device through the real-world environment; The computer-implemented method of claim 1 , wherein the at least one virtual object is adapted to the at least one available space along a first path of the plurality of paths.

[0087] Embodiment 7 wherein rendering the at least one virtual object includes receiving sensor data indicative of a position of the computing device along the first path; 7. The computer-implemented method of claim 6, wherein the location on the display corresponds to the location of the computing device within the at least one available space along the first path.

[0088] Embodiment 8 The computer-implemented method of claim 1 , wherein the at least one virtual object is a static virtual object.

[0089] Embodiment 9 The computer-implemented method of claim 1 , wherein the at least one virtual object is a dynamic virtual object.

[0090] Embodiment 10 The display and one or more sensors; one or more processors; a memory for storing instructions; 1. A computing device comprising: When executed on the one or more processors, the instructions obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment by the one or more sensors; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on the display associated with the at least one available space in the real-world environment based on the matching; A computing device on which processing including

[0091] Embodiment 11 11. The computing device of claim 10, wherein determining a fit between the at least one virtual object and the at least one available space in the real-world environment includes determining a size of the at least one virtual object that blends into the at least one available space.

[0092] Embodiment 12 12. The computing device of claim 11, wherein rendering the at least one virtual object comprises rendering the at least one virtual object on the display at the determined size.

[0093] Embodiment 13 the processing further comprises determining a plurality of paths for the computing device through the real-world environment; The computing device of claim 10 , wherein the at least one second representation further indicates the at least one available space along a first path of the plurality of paths.

[0094] Embodiment 14 14. The computing device of claim 13, wherein the plurality of paths comprises a first set of paths through the real-world environment based on a first height range and a second set of paths through the real-world environment based on a second height range.

[0095] Embodiment 15 the processing further comprises determining a plurality of paths for the computing device through the real-world environment; The computing device of claim 10 , wherein the at least one virtual object is adapted to the at least one available space along a first path of the plurality of paths.

[0096] Embodiment 16 wherein rendering the at least one virtual object includes receiving sensor data indicative of a position of the computing device along the first path; 16. The computing device of claim 15, wherein the location on the display corresponds to the location of the computing device within the at least one available space along the first path.

[0097] Embodiment 17 1. A computer-implemented method comprising: obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment with one or more sensors of a computing device; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on a display of the computing device, the location associated with the at least one available space in the real-world environment, based on the matching; Including, A method for locating a user within the real-world environment and automatically rendering virtual objects in available space based on the user's location, user attributes, and correspondence information.

[0098] Embodiment 18 The display and one or more sensors; one or more processors; a memory for storing instructions; 1. A computing device comprising: When executed on the one or more processors, the instructions obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment by the one or more sensors; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on the display associated with the at least one available space in the real-world environment based on the matching; The process including A computing device that determines a user's location within the real-world environment and automatically renders virtual objects in the available space based on the user's location, user attributes, and correspondence information. [Explanation of symbols]

[0099] 110 Augmented Reality Devices 120 processors 130 memory 132 Applications 134 Extended Components 136 Visualization Tools 138 Route Planning Tool 140 Storage device 150 cameras 160 Display Devices 170 sensors 180 Network Interface 190 External Unit 202 Environmental expression 204, 304 Transformational environment representation 206, 306 Correspondence information 208 Virtual Objects 210, 310 conversion tool 212 Rendering Information 214 Standards 220, 320 compatible components 230, 330 Insertion Tool 340 Sensor Data 360 User Attributes 370 Route Prediction Tool 380 routes

Claims

1. 1. A computer-implemented method comprising: obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment with one or more sensors of a computing device; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on a display of the computing device, the location associated with the at least one available space in the real-world environment, based on the matching; Including, determining a user's location within the real-world environment and automatically rendering virtual objects in the available space based on the user's location, user attributes, and correspondence information; The method, wherein the at least one second representation of a real-world environment is a 3D point cloud or a 3D mesh representing empty space within the real-world environment.

2. 2. The computer-implemented method of claim 1, wherein determining a fit between the at least one virtual object and the at least one available space in the real-world environment includes determining a size of the at least one virtual object that blends into the at least one available space.

3. 3. The computer-implemented method of claim 2, wherein rendering the at least one virtual object comprises rendering the at least one virtual object on the display of the computing device at the determined size.

4. determining a plurality of paths for the computing device through the real-world environment; The computer-implemented method of claim 1 , wherein the at least one second representation is indicative of the at least one available space along a first path of the plurality of paths.

5. 5. The computer-implemented method of claim 4, wherein the plurality of paths comprises a first set of paths through the real-world environment based on a first height range and a second set of paths through the real-world environment based on a second height range.

6. determining a plurality of paths for the computing device through the real-world environment; The computer-implemented method of claim 1 , wherein the at least one virtual object is adapted to the at least one available space along a first path of the plurality of paths.

7. wherein rendering the at least one virtual object includes receiving sensor data indicative of a position of the computing device along the first path; 7. The computer-implemented method of claim 6, wherein the location on the display corresponds to the location of the computing device within the at least one available space along the first path.

8. The computer-implemented method of claim 1 , wherein the at least one virtual object is a static virtual object.

9. The computer-implemented method of claim 1 , wherein the at least one virtual object is a dynamic virtual object.

10. The display and one or more sensors; one or more processors; a memory for storing instructions; 1. A computing device comprising: When the instructions are executed on the one or more processors, obtaining an indication of at least one virtual object placeable within a real-world environment as part of an augmented reality (AR) or mixed reality (MR) simulation; generating a first representation of the real-world environment based on scanning the real-world environment with the one or more sensors; generating, from the first representation of the real-world environment, at least one second representation of the real-world environment, the at least one second representation showing at least one available space within the real-world environment; evaluating the at least one second representation of the at least one virtual object and the real-world environment using one or more machine learning models and determining a match between the at least one virtual object and the at least one available space in the real-world environment based at least in part on the evaluation; rendering the at least one virtual object at a location on the display associated with the at least one available space in the real-world environment based on the matching; The process including determining a user's location within the real-world environment and automatically rendering virtual objects in the available space based on the user's location, user attributes, and correspondence information; a computing device, wherein the at least one second representation of a real-world environment is a 3D point cloud or a 3D mesh representing empty space within the real-world environment;

11. 11. The computing device of claim 10, wherein determining a fit between the at least one virtual object and the at least one available space in the real-world environment comprises determining a size of the at least one virtual object that blends into the at least one available space.

12. The computing device of claim 11 , wherein rendering the at least one virtual object comprises rendering the at least one virtual object on the display at the determined size.

13. the processing further comprises determining a plurality of paths for the computing device through the real-world environment; The computing device of claim 10 , wherein the at least one second representation further indicates the at least one available space along a first path of the plurality of paths.

14. 14. The computing device of claim 13, wherein the plurality of paths comprises a first set of paths through the real-world environment based on a first height range and a second set of paths through the real-world environment based on a second height range.

15. the processing further comprises determining a plurality of paths for the computing device through the real-world environment; The computing device of claim 10 , wherein the at least one virtual object is adapted to the at least one available space along a first path of the plurality of paths.

16. wherein rendering the at least one virtual object includes receiving sensor data indicative of a position of the computing device along the first path; The computing device of claim 15 , wherein the location on the display corresponds to the location of the computing device within the at least one available space along the first path.

Images