Adapting space and content for augmented and mixed reality

Abstract

To allow a user to experience immersiveness in a physical environment without relying on operation of the user for setting of AR / MR experience.SOLUTION: At the time of an AR simulation or MR simulation, a designation of at least one virtual object which can be arranged in a real world environment is obtained. On the basis of scanning of the real world environment, a first representation expressing the real world environment is generated. From the first representation, a second representation expressing at least one real world environment is generated. Using a machine learning model, the at least one virtual object and the at least one second representation are evaluated. Based at least in part on the evaluation, adaptation of the at least one virtual object and at least one usable space present in the real world environment is determined. On the basis of this adaptation, the at least one virtual object is rendered on a computing device.SELECTED DRAWING: Figure 4

Description

Cross-Reference to Related Art

[0001] This application is a divisional application of Japanese Patent Application No. 2023-79217 filed on May 12, 2023. Technical Field

[0002] The present disclosure relates to spatial and content adaptation for augmented reality and mixed reality. Background Art

[0003] Nowadays, augmented reality (AR) technology and mixed reality (MR) technology are being used in a wide variety of fields, for example, such fields as medical care, social networks and social participation, communication, shopping, entertainment industry, travel, navigation, and education. AR is a technology that superimposes computer-generated imagery on the real-world environment viewed by a user. Using an image device, an AR system can display continuous images of the real-world environment (e.g., a video feed) to a user with various virtual objects inserted at appropriate positions in the real-world environment. For example, by identifying a real-world object that is a "table", an AR system can display a virtual object of a "cup" on the image device such that the virtual object of the "cup" appears (e.g., from the perspective of the image device) to be placed on the "table". On the other hand, MR is generally considered an extended version of AR that enables interaction between real objects and virtual objects within an environment. For example, in an MR experience, a user can adapt and operate visual content (e.g., graphics for games or interactive narratives) to be overlaid on the physical environment being viewed.

[0004] However, in most AR / MR experiences, the placement and resizing of virtual objects on the user's display device—which is necessary to "fit" the virtual objects into the user's real-world environment—is left to the user. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, relying on user input to configure AR / MR experiences in this way could potentially hinder users from having an immersive experience within a physical environment. [Means for solving the problem]

[0006] One embodiment described herein is a computer-implemented method. The computer-implemented method includes the step of obtaining a representation of at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation. The computer-implemented method further includes the step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of a computing device. The computer-implemented method further includes the step of generating at least one second representation of the real-world environment from the first representation of the real-world environment, the second representation of the real-world environment which indicates at least one available space within the real-world environment. The computer-implemented method further includes the step of evaluating the at least one virtual object and the at least one second representation of the real-world environment using one or more machine learning models, and determining, at least in part, that the at least one virtual object and the at least one available space within the real-world environment are compatible. A computer-implemented version of this method, based on this adaptation, further includes the step of rendering at least one virtual object at a location on the display of a computing device, which is associated with at least one available space within a real-world environment.

[0007] Other embodiments described herein are computing devices. These computing devices include a display, one or more sensors, one or more processors, and memory for storing instructions, wherein when an instruction is executed on one or more processors, processing is performed. This processing includes the step of obtaining something that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation. This processing further includes the step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors. This processing further includes the step of generating at least one second representation of the real-world environment from the first representation of the real-world environment, the second representation of the real-world environment which indicates at least one available space within the real-world environment. This processing further includes the step of evaluating at least one virtual object and at least one second representation of the real-world environment using one or more machine learning models, and determining a match between at least one virtual object and at least one available space within the real-world environment, at least in part, based on the evaluation. Based on this adaptation, the process further includes the step of rendering at least one virtual object at a location on the display that is associated with at least one available space within a real-world environment.

[0008] Other embodiments described herein include a non-temporary computer-readable medium containing computer program code, and the process is performed when the computer program code is executed by the operation of one or more computer processors. The process includes the step of obtaining something that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation. The process further includes the step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of a computing device. The process further includes the step of generating at least one second representation of the real-world environment from the first representation of the real-world environment, the second representation of the real-world environment which indicates at least one available space within the real-world environment. The process further includes the step of evaluating at least one virtual object and at least one second representation of the real-world environment using one or more machine learning models, and determining a match between at least one virtual object and at least one available space within the real-world environment, at least in part, based on the evaluation. Based on this adaptation, the process further includes the step of rendering at least one virtual object at a location on the display of a computing device, which is associated with at least one available space within a real-world environment.

[0009] To enable the realization and detailed understanding of each of the above embodiments, the embodiments described herein, which have been briefly summarized above, will be described in more detail with reference to the attached drawings.

[0010] However, it should be noted that the attached drawings illustrate typical embodiments and are therefore not intended to be limiting, and other embodiments with similar effects are conceivable. [Brief explanation of the drawing]

[0011] [Figure 1]Block diagram showing an exemplary augmented reality device according to one embodiment. [Figure 2] A block diagram illustrating an exemplary workflow for automatically placing and manipulating (one or more) virtual objects as part of an AR / MR experience, according to one embodiment. [Figure 3] A block diagram illustrating another exemplary workflow for automatically placing and manipulating (one or more) virtual objects as part of an AR / MR experience, according to one embodiment. [Figure 4] A flowchart illustrating a method for automatically arranging and manipulating (one or more) virtual objects according to one embodiment. [Figure 5] A flowchart illustrating another method for automatically locating and manipulating (one or more) virtual objects, according to one embodiment. [Modes for carrying out the invention]

[0012] Many AR / MR experiences allow users to view virtual objects within real-world environments and interact with them virtually. These virtual objects can be digital representations (or include digital representations) of people, fictional characters, places, items, brands, logos, and other identifiers that constitute a virtual reality (VR), AR, or MR environment. Furthermore, virtual objects can depict a virtual world that any number of users can experience continuously and synchronously, and can also provide continuity in data such as personal identity, user history, entitlement, ownership, and payments. For example, AR / MR experiences can allow users to "preview" various virtual furniture items in a specific real-world environment (e.g., a living room, workplace, bedroom, or other physical space), view and interact with virtual characters in their real-world environment as part of an interactive narrative experience, view and interact with renderings while playing an interactive game, or view and interact with other virtual users (e.g., avatars) in a virtual world.

[0013] However, traditional AR / MR experiences have a problem in that they largely leave it up to the user to position and resize virtual objects on their computing device so that the virtual objects blend into the user's real-world environment. For example, consider a case where a user starts a new interactive story experience on their computing device. In this case, as part of the interactive story experience, the computing device renders virtual objects (for example, representations of characters or items in the interactive story, or representations of virtual scenery such as natural features and landscapes in the interactive story) on the computing device, allowing the user to visualize these virtual objects in their real-world environment (via the computing device).

[0014] However, when computing devices render virtual objects, there was a risk that they might render them without context for one or more attributes of the real-world environment (for example, the size of the real-world environment, or the size / placement / orientation of real items in the real-world environment). For example, consider a real-world environment that includes a "real" physical table, and the virtual object is a virtual character. In this case, the computing device might render the virtual character on the device in a way that makes it appear as if the virtual character is standing on the table, rather than appearing as if the virtual character is standing on the floor next to the table. In addition to or instead of this, the computing device might render the virtual character at an inaccurate size on the screen, resulting in the virtual character appearing inaccurately sized relative to the table (for example, too small or too large relative to the table from the computing device's perspective).

[0015] In conventional AR / MR experiences, the placement and sizing of virtual objects generally relied on an initial calibration performed by the user at the start of the AR / MR experience. During the initial calibration, the user was prompted to place a virtual character on a table and adjust its size appropriately (for example, by adjusting the size of the virtual character on the computing device's screen), which allowed subsequent virtual objects (for example, of the same or different type as the first virtual object) to be placed on the same table at the same size. After the initial calibration, subsequent virtual objects could be automatically placed under the assumption that the environmental information had not changed since the initial calibration. However, relying on users to manually perform this initial calibration could confuse them, hinder the creation of an immersive illusion, and potentially lead to negative user experiences, such as users accidentally bumping into real-world objects while trying to manipulate virtual objects.

[0016] To address this, embodiments of this specification describe a technique for automatically adjusting the attributes (e.g., size, position, etc.) of (one or more) virtual objects rendered on a computing device so that they conform to a real-world environment. In other words, according to several embodiments, it is possible to automatically adjust and customize (one or more) virtual objects to a given real-world environment without manual intervention by the user, by automatically determining how well or poorly the user's real-world environment conforms to the planned (one or more) virtual objects.

[0017] In one embodiment described below, the computing device includes an extension component configured to generate a three-dimensional (3D) mesh representing available space (e.g., empty space) within the real-world environment in which the user is located. This real-world environment may be an internal physical environment (e.g., an indoor space such as a room) or an external physical environment (e.g., an outdoor space such as a backyard or park). The extension component can also retrieve (one or more) virtual objects planned for the real-world environment (e.g., from an AR / MR application running on the computing device). For example, (one or more) virtual objects may be (or include) digital representations of people, fictional characters, places, item characters, items, etc., associated with an AR / MR experience, and by rendering (one or more) virtual objects on the computing device, the user can visualize these virtual objects within the real-world environment. Such (one or more) virtual objects may include (one or more) static virtual objects and / or (one or more) dynamic virtual objects.

[0018] The extension component can use one or more machine learning techniques to evaluate (i) a 3D mesh representing available space in a real-world environment and (ii) one or more virtual objects, thereby identifying a pair of correspondences between these virtual objects and the available space in the real-world environment. For example, by making such identification, the extension component can fit the virtual object of a "talking teapot" in an interactive AR / MR experience to the flat surface of a real-world table as seen on the user's computing device. The extension component can then automatically render the virtual object on the computing device so that, from the computing device's perspective, the virtual object of the "talking teapot" appears on the flat surface of the table.

[0019] In addition, in some embodiments, the augmented component can identify multiple correspondences between available space in a real-world environment and (one or more) dynamic (e.g., changing) virtual objects. For example, the augmented component can identify a set of user paths in a real-world environment (e.g., a room) and, for each user path, identify a 3D mesh representing the available space in the real-world environment.

[0020] For example, in a real-world environment, a user walking around a real object (e.g., a table) in a first direction (e.g., right) can be represented by a first path, and a user walking around the same real object in a second direction (e.g., left) can be represented by a second path. In such an example, the augmented component can generate a first 3D mesh representing the available space in the real-world environment associated with the first path, and a second 3D mesh representing the available space in the real-world environment associated with the second path. The augmented component can then use machine learning techniques to identify the correspondence between the available space in the real-world environment represented by each 3D mesh and (one or more) virtual objects for each user path.

[0021] Then, as the user moves within the real-world environment, the extension component identifies the user's position along the obtained path (e.g., based on one or more sensors of the computing device), and can automatically render the one or more virtual objects in the available space based on the user's position, the user's attributes (e.g., the user's height and accessibility limitations), and correspondence information. As a reference example, considering the virtual object of a "superhero character" in an interactive gaming experience, when the user is on a first path, the virtual object can be rendered to be visible on a first side of the user, and when the user is on a second path, the virtual object can be rendered to be visible on a second side that is the other side of the user.

[0022] As described above, according to multiple embodiments, placement and manipulation of one or more virtual objects to be arranged on a user's computing device as part of an AR / MR experience can be automatically performed, and this can be achieved without relying on the user to manually manipulate the virtual objects. As a result, according to multiple embodiments, a computing device can be caused to generate a dynamic AR / MR experience tailored to an individual user that automatically adapts to multiple users and real-world environments.

[0023] It should be noted that in many of the following embodiments, an interactive narrative experience is used as a reference example of an AR / MR experience that can use the technology for automatically performing placement and manipulation of virtual objects described in the present specification. However, it should be noted that the technology described in the present specification can be used in a wide variety of AR / MR experiences, such as, for example, shopping, travel, social interaction (e.g., "multiverse"), navigation, gaming, and the like.

[0024] FIG. 1 is a block diagram showing an augmented reality device 110 configured to include an extension component 134 according to an embodiment. The augmented reality device 110 generally refers to various computing devices such as smartphones, tablet terminals, laptop computers, headsets, visors, head mounted displays (HMDs), and eyeglasses. The augmented reality device 110 may be an AR-enabled computing device, an MR-enabled computing device, or an AR / MR-enabled computing device. The augmented reality device 110 can implement one or more techniques described herein for automatically placing and manipulating virtual objects as part of an AR / MR experience.

[0025] The augmented reality device 110 includes a processor 120, a memory 130, a storage device 140, one or more cameras 150, one or more display devices 160, one or more sensors 170, and a network interface 180. The processor 120 refers to any number of processing elements and may include any number of processing cores. The memory 130 may include volatile memory, non-volatile memory, and combinations thereof. The memory 130 generally contains program code for executing various functions related to applications (e.g., application 132) hosted on the augmented reality device 110.

[0026] The application 132 refers to a client-server application (or other distributed application) component. The application 132 may be a "thin" client: most of the processing execution instructions are issued by the application 132 side, while the processing execution may be performed by a computing system of a backend server (not shown) or a conventional software application installed on the augmented reality device 110. The application 132 includes the extension component 134. Details of the extension component 134 will be described later.

[0027] The storage device 140 can be a disk drive storage device. Although the diagram shows the storage device 140 as a single unit, the storage device 140 can be a combination of a fixed storage device and / or a removable storage device. Examples of such storage devices include a fixed disk drive, a removable memory card, or an optical storage device, a network-attached storage device (NAS), or a storage area network (SAN). The network interface 180 can be any type of network communication interface that enables the augmented reality device 110 to communicate with other computers and / or components in the computing environment (e.g., an external unit 190) via a data communication network.

[0028] One or more display devices 160 and one or more cameras 150 enable the user to view the real-world environment in which the augmented reality device 110 is placed, from the viewpoint of the augmented reality device 110. For example, one or more cameras 150 can capture a visual scene. In this specification, a visual scene refers to one or more views of the real-world environment in which the augmented reality device 110 is being used. For example, a visual scene can be a series of images (e.g., a video feed) of the real-world environment. The visual scene can be displayed on one or more display devices 160. For example, by activating one or more cameras 150, the visual scene can be projected onto one or more display devices 160, and virtual objects can be overlaid on this visual scene. The (one or more) display devices 160 include any suitable display technology that performs a physical conversion from signal to light, such as (one or more) liquid crystal displays (LCDs), (one or more) light-emitting diodes (LEDs), (one or more) organic light-emitting diode displays (OLEDs), or (one or more) quantum dot (QD) displays. The (one or more) cameras 150 may be configured to work in conjunction with image recognition software (for example, stored in memory 130) for identifying real objects within the field of view of the cameras 150.

[0029] One or more sensors 170 are configured to sense information from the real-world environment. In one embodiment, one or more sensors 170 include an accelerometer, a gyroscope, or a combination thereof. The accelerometer can measure the accelerating force acting on the augmented reality device 110 and can provide information about whether the augmented reality device 110 is moving and in which direction it is moving. The accelerometer can also be used to determine the tilt of the augmented reality device 110. The gyroscope can measure the orientation of the augmented reality device 110 and can provide information about whether the augmented reality device 110 is horizontal or what the tilt of the augmented reality device 110 is in one or more planes. In one embodiment, by combining the accelerometer and gyroscope, information about the augmented reality device 110's sense of direction can also be provided in terms of gravity-referential pitch and roll.

[0030] In general, the augmented reality device 110 may be equipped with any number of sensors as a configuration for determining the orientation (e.g., tilt) of the augmented reality device 110, and / or may utilize any technology or combination of technology suitable for the functions described herein. Similarly, the (one or more) sensors 170 may include various types of sensors and are not limited to accelerometers and gyroscopes. Other types of sensors 170, but not limited to, include any type of sensor that provides information about the attitude and position of the augmented reality device 110 in a real-world environment, such as a Light Detection and Ranging (LiDAR) sensor, a Global Positioning System (GPS) receiver, or an inertial motion unit (IMU).

[0031] The extension component 134 generally enables the user to visualize (one or more) virtual objects and interact with them virtually in a real-world environment. In the embodiments described herein, the extension component 134 can automatically arrange and manipulate virtual objects that are overlaid on a visual scene of a real-world environment. The extension component 134 comprises a visualization tool 136 and a routing tool 138, each of which may include software components, hardware components, or a combination thereof. Details of the visualization tool 136 and the routing tool 138 will be described later.

[0032] In some embodiments, the augmented reality device 110 may also provide the user with an AR / XR experience by using and / or interacting with an external unit 190. In such embodiments, one or more components of the augmented reality device 110 may be included within the external unit 190. For example, the external unit 190 may include an AR headset, which may be equipped with (one or more) display devices 160, (one or more) cameras 150, and / or (one or more) sensors 170. The AR headset may be a user-worn headset. Examples of such user-worn headsets include HMDs, glasses, AR / MR glasses, AR / MR visors, helmets, etc. In embodiments where the AR headset is separate from the augmented reality device 110, the AR headset may be communicatively connected to the augmented reality device 110 and exchange information with the augmentation component 134 to realize the technology described herein.

[0033] Figure 2 is a block diagram illustrating an exemplary workflow 200 for automatically placing and manipulating (one or more) virtual objects as part of an AR / MR experience, according to one embodiment. The workflow 200 can be implemented by the augmentation component 134, one or more components of the augmented reality device 110, or any combination thereof.

[0034] In one embodiment, the augmented component 134 can generate an environment representation 202. The environment representation 202 is generally a 3D point cloud or 3D mesh representing the real-world environment in which the augmented reality device 110 is placed. The augmented component 134 can generate this environment representation 202 as part of an initial scan of the real-world environment using (one or more) cameras 150 and / or (one or more) sensors 170 (e.g., LiDAR sensors). The augmented component 134 can prompt the user to scan the real-world environment. For example, when scanning an indoor environment, the user can scan the floor, walls, (one or more) real objects within that indoor environment. When scanning an outdoor environment, the user can scan natural features within that outdoor environment (e.g., trees, bushes, etc.). In one example, the augmented component 134 can use (one or more) cameras 150 to detect multiple surfaces within the real-world environment during the scan. One or more cameras 150 can capture one or more images of the real-world environment, and the extension component 134 can determine the 3D geometry of the real-world environment based on the captured images. In another example, the extension component 134 can also determine the 3D geometry of the real-world environment based on LiDAR scan results obtained by scanning the real-world environment using one or more LiDAR sensors.

[0035] The augmented component 134, based on (one or more) cameras 150 and / or (one or more) sensors 170, can detect the planar geometry of various surfaces in a real-world environment using various computer vision techniques (e.g., scale-invariant feature transform: SIFT) and / or software development kits (SDKs) (e.g., ARKit, ARCore, Wikitude, etc.). Using one or more of these tools (e.g., using SIFT), the augmented component 134 can process each image and extract a set of feature points (e.g., object edges, corners, object centers, etc.) from each image. The augmented component 134 can track feature points across multiple images (i.e., multiple frames) as the augmented reality device 110 moves during scanning.

[0036] The augmented component 134 then performs plane fitting on these feature points to find the plane with the best match in terms of scale, orientation, and position. The detected plane can then be continuously updated by the augmented component 134 (for example, based on feature extraction and plane fitting) as the augmented reality device 110 moves during scanning. In this way, the augmented component 134 can determine planes for various surfaces in the real-world environment and generate an environment representation 202.

[0037] In some cases, the environment representation 202 may have missing areas (e.g., holes, gaps, etc.). Such missing areas may result from inadequate or incomplete scans of the real-world environment. For example, the area scanned by the user may not cover all areas in the real-world environment (e.g., the ceiling of an indoor environment or the sky of an outdoor environment may not have been scanned). In such cases, the extension component 134 can use computer image recognition techniques (e.g., Poisson reconstruction, open-source libraries (MeshFix, etc.)) and / or deep learning techniques (e.g., Point2Mesh, Generative Adversarial Network (GAN)-based 3D model repair (inpainting), etc.) to detect and fill in the missing areas so that the environment representation 202 forms a watertight mesh. An example of an algorithm that can be used to detect and fill in missing areas is the algorithm described in Attene, M, "A lightweight approach to repairing digitized polygon meshes," Visual Computer 26, 1393-1406 (2010).

[0038] In some cases, the extended component 134 can also detect missing areas (e.g., the ceiling in an indoor environment, the sky in an outdoor environment) by checking the range in the "y-up" direction and confirming whether there are closed meshes within some predefined thresholds (e.g., a predefined gradient (%)) in the range in the "y-up" direction. If there are no closed meshes within the "y-up" range, the extended component 134 can use either the computer image recognition technique and / or deep learning technique described above to fill in the mesh within the "y-up" range so that the environment representation 202 forms a watertight mesh.

[0039] As shown in Figure 2, the extension component 134 includes a visualization tool 136. The visualization tool 136 is generally configured to generate visualizations of (one or more) virtual objects for an AR / MR experience. In one embodiment, the visualization tool 136 can determine how to automatically position and manipulate (e.g., adjust the size and / or orientation) the (one or more) virtual objects 208 and the environment representation 202 based on an evaluation of them. This visualization tool 136 includes a transformation tool 210, a fitting component 220, and an insertion tool 230, each of which may include a software component, a hardware component, or a combination thereof.

[0040] The conversion tool 210 receives an environment representation 202 and generates a converted environment representation 204 by performing one or more computer image recognition techniques on the environment representation 202. In one embodiment, the converted environment representation 204 is a 3D point cloud or 3D mesh representing available space (e.g., empty space) within a real-world environment. For example, the conversion tool 210 can compute an inverse representation of the environment representation 202 in order to generate a 3D point cloud or 3D mesh representing available space within a real-world environment.

[0041] The fitting component 220 receives a transformation environment representation 204 and (one or more) virtual objects 208. In one embodiment, the (one or more) virtual objects 208 include one or more static virtual objects. Each of the static virtual objects can take the form of a 3D mesh. The fitting component 220 can use machine learning techniques (e.g., a deep learning machine learning model) to evaluate the transformation environment representation 204 and fit the 3D mesh representing (one or more) static virtual objects to the available space within the transformation environment representation 204.

[0042] In some embodiments, (as part of the adaptation step) the adaptation component 220 can automatically adjust the size and / or orientation of (one or more) virtual objects 208 so that they blend into the available space within the real-world environment. For example, consider a case where (one or more) virtual objects 208 include a virtual object of "Earth (globe)". In this case, the adaptation component 220 can adjust the size of "Earth" and float it in the largest available space in the real-world environment. The largest available space is thought to vary depending on the type of real-world environment. For example, if the real-world environment is a living room, the largest available space may be the space above the coffee table in the living room. In another example, the largest available space may be the empty space in a corner of the living room.

[0043] In some embodiments, the fitting component 220 may consider the context of the virtual object 208 (e.g., semantic meaning) and the context of the available space in the real-world environment when determining how to fit one or more virtual objects 208 into the available space in the transformed environment representation 204. For example, the fitting component 220 may perform semantic segmentation to identify the "class" of each real item in the real-world environment and to identify the "class" of one or more virtual objects 208.

[0044] For example, consider a case where one or more virtual objects 208 include a virtual object called "dog," and the real-world environment includes a first available space, which is the space on the floor, and a second available space, which is the space on the table. In this particular example, the fitting component 220 can determine, based on performing semantic segmentation, that the class label of the first available space is "floor," the class label of the second available space is "table," and the class label of the virtual object is "dog." Based on these class labels, the fitting component 220 can then fit the virtual object called "dog" to the first available space (which has the class label "floor") rather than the second available space (which has the class label "table").

[0045] In some embodiments, the fitting component 220 can automatically determine the semantic meaning of various available spaces (for example, based on performing semantic segmentation using a semantic deep learning neural network). In other embodiments, the fitting component 220 can also determine the semantic meaning of various available spaces based on user input in the form of a criterion 214. For example, the criterion 214 can specify the user's preference for fitting different types of (one or more) virtual objects to different types of available spaces. Continuing with the above example, the criterion 214 can specify that (one or more) "dog" virtual objects should be fitted to the available space on the "floor" rather than other types of available spaces such as on a "table" or on a "chair".

[0046] Continuing to refer to Figure 2, the matching component 220 outputs correspondence information 206 based on its evaluation of the transformation environment representation 204 and (one or more) virtual objects 208. The correspondence information 206 may include a mapping between each of the (one or more) virtual objects 208 and the available space within the transformation environment representation 204.

[0047] The insertion tool 230 receives correspondence information 206 and generates rendering information 212. Rendering information 212 is information for rendering (one or more) virtual objects 208 onto (one or more) display devices 160 of the augmented reality device 110 so that the virtual objects 208 appear in the available space of the target (real-world environment) indicated in the correspondence information 206. In some embodiments, the insertion tool 230 can obtain rendering information 212 based at least in part on the transformation environment representation 204 and / or (one or more) virtual objects 208. Rendering information 212 may include, for example, the position of each (one or more) virtual object 208 in screen space (e.g., a two-dimensional (2D) position on (one or more) display devices 160).

[0048] Figure 3 is a block diagram illustrating an exemplary workflow 300 for automatically placing and manipulating (one or more) virtual objects as part of an AR / MR experience, according to one embodiment. The workflow 300 can be implemented by the augmentation component 134, one or more components of the augmented reality device 110, or any combination thereof. In one embodiment, the workflow 300 can be used to automatically place and manipulate (one or more) dynamic virtual objects as part of an AR / MR experience.

[0049] As described in the explanation of Figure 2 above, the extension component 134 can generate an environment representation 202 that represents the real-world environment in which the augmented reality device 110 is located. The environment representation 202 can be input to a route planning tool 138 equipped with a route prediction tool 370. The route prediction tool 370 is generally configured to determine a predefined set of routes 380 that a user may take through the real-world environment. Each route 380 may include a 3D volume representing the amount of space the user occupies as they travel through the real-world environment. The route prediction tool 370 can implement various route planning algorithms, such as heuristic search methods and intelligent algorithms (e.g., particle swarm algorithms, genetic algorithms, etc.).

[0050] In some embodiments, a set of paths 380 may include various paths through a real-world environment for a single user having a set of user attributes 360. For example, consider the case where the user is an adult. In this case, the path prediction tool 370 can generate multiple paths 380 based on the user's height specified in the user attributes 360. That is, each path 380 may include a 3D volume that is partly based on the user's height (e.g., vertical distance from the floor). In other embodiments, a set of paths 380 may include various paths through a real-world environment for multiple users having different sets of user attributes 360. For example, a first set of paths 380 may include various paths (e.g., multiple 3D volumes based on a specific height range) for an "adult" user with a specific height range. Similarly, a second set of paths 380 may include various paths (e.g., multiple 3D volumes based on other specific height ranges) for a "child" user with a different height range.

[0051] The path prediction tool 370 can determine a set of paths 380 and provide them to the transformation tool 310 of the visualization tool 136. The transformation tool 310 can perform the same operations as the transformation tool 210 described in the explanation of Figure 2. For example, the transformation tool 310 can calculate the inverse representation of the environment representation 202 to generate multiple transformed environment representations 304 corresponding to a set of paths 380. Each transformed environment representation 304 can be a 3D point cloud or 3D mesh representing the available space along different paths 380 through the real-world environment.

[0052] The fitting component 320 can perform operations similar to those of the fitting component 220. For example, the fitting component 320 can receive a transformation environment representation 304 (corresponding to a path 380) along with (one or more) virtual objects 208, and by evaluating this information using one or more machine learning techniques (e.g., a (semantic) deep learning neural network model), it can derive multiple correspondence information 306. Each correspondence information 306 may include a fit between (one or more) virtual objects 208 and the available space in the transformation environment representation 304 corresponding to the path 380, for each resulting path 380.

[0053] In some embodiments, like the conformance component 220, the conformance component 320 can be configured to automatically consider the context (e.g., semantic meaning) of (one or more) virtual objects 208 in accordance with the context of the available space of the transformation environment representation 304 when performing the conformance process. The context is identified based on the performance of semantic segmentation. In other embodiments, the conformance component 320 can also identify the context from the criterion 214.

[0054] The insertion tool 330 can perform operations similar to those of the insertion tool 230. For example, the insertion tool 330 can receive multiple correspondence information 306 and generate rendering information 212 for rendering (one or more) virtual objects 208 onto (one or more) display devices 160 of the augmented reality device 110. The rendering of the virtual objects 208 can be performed such that (one or more) virtual objects 208 appear in the available space of the target to be adapted along the path indicated in the correspondence information 306 corresponding to the obtained path. In some embodiments, the insertion tool 330 can also obtain the rendering information 212 based on at least some of multiple transformation environment representations 304 and / or (one or more) virtual objects 208.

[0055] In some embodiments, the insertion tool 330 can output various sets of rendering information 212 based on the user's current location in the real-world environment. For example, the insertion tool 330 can receive sensor data 340 indicating the user's current location. Based on this user location, the insertion tool 330 can determine from a plurality of correspondence information 306 whether or not to update the rendering of (one or more) virtual objects 208, and include the updated set of rendering information 212. In this way, the extension component 134 can automatically place and manipulate (one or more) virtual objects based on the user's current location in the real-world environment as part of the AR / MR experience.

[0056] Figure 4 is a flowchart of a method 400 for automatically arranging and manipulating (one or more) virtual objects according to one embodiment. Method 400 can be performed by one or more components of an augmented reality device (e.g., augmented reality device 110). In one embodiment, method 400 is performed by an extension component of the augmented reality device (e.g., extension component 134).

[0057] Method 400 begins in block 402. In block 402, the extension component begins scanning the real-world environment using one or more cameras (e.g., camera 150) and / or one or more sensors (e.g., sensor 170). In block 404, the extension component generates a first representation (e.g., environment representation 202) of the real-world environment based on the scan. The first representation of the real-world environment may include a 3D point cloud or 3D mesh representing the real-world environment (e.g., 3D representations of boundaries (floors, walls, etc.) and real objects (tables, chairs, trees, etc.) in the real-world environment).

[0058] In block 406, the extension component generates a second representation of the real-world environment (e.g., a transformed environment representation 204) based on a first representation of the real-world environment. The second representation of the real-world environment may include a 3D point cloud or 3D mesh representing available space within the real-world environment (e.g., empty space) (e.g., a 3D representation of available space within a boundary (e.g., a floor) or a 3D representation of available space above or below a specific real object (e.g., the surface of a table) within the real-world environment). In one embodiment, the second representation of the real-world environment can be generated by computing the inverse representation of the first representation of the real-world environment.

[0059] In block 408, the extension component obtains a directive for (one or more) virtual objects (e.g., (one or more) virtual objects 208) that can be placed in a real-world environment (e.g., as part of an AR / MR experience). The (one or more) virtual objects may include at least one static or dynamic virtual object that can take the form of a 3D mesh.

[0060] In block 410, the extension component adapts (one or more) virtual objects to (one or more) available spaces within the second representation of the real-world environment, based on an evaluation of the second representation of (one or more) virtual objects and the real-world environment performed using one or more machine learning techniques. In one embodiment, the adaptation step in block 410 may include a step of determining and / or adjusting the size and / or orientation of (one or more) virtual objects to match the available spaces within the real-world environment to be adapted. The extension component can generate a set of information (e.g., correspondence information 206) containing information indicating such adaptation. In one embodiment, the processing in block 410 may be performed by the adaptation component 220, which is described in detail in the above-mentioned Figure 2 reference.

[0061] In block 412, the augmentation component generates information (e.g., rendering information 212) for rendering (one or more) virtual objects onto the display devices of the augmented reality device (e.g., (one or more) display devices 160) based on the above-described adaptations. For example, the rendering information may include at least one of the following: (i) where in screen space to render (one or more) virtual objects; (ii) in what orientation to render (one or more) virtual objects; or (iii) at what size to render (one or more) virtual objects. Such rendering information allows (one or more) virtual objects to appear on the display device as if they were placed in the available space within the real-world environment, with the appropriate size, position, and / or orientation according to the adaptations made in block 410.

[0062] In block 414, the extension component renders (one or more) virtual objects onto the display device of the augmented reality device according to rendering information. In embodiments where the display device is part of an external unit (e.g., external unit 190), the extension component (in block 414) can render (one or more) virtual objects onto the display device of the external unit.

[0063] Figure 5 is a flowchart of a method 500 for automatically arranging and manipulating (one or more) virtual objects according to one embodiment. Method 500 can be performed by one or more components of an augmented reality device (e.g., augmented reality device 110). In one embodiment, method 500 is performed by an extension component of the augmented reality device (e.g., extension component 134).

[0064] Method 500 begins in block 502. In block 502, the extension component begins scanning the real-world environment using one or more cameras (e.g., camera 150) and / or one or more sensors (e.g., sensor 170). In block 504, the extension component generates a first representation (e.g., environment representation 202) of the real-world environment based on the scan. The first representation of the real-world environment may include a 3D point cloud or 3D mesh representing the real-world environment (e.g., 3D representations of boundaries (floors, walls, etc.) and real objects (tables, chairs, etc.) in the real-world environment).

[0065] In block 506, the extension component generates a set of user paths (e.g., path 380) through the real-world environment, based in part on a first representation of the real-world environment. In some embodiments, the extension component can also generate a set of user paths based on one or more user attributes (e.g., user attribute 360) in addition to the first representation. In one embodiment, the processing in block 506 can be performed by the routing tool 138, which is described in detail in the above-mentioned Figure 3.

[0066] In block 508, the extension component generates a number of second representations of the real-world environment (e.g., transformed environment representations 304) based on a first representation of the real-world environment and a set of user paths. Each of the second representations of the real-world environment may include a 3D point cloud or 3D mesh representing the available space along different paths within the real-world environment. In one embodiment, each of the second representations of the real-world environment can be generated by calculating the inverse representation of each path in the environment representation 202.

[0067] In block 510, the extension component obtains a directive for (one or more) virtual objects (e.g., (one or more) virtual objects 208) that can be placed in a real-world environment (e.g., as part of an AR / MR experience). The (one or more) virtual objects may include at least one dynamic (e.g., changing) virtual object that can take the form of a 3D mesh.

[0068] In block 512, the extension component generates a set of information (e.g., correspondence information 306) for each of the second representations of a real-world environment that represent the real-world environment, for fitting one or more virtual objects into the available space within the second representation of the real-world environment. The generation of such information is based on an evaluation of the one or more virtual objects and the second representation of the real-world environment, performed using one or more machine learning techniques. In one embodiment, the fitting step in block 512 may include a step of determining and / or adjusting the size and / or orientation of one or more virtual objects to match the available space within each user path in the real-world environment being fitted. In one embodiment, the processing in block 512 can be performed by the fitting component 320, which is described in detail in the above-mentioned Figure 3 reference.

[0069] In block 514, the augmentation component generates information (e.g., rendering information 212) for rendering (one or more) virtual objects onto the display devices of the augmentation device (e.g., (one or more) display devices 160) based on the aforementioned alignment information and the current position of the augmented reality device. For example, the rendering information may include at least one of the following: (i) where in screen space to render (one or more) virtual objects, (ii) in what orientation to render (one or more) virtual objects, or (iii) at what size to render (one or more) virtual objects. Such rendering information allows (one or more) virtual objects to appear on the display devices in a manner that makes them appear to be placed in the available space associated with the current user's position in the real-world environment, with appropriate size, position, and / or orientation.

[0070] In block 516, the extension component renders (one or more) virtual objects onto the display device of the augmented reality device according to the rendering information. In embodiments where the display device is part of an external unit (e.g., external unit 190), the extension component (in block 516) can render (one or more) virtual objects onto the display device of the external unit.

[0071] While this disclosure refers to various embodiments, it should be understood that this disclosure is not limited to the specific embodiments described. Rather, it is intended that the following features and elements can be arbitrarily combined, regardless of whether the embodiments to which they are associated differ, in order to implement and carry out what is taught herein. Furthermore, where elements of an embodiment are described in the form of "at least one of A and B," it should be understood that embodiments including only element A, embodiments including only element B, and embodiments including both element A and element B are intended. Furthermore, while some embodiments may provide advantages not found in other possible solutions or in the prior art, whether a particular advantage is provided by a given embodiment is not a limitation of this disclosure. Accordingly, the aspects, features, embodiments, and advantages disclosed herein are merely illustrative and should not be considered elements or limitations of the appended claims unless expressly stated in one or more claims. Similarly, any reference to "the invention" should not be interpreted as a generalization of any inventive subject matter disclosed herein, and shall not be considered an element or limitation of the appended claims unless explicitly stated in one or more claims.

[0072] As those skilled in the art will understand, the embodiments described herein can be embodied as systems, methods, or computer program products. Accordingly, embodiments can take the form of entirely hardware-only embodiments, entirely software-only embodiments (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware aspects, all of which may be collectively referred to herein as “circuits,” “modules,” or “systems.” Furthermore, embodiments described herein can also take the form of computer program products, which can be embodied as one or more computer-readable media, or as computer-readable program code on such computer-readable media.

[0073] Program code, embodied on a computer-readable medium, can be transmitted using any suitable medium. Such suitable mediums include, but are not limited to, wireless, wired, fiber optic cables, RF, or any suitable combination thereof.

[0074] Computer program code for executing the processes according to the embodiments of this disclosure can be written in any combination of one or more programming languages. Such programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" programming language, or similar programming languages. The program code can be executed as a standalone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (such as a local area network (LAN) or wide area network (WAN)), and connections can also be made to external computers (for example, via the Internet using an Internet service provider).

[0075] This specification describes aspects of the disclosure with reference to flowcharts and block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block in a flowchart or block diagram, and combinations of blocks in a flowchart or block diagram, can be implemented by computer program instructions. By providing these computer program instructions to the processor of a programmable data processing device that forms a machine such as a general-purpose computer or a dedicated computer, such instructions can be executed via the processor of the programmable data processing device such as a computer, and can constitute a means of performing the function / operation specified in one or more blocks of a flowchart or block diagram.

[0076] These computer program instructions can be stored in a computer-readable medium capable of instructing a computer or other programmable data processing device to function in a specific manner, thereby generating a manufactured article containing instructions that perform functions / operations specified in one or more blocks of a flowchart or block diagram.

[0077] Furthermore, a computer implementation process can be generated by loading computer program instructions into a computer or other programmable data processing device, causing a series of processing steps to be executed on the computer or other programmable data processing device, thereby providing a process for performing functions / operations specified in one or more blocks of a flowchart or block diagram, through instructions executed on the computer or other programmable data processing device.

[0078] The flowcharts and block diagrams shown in the drawings illustrate architectures, functionalities, and operations that may constitute embodiments of systems, methods, and computer program products according to various embodiments of the present disclosure. In this sense, each block in a flowchart or block diagram can be considered to represent a module, segment, or portion of code containing one or more executable instructions for implementing the (one or more) logical functions specified therein. It should also be noted that in some alternative embodiments, the functions described in the blocks may be executed in an order different from that shown in the drawings. For example, two blocks shown as consecutive blocks may actually be executed substantially simultaneously, or these blocks may be executed in reverse or different orders depending on the related functions. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented by a dedicated hardware-based system that performs the functions or operations specified therein, or by a combination of dedicated hardware and computer instructions.

[0079] While the above description is directed towards embodiments of the present disclosure, it is possible to devise other embodiments and further embodiments of the present disclosure without departing from the basic scope of the present disclosure, which are defined by the following claims.

[0080] The present invention can be implemented in the following embodiments.

[0081] Embodiment 1 A method implemented by a computer, The steps include obtaining a point that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of a computing device, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a location on the display of the computing device, at a location associated with the at least one available space within the real-world environment, A method that includes this.

[0082] Embodiment 2 A computer-implemented method according to claim 1, wherein the step of determining the compatibility between the at least one virtual object and the at least one available space in the real-world environment includes the step of determining the size of the at least one virtual object such that it blends into the at least one available space.

[0083] Embodiment 3 The computer-based method according to claim 2, wherein the step of rendering the at least one virtual object includes the step of rendering the at least one virtual object on the display of the computing device at the determined size.

[0084] Embodiment 4 The process further includes determining multiple paths for the computing device through the aforementioned real-world environment, The computer-implemented method according to claim 1, wherein the at least one second representation indicates the at least one available space along a first path among the plurality of paths.

[0085] Embodiment 5 The computer-implemented method according to claim 4, wherein the plurality of paths include a first set of paths passing through the real-world environment based on a first height range and a second set of paths passing through the real-world environment based on a second height range.

[0086] Embodiment 6 The process further includes determining multiple paths for the computing device through the aforementioned real-world environment, The computer-implemented method according to claim 1, wherein the at least one virtual object is adapted to the at least one available space along a first path among the plurality of paths.

[0087] Embodiment 7 The step of rendering the at least one virtual object includes receiving sensor data indicating the location of the computing device along the first path, The computer-implemented method according to claim 6, wherein the position on the display corresponds to the position of the computing device in the at least one available space along the first path.

[0088] Embodiment 8 The method implemented on a computer according to claim 1, wherein the at least one virtual object is a static virtual object.

[0089] Embodiment 9 The method implemented on a computer according to claim 1, wherein the at least one virtual object is a dynamic virtual object.

[0090] Embodiment 10 The display and One or more sensors, One or more processors, Memory for storing instructions, A computing device equipped with, When the instruction is executed on one or more of the aforementioned processors, The steps include obtaining a point that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a position on the display that is associated with the at least one available space within the real-world environment, A computing device on which processing including [specific processes] is performed.

[0091] Embodiment 11 The computing device according to claim 10, wherein the step of determining the fit between the at least one virtual object and the at least one available space in the real-world environment includes the step of determining the size of the at least one virtual object such that it blends into the at least one available space.

[0092] Embodiment 12 The computing device according to claim 11, wherein the step of rendering the at least one virtual object includes the step of rendering the at least one virtual object on the display at the determined size.

[0093] Embodiment 13 The process further includes the step of determining a plurality of paths for the computing device through the real-world environment, The computing device according to claim 10, wherein the at least one second representation further indicates the at least one available space along the first path among the plurality of paths.

[0094] Embodiment 14 The computing device according to claim 13, wherein the plurality of paths include a first set of paths passing through the real-world environment based on a first height range and a second set of paths passing through the real-world environment based on a second height range.

[0095] Embodiment 15 The process further includes the step of determining a plurality of paths for the computing device through the real-world environment, The computing device according to claim 10, wherein the at least one virtual object is adapted to the at least one available space along a first path among the plurality of paths.

[0096] Embodiment 16 The step of rendering the at least one virtual object includes receiving sensor data indicating the location of the computing device along the first path, The computing device according to claim 15, wherein the position on the display corresponds to the position of the computing device in the at least one available space along the first path.

[0097] Embodiment 17 A method implemented by a computer, The steps include obtaining a point that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of a computing device, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a location on the display of the computing device, at a location associated with the at least one available space within the real-world environment, Includes, A method for identifying the location of a user within the real-world environment and automatically rendering a virtual object in the available space based on the user's location, user attributes, and correspondence information.

[0098] Embodiment 18 The display and One or more sensors, One or more processors, Memory for storing instructions, A computing device equipped with, When the instruction is executed on one or more of the aforementioned processors, The steps include obtaining a point that points to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a position on the display that is associated with the at least one available space within the real-world environment, The process including this is executed, A computing device that identifies the location of a user within the aforementioned real-world environment and automatically renders virtual objects into the available space based on the user's location, user attributes, and correspondence information. [Explanation of Symbols]

[0099] 110 Augmented Reality Devices 120 processors 130 memory 132 Applications 134 Extension Components 136 Visualization Tools 138 Route Planning Tools 140 Storage device 150 Cameras 160 display devices 170 sensors 180 network interfaces 190 External Units 202 Environmental expression 204, 304 Conversion Environment Expression Correspondence information for 206 and 306 208 virtual objects 210, 310 conversion tool 212 Rendering Information 214 Criteria 220, 320 compatible components 230, 330 Insertion Tool 340 Sensor Data 360 User Attributes 370 Route Prediction Tools 380 routes

Claims

1. A method implemented by a computer, The steps include obtaining a reference to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors of a computing device, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a location on the display of the computing device, where the location is associated with the at least one available space within the real-world environment, Includes, The system identifies the user's location within the real-world environment and automatically renders a virtual object in the available space based on the user's location, user attributes, and correspondence information. A method in which the at least one second representation of a real-world environment is a 3D point cloud or 3D mesh representing empty space within the real-world environment.

2. A computer-implemented method according to claim 1, wherein the step of determining the compatibility between the at least one virtual object and the at least one available space in the real-world environment includes the step of determining the size of the at least one virtual object such that it blends into the at least one available space.

3. The computer-implemented method according to claim 2, wherein the step of rendering the at least one virtual object includes the step of rendering the at least one virtual object on the display of the computing device at the determined size.

4. The process further includes determining multiple paths for the computing device through the aforementioned real-world environment, The computer-implemented method according to claim 1, wherein the at least one second representation indicates the at least one available space along a first path among the plurality of paths.

5. The computer-implemented method according to claim 4, wherein the plurality of paths include a first set of paths passing through the real-world environment based on a first height range and a second set of paths passing through the real-world environment based on a second height range.

6. The process further includes determining multiple paths for the computing device through the aforementioned real-world environment, The computer-implemented method according to claim 1, wherein the at least one virtual object is adapted to the at least one available space along a first path among the plurality of paths.

7. The step of rendering the at least one virtual object includes receiving sensor data indicating the location of the computing device along the first path, The computer-implemented method according to claim 6, wherein the position on the display corresponds to the position of the computing device in the at least one available space along the first path.

8. The method implemented on a computer according to claim 1, wherein the at least one virtual object is a static virtual object.

9. The method implemented on a computer according to claim 1, wherein the at least one virtual object is a dynamic virtual object.

10. The display and One or more sensors, One or more processors, Memory for storing instructions, A computing device equipped with, When the instruction is executed on one or more of the aforementioned processors, The steps include obtaining a reference to at least one virtual object that can be placed in a real-world environment as part of an augmented reality (AR) simulation or a mixed reality (MR) simulation, A step of generating a first representation of the real-world environment based on a scan of the real-world environment by one or more sensors, The steps of generating, from the first representation representing the real-world environment, at least one second representation representing the real-world environment, which indicates at least one available space within the real-world environment; A step of using one or more machine learning models to evaluate the at least one virtual object and the at least one second representation representing the real-world environment, and determining the fit between the at least one virtual object and the at least one available space within the real-world environment, Based on the above adaptation, the steps include rendering the at least one virtual object at a position on the display that is associated with the at least one available space within the real-world environment, The process including this is executed, The system identifies the user's location within the real-world environment and automatically renders a virtual object in the available space based on the user's location, user attributes, and correspondence information. A computing device in which the at least one second representation of a real-world environment is a 3D point cloud or 3D mesh representing empty space within the real-world environment.

11. The computing device according to claim 10, wherein the step of determining the compatibility between the at least one virtual object and the at least one available space in the real-world environment includes the step of determining the size of the at least one virtual object such that it blends into the at least one available space.

12. The computing device according to claim 11, wherein the step of rendering the at least one virtual object includes the step of rendering the at least one virtual object on the display at the determined size.

13. The process further includes the step of determining a plurality of paths for the computing device through the real-world environment, The computing device according to claim 10, wherein the at least one second representation further indicates the at least one available space along the first path among the plurality of paths.

14. The computing device according to claim 13, wherein the plurality of paths include a first set of paths passing through the real-world environment based on a first height range and a second set of paths passing through the real-world environment based on a second height range.

15. The process further includes the step of determining a plurality of paths for the computing device through the real-world environment, The computing device according to claim 10, wherein the at least one virtual object is adapted to the at least one available space along a first path among the plurality of paths.

16. The step of rendering the at least one virtual object includes receiving sensor data indicating the location of the computing device along the first path, The computing device according to claim 15, wherein the position on the display corresponds to the position of the computing device in the at least one available space along the first path.

Images