Method and system for composing and executing scenarios

Abstract

This disclosure relates to methods and systems for constructing and executing scenarios. In one embodiment, a method of displaying content is performed at a device including a display, one or more processors, and nontransitory memory. The method includes scanning a first physical environment to detect a first physical object and a second physical object in the first physical environment, wherein the first physical object conforms to at least one first object criterion, and the second physical object conforms to at least one second object criterion. The method includes displaying, in association with the first physical environment, a virtual object moving along a first path from the first physical object to the second physical object. The method includes scanning a second physical environment to detect a third physical object in the second physical environment and a fourth physical object in the second physical environment, wherein the third physical object conforms to at least one first object criterion, and the fourth physical object conforms to at least one second object criterion. The method includes displaying, in association with the second physical environment, a virtual object moving along a second path from the third physical object to the fourth physical object, wherein the second path is different from the first path.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 246,631, filed September 21, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to a graphical user interface for constructing and executing scenes in an extended reality (XR) environment. Background Technology

[0004] In various specific implementations, scenarios include virtual content presented in an XR environment based on a physical environment. It may be desirable to present scenarios in various different XR environments based on various different physical environments. Attached Figure Description

[0005] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0006] Figure 1 This describes the physical environment of the electronic devices that are being investigated.

[0007] Figures 2A to 2W This describes the graphical user interface (GUI) used to construct the scene. Figure 1 Electronic devices.

[0008] Figures 3A to 3I This illustrates how to present a scene in a first XR environment based on a first physical environment. Figure 1 Electronic devices.

[0009] Figures 4A to 4F This illustrates how to present a scene in a second XR environment based on a second physical environment. Figure 1 Electronic devices.

[0010] Figure 5 It is a flowchart representation of a method for displaying content based on some specific implementations.

[0011] Figure 6 It is a flowchart representation of a method for associating tags with objects based on some specific implementations.

[0012] Figure 7 It is a flowchart representation of a method based on some specific implementation scenarios.

[0013] Figure 8 It is a flowchart representation of the methods that constitute a specific implementation scenario.

[0014] Figure 9It is a block diagram based on some specific implementations of electronic devices.

[0015] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Summary of the Invention

[0016] The various embodiments disclosed herein include devices, systems, and methods for displaying content. In various embodiments, the method is performed at a device including a display, one or more processors, and nontransitory memory. The method includes scanning a first physical environment to detect a first physical object and a second physical object within the first physical environment, wherein the first physical object conforms to at least one first object criterion, and the second physical object conforms to at least one second object criterion. The method includes displaying, in association with the first physical environment, a virtual object moving along a first path from the first physical object to the second physical object. The method includes scanning a second physical environment to detect a third physical object and a fourth physical object within the second physical environment, wherein the third physical object conforms to the at least one first object criterion, and the fourth physical object conforms to the at least one second object criterion. The method includes displaying, in association with the second physical environment, the virtual object moving along a second path from the third physical object to the fourth physical object, wherein the second path is different from the first path.

[0017] The various embodiments disclosed herein include devices, systems, and methods for associating tags with objects. In various embodiments, the method is performed at a device including one or more processors and non-transitory memory. The method includes obtaining tags associated with multiple criteria. The method includes detecting physical objects in a physical environment. The method includes determining that the physical object satisfies the functionality of the multiple criteria. The method includes generating a feature vector of the physical object, the feature vector including an object identifier and a tag of the physical object, in response to determining that the physical object satisfies the functionality of the multiple criteria.

[0018] The various embodiments disclosed herein include devices, systems, and methods for executing a scenario. In various embodiments, the method is executed at a device including a display, one or more processors, and non-transitory memory. The method includes obtaining a scenario associated with one or more execution constraints. The method includes scanning a physical environment to determine whether the physical environment conforms to the one or more execution constraints, wherein the one or more execution constraints include execution constraints that are conformed when the physical environment includes physical objects having specific attributes. The method includes executing the scenario in response to determining that the physical environment conforms to the one or more execution constraints, wherein executing the scenario includes displaying content associated with the physical objects.

[0019] The various embodiments disclosed herein include devices, systems, and methods for constituting a scene. In various embodiments, the method is performed on a device including a display, one or more processors, and nontransitory memory. The method includes displaying a representation of the scene. The method includes receiving user input that associates a plurality of physical assets with the scene. The method includes receiving user input that associates a plurality of virtual assets with the scene. The method includes receiving user input that associates a first attribute with a first virtual asset among the plurality of virtual assets, wherein the first attribute is a spatial relationship between the first virtual asset and a first physical asset among the plurality of physical assets. The method includes displaying, in the representation of the scene, a representation of the first virtual asset that is in the spatial relationship with a representation of the first physical asset.

[0020] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors. The one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Detailed Implementation

[0021] A physical environment refers to a physical location that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner consistent with at least one physical law. As another example, an XR system can detect the movement of electronic devices (e.g., mobile phones, tablets, laptops, headsets, etc.) presenting the XR environment, and in response, adjust the graphical content and sound field presented to the person by the electronic device in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), XR systems can adapt the characteristics of graphical content in an XR environment in response to representations of physical motion (e.g., voice commands).

[0022] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0023] In various specific implementations, scenes including virtual content are presented in various XR environments based on different physical environments with different physical characteristics (such as different sets of physical objects existing in the physical environment). The graphical user interface (GUI) for constructing scenes for presentation in various XR environments is described below. Methods and systems for presenting scenes in various XR environments are also described below.

[0024] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0025] Figure 1The physical environment 101, which includes an electronic device 110 for investigating the physical environment 101, is described. The physical environment 101 includes a picture 102 hanging on a wall 103, a table 105 on a floor 106, and a ball 104 on the table 105.

[0026] Electronic device 110 displays an image of an XR environment 121 on a display, including a representation of a physical environment 111 and a representation of virtual objects 119. In various specific embodiments, the representation of the physical environment 111 is generated based on images of the physical environment 101 captured by one or more cameras of electronic device 110, said cameras having a field of view facing the physical environment 101. Suitable cameras include scene cameras, event cameras, depth cameras, etc. Thus, the representation of the physical environment 111 includes a representation of a picture 112 on a representation of a wall 113, a representation of a table 115 on a representation of a floor 116, and a representation of a ball 114 on a representation of a table 115.

[0027] In addition to the representation of real objects in the physical environment 101, the images in the XR environment 121 include a representation of virtual objects 119. The visual appearance of the virtual objects 119 is defined by software on the electronic device 110. The electronic device 110 presents the virtual objects 119 as resting on the top surface of the representation on the table 115 by taking into account the position and orientation of the device 110 relative to the table 105.

[0028] Figure 2A An electronic device 110 is described, which displays a graphical user interface (GUI) 201 for constructing a scene. Specifically, the GUI 201 includes a representation of the scene. In various specific implementations, applications executed by the electronic device 110 or different electronic devices are used to render the scene in an XR environment, such as a virtual environment or a representation associated with a physical environment.

[0029] The GUI 201 includes a toolbar area 211, an asset area 212, and a view area 213. The toolbar area 211 includes an asset addition indicator 221 for adding assets to the scene, an attribute indicator 222 for manipulating the properties of the selected asset, and a preview indicator 229 for previewing the scene.

[0030] Asset area 212 includes a list of assets associated with a scene. Scene-associated assets include virtual assets, physical assets, and motion assets. In various implementations, asset area 212 includes an asset type selection enablement 231, which is used to select which type of asset to list in asset area 212, such as a list of virtual assets, a list of physical assets, or a list of motion assets.

[0031] View area 213 includes a representation of the scene. In various implementations, the scene representation includes a representation of virtual assets associated with the scene. In various implementations, the scene representation includes a representation of physical assets associated with the scene. In various implementations, the scene representation includes a representation of motion assets associated with the scene.

[0032] In various implementations, virtual assets associated with a scene include descriptions of virtual content displayed in relation to the physical environment when the scene is executed. In various implementations, virtual assets include descriptions of one or more virtual objects. In various implementations, virtual assets include descriptions of virtual goal implementer objects. In various implementations, a virtual goal implementer object receives goals and determines actions to achieve those goals, where each action is associated with an animation or animation heuristic of the virtual goal implementer object, causing the virtual goal implementer object to be displayed as performing an action. For example, in various implementations, the goal of a virtual dog goal implementer object might be to pick up a virtual bone object from the physical floor. To achieve the goal, the virtual dog goal implementer object determines a series of actions: jumping from a physical sofa onto the physical floor (associated with a jump animation), walking along the physical floor to the location of the virtual bone object (associated with a walking animation), and picking up the virtual bone object (associated with a pick-up animation).

[0033] In various implementations, physical assets associated with a scene include descriptions of physical objects that may or may not exist in the physical environment. Specifically, in various implementations, physical assets include descriptions of at least one object standard that can be conformed to by physical objects in the physical environment. For example, in various implementations, physical assets include descriptions of horizontal planes with specific heights and widths. In various implementations, physical assets correspond to the top of a physical table in a first physical environment and the top of a physical desk in a second physical environment.

[0034] In various implementations, scene-related action assets include descriptions of actions performed in response to triggers. These actions include moving virtual objects, playing audio, and changing lighting conditions.

[0035] exist Figure 2A In the asset type selection indicator 231, the virtual assets are listed in asset area 212. Figure 2A In this scene, no assets are yet associated with it. Therefore, asset area 212 and view area 213 are empty.

[0036] Figure 2AThis describes user input 299A for the asset addition capability indicator 221. In various embodiments, user input 299A is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299A is input by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0037] Figure 2B This describes the response to the detection of user input 299A for adding an enablement representation 221 to the asset and additional user input for selecting a virtual professor target implementer object. Figure 2A The GUI 201. In various specific implementations, additional user input includes selecting a virtual professor target implementer object from the virtual content library. In response to detecting user input 299A for adding an enablement representation 221 to the asset and additional user input for selecting a virtual professor target implementer object, the scene is associated with the virtual professor target implementer object as a virtual asset. Therefore, a text representation of the virtual professor target implementer object 232A is displayed in the asset area 212, and a graphical representation of the virtual professor target implementer object 242A is displayed in the view area 213.

[0038] The text representation of the virtual professor target implementer object 232A is displayed in association with an icon indicating the object type of the virtual professor target implementer object, such as an icon indicating whether the virtual professor target implementer object is a target implementer object or an active object.

[0039] Figure 2B This describes user input 299B for the asset addition capability indicator 221. In various embodiments, user input 299B is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299B is input by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0040] Figure 2C This describes the response to the detection of user input 299B for adding an enablement display 221 to the asset and additional user input for selecting a virtual blackboard object. Figure 2C The GUI 201. In various specific implementations, additional user input includes selecting a virtual blackboard object from the virtual content library. In response to detecting user input 299B for adding an enablement representation 221 to the asset and additional user input for selecting a blackboard object, the scene is associated with the blackboard object as a virtual asset. Therefore, the text representation of the virtual blackboard object 232B is displayed in the asset area 212, and the graphical representation of the virtual blackboard object 242B is displayed in the view area 213.

[0041] The text representation of the virtual blackboard object 232B is displayed in association with an icon indicating the object type of the virtual blackboard object, such as an icon indicating that the virtual blackboard object is a passive object.

[0042] Figure 2C This describes user input 299C for the asset addition capability indicator 221. In various embodiments, user input 299C is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299C is input by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0043] Figure 2D This describes the response to the detection of user input 299C for adding an indicator 221 to an asset, additional user input for selecting a virtual paper object, and additional user input for associating a virtual paper stack object and a virtual mug object with the scene. Figure 2C The GUI 201. In various implementations, additional user input for selecting virtual paper objects includes selecting a blackboard object from the virtual content library. In various implementations, additional user input for associating virtual paper stack objects and virtual mug objects with the scene includes selecting asset addition display 221 and selecting virtual paper stack objects and virtual mug objects from the virtual content library.

[0044] In response to the detection of user input 299C and additional user input, the scene is associated with virtual paper objects, virtual paper stack objects, and virtual mug objects as virtual assets. Therefore, text representations of virtual paper object 232C, virtual paper stack object 232D, and virtual mug object 232E are displayed in asset area 212. Furthermore, graphic representations of virtual paper object 242C, virtual paper stack object 242D, and virtual mug object 242E are displayed in view area 213.

[0045] Figure 2D This describes user input 299D for selecting the asset type display 231, which selects a list of physical assets. In various embodiments, user input 299D is entered by the user tapping their finger or using a stylus on a touch-sensitive display at the location of the asset type selection display 231. In various embodiments, user input 299D is entered by the user clicking a mouse button while the cursor is displayed at the location of the asset type selection display 231.

[0046] Figure 2E This describes the response to the detection of user input 299D for asset type selection indication 231. Figure 2D The GUI. (and) Figure 2DIn contrast, Asset Zone 212 displays a list of physical assets associated with the scene, rather than a list of virtual assets associated with the scene. Figure 2E Currently, no physical assets are associated with the scene, and asset area 212 is blank.

[0047] Figure 2E This describes user input 299E for the asset addition capability indicator 221. In various embodiments, user input 299E is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299E is input by the user clicking a mouse button when the cursor is displayed at the location of the asset addition capability indicator 221.

[0048] Figure 2F This describes the response to the detection of user input 299E for adding an indicator 221 to the asset and additional user input for selecting the physical floor. Figure 2E The GUI 201. In various implementations, additional user input includes selecting a physical floor from a library of physical properties that can exist in various physical environments. In various implementations, physical properties include physical surfaces, such as physical planes. In various implementations, physical properties include physical horizontal planes or physical vertical planes. In various implementations, physical properties include physical floors, physical walls, or physical ceilings. In various implementations, physical properties include physical objects, such as chairs, trash cans, baseballs, etc. In various implementations, physical properties include environmental properties, such as temperature, humidity, ambient lighting conditions, location, or time of day.

[0049] In response to the detection of user input 299E for adding an enablement representation 221 to the asset and additional user input for selecting a physical floor, the scene is associated with the physical floor as a physical asset. Therefore, a text representation of the physical floor 233A is displayed in the asset area 212, and a graphical representation of the physical floor 243A is displayed in the view area 213.

[0050] The text representation of physical floor 233A is displayed in association with an icon indicating the type of object of the physical floor, such as an icon indicating that the physical floor is flat.

[0051] Figure 2F This describes user input 299F for the asset addition capability indicator 221. In various embodiments, user input 299F is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299F is input by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0052] Figure 2G This describes the response to the detection of user input 299F for adding an indicator 221 to the asset and additional user input for selecting the physical vertical plane. Figure 2F The GUI 201. In response to the detection of user input 299F for adding an enable representation 221 to the asset and additional user input for selecting a physical vertical plane, the scene is associated with the physical vertical plane as a physical asset. Therefore, the text representation of the physical vertical plane 233B is displayed in the asset area 212, and the graphical representation of the physical vertical plane 243B is displayed in the view area 213.

[0053] Figure 2G This describes user input 299G for the asset addition capability indicator 221. In various embodiments, user input 299G is entered by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299G is entered by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0054] Figure 2H This describes the response to the detection of user input 299G for adding an indicator 221 to the asset and additional user input for selecting a first physical horizontal plane. Figure 2G The GUI 201. In response to the detection of user input 299G for adding prompts 221 to the asset and additional user input for selecting a first physical horizontal plane, the scene is associated with the first physical horizontal plane as a physical asset. Therefore, the text representation of the first physical horizontal plane 233C is displayed in the asset area 212, and the graphical representation of the first physical horizontal plane 243C is displayed in the view area 213.

[0055] Figure 2H This describes user input 299H for the asset addition capability indicator 221. In various embodiments, user input 299H is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299H is input by the user clicking a mouse button when the cursor is displayed at the location of the asset addition capability indicator 221.

[0056] Figure 2I This describes the response to the detection of user input 299H for adding an indicator 221 to the asset and additional user input for selecting a second physical horizontal plane. Figure 2HThe GUI 201. In response to the detection of user input 299H for adding prompts 221 to the asset and additional user input for selecting a second physical horizontal plane, the scene is associated with the second physical horizontal plane as a physical asset. Therefore, the text representation of the second physical horizontal plane 233D is displayed in the asset area 212, and the graphical representation of the second physical horizontal plane 243D is displayed in the view area 213.

[0057] Figure 2I This describes user input 299I for the asset addition capability indicator 221. In various embodiments, user input 299I is input by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299I is input by the user clicking a mouse button when the cursor is displayed at the location of the asset addition capability indicator 221.

[0058] Figure 2J This describes the response to the detection of user input 299I for adding an indicator 221 to the asset and additional user input for selecting a physical trash can. Figure 2J The GUI 201. In response to the detection of user input 299I for adding an enablement representation 221 to an asset and additional user input for selecting a physical trash can, the scene is associated with the physical trash can as a physical asset. Therefore, the text representation of the physical trash can 233E is displayed in the asset area 212, and the graphical representation of the physical trash can 243E is displayed in the view area 213.

[0059] The text representation of the physical trash can 233E is displayed in association with an icon indicating the object type of the physical trash can, such as an icon indicating that the physical trash can is a three-dimensional object.

[0060] Figure 2J The text representation for the second physical horizontal plane 233D is described using user input 299J. In various embodiments, user input 299J is input by the user tapping their finger or using a stylus at the location of the asset addition display 221 on a touch-sensitive display. In various embodiments, user input 299J is input by the user clicking a mouse button when the cursor is displayed at the location of the asset addition display 221.

[0061] Figure 2K This describes the response to detecting user input 299J representing text for the second physical horizontal plane 233D. Figure 2KThe GUI 201. In response to detecting user input 299J regarding the text representation of the second physical horizontal plane 233D, the text representation of the second physical horizontal plane 233D is displayed differently to indicate that the second physical horizontal plane has been selected. In various embodiments, the graphical representation of the second physical horizontal plane 243D is also (or alternatively) displayed differently to indicate that the second physical horizontal plane has been selected. For example, in various embodiments, the second physical horizontal plane is highlighted or displayed as if surrounded by a glow. In various embodiments, user input regarding the graphical representation of the second physical horizontal plane 243D selects the second physical horizontal plane.

[0062] Figure 2K This describes user input 299K for attribute display 222. In various embodiments, user input 299K is entered by the user tapping their finger or using a stylus at the location of attribute display 222 onto the touch-sensitive display. In various embodiments, user input 299K is entered by the user clicking a mouse button while the cursor is displayed at the location of attribute display 222.

[0063] Figure 2L This describes the response to detecting user input 299K for attribute display 222 and additional user input for selecting a first attribute of the second physical horizontal plane. Figure 2K The GUI 201. Specifically, additional user input for selecting the first attribute includes user input that associates the second physical horizontal plane with attributes whose height value is greater than 0.5 meters. In response to detecting user input 299K for attribute display 222 and additional user input for selecting the first attribute of the second physical horizontal plane, the second physical horizontal plane is associated with the first attribute. Therefore, the text representation of the second physical horizontal plane 233D is displayed together with the text representation of the first attribute in the asset area 212, and the graphical representation of the second physical horizontal plane 243D is displayed in the view area 213 with the first attribute.

[0064] Figure 2L This describes user input 299L for attribute display 222. In various embodiments, user input 299L is input by the user tapping their finger or using a stylus at the location of attribute display 222 onto the touch-sensitive display. In various embodiments, user input 299L is input by the user clicking a mouse button when the cursor is displayed at the location of attribute display 222.

[0065] Figure 2M This describes the response to the detection of user input 299L for the attribute display 222 and additional user input for selecting the second and third attributes of the second physical horizontal plane. Figure 2LThe GUI 201. Specifically, additional user input for selecting the second and third attributes includes user input that associates the second physical horizontal plane with attributes whose width value is greater than 1 meter and whose length value is greater than 0.5 meters. In response to detecting user input 299L for attribute display 222 and additional user input selecting the second and third attributes of the second physical horizontal plane, the second physical horizontal plane is associated with the second and third attributes. Therefore, the text representation of the second physical horizontal plane 233D is displayed in the asset area 212 along with the text representations of the second and third attributes, and the graphical representation of the second physical horizontal plane 243D is displayed in the view area 213 having the second and third attributes.

[0066] Figure 2M This describes user input 299M for the text representation of the first physical horizontal plane 233C. In various embodiments, user input 299M is entered by the user tapping their finger or stylus at the location of the text representation on the first physical horizontal plane 233C onto a touch-sensitive display. In various embodiments, user input 299M is entered by the user clicking a mouse button while the cursor is displayed at the location of the text representation on the first physical horizontal plane 233C.

[0067] Figure 2N This describes the response to detecting user input 299M representing text for the first physical horizontal plane 233C. Figure 2M The GUI 201. In response to detecting user input 299M for a text representation of the first physical horizontal plane 233C, the text representation of the first physical horizontal plane 233C is displayed differently to indicate that the first physical horizontal plane has been selected. In various embodiments, a graphical representation of the first physical horizontal plane 243C is also (or alternatively) displayed differently to indicate that the first physical horizontal plane has been selected. For example, in various embodiments, the first physical horizontal plane is highlighted or displayed as if surrounded by a glow. In various embodiments, user input for the graphical representation of the first physical horizontal plane 243C selects the first physical horizontal plane.

[0068] Figure 2N This describes user input 299N for attribute display 222. In various embodiments, user input 299N is input by the user tapping their finger or using a stylus at the location of attribute display 222 onto the touch-sensitive display. In various embodiments, user input 299N is input by the user clicking a mouse button when the cursor is displayed at the location of attribute display 222.

[0069] Figure 2O This describes the response to detecting user input 299N for attribute display 222 and additional user input for selecting a first attribute of the first physical horizontal plane. Figure 2N The GUI 201. Specifically, additional user input for selecting the first attribute includes user input associating the first physical horizontal plane with its attribute that it is a seatable surface for the virtual professor target implementer object. The first attribute indicates that the first physical horizontal plane can be sat upon by the virtual professor target implementer object. Similarly, the first attribute indicates that the virtual professor target implementer object can (including, in various embodiments, is permitted) sit upon the first physical horizontal plane. In various embodiments, the seatable surface for the professor attribute is defined by the user of GUI 201, by the creator of the virtual professor target implementer object, or by the creator of GUI 201. In various embodiments, the seatable surface for the professor attribute is defined as a function of various criteria. For example, in various embodiments, criteria include height, length, and width values ​​within a specific range. In various embodiments, criteria include associating with an object of an object type (e.g., "chair," "stool," "sofa," etc.) from a specific set of object types. In various embodiments, criteria include being specified by the user as seatable for the professor after a horizontal plane is detected. In various embodiments, the function of the various criteria is that all defined criteria must be met for a horizontal plane to be determined as seatable for the professor. In various implementations, the functionality of various standards does not need to conform to all defined standards. For example, in various implementations, if a horizontal plane (1) is associated with the object type "chair" or (2) its height, length, and width values ​​are within a specific range, then the horizontal plane is determined to be sitable for the professor, and the user designates the horizontal plane as sitable for the professor after detecting a horizontal plane with height, length, and width values ​​within the specific range. Thus, as an example, the electronic device detects the seat of a chair as a horizontal plane, detects the chair and assigns it the object type "chair," and determines that the horizontal plane is sitable for the professor. Alternatively, as another example, the electronic device detects the top of a flat rock as a horizontal plane, detects the rock and assigns it the object type "rock" (and does not assign it the object type "chair"), determines that the height, length, and width of the horizontal plane are within a specific range, requests the user to designate the horizontal plane as sitable for the professor, and determines that the horizontal plane is sitable for the professor in response to a positive response from the user.

[0070] In response to the detection of user input 299N for attribute display 222 and additional user input selecting a first attribute of the first physical horizontal plane, the first physical horizontal plane is associated with the first attribute. Therefore, a textual representation of the first physical horizontal plane 233C is displayed together with a textual representation of the first attribute in asset area 212. In various specific embodiments, a graphical representation of the first physical horizontal plane 243C is displayed in view area 213 having the first attribute; for example, the first physical horizontal plane is displayed as a chair or stool instead of a general horizontal plane.

[0071] Figure 2O This describes user input 299O for the text representation on the physical vertical plane 233B. In various embodiments, user input 299O is input by the user tapping their finger or using a stylus at the location of the text representation on the physical vertical plane 233B onto a touch-sensitive display. In various embodiments, user input 299O is input by the user clicking a mouse button when the cursor is displayed at the location of the text representation on the physical vertical plane 233B.

[0072] Figure 2P This describes the response to detecting user input 299O representing text for the physical vertical plane 233B. Figure 2O The GUI 201. In response to detecting user input 299O for the text representation of the physical vertical plane 233B, the text representation of the physical vertical plane 233B is displayed differently to indicate that the physical vertical plane has been selected. In various embodiments, the graphical representation of the physical vertical plane 243B is also (or alternatively) displayed differently to indicate that the physical vertical plane has been selected. For example, in various embodiments, the physical vertical plane is highlighted or displayed as if surrounded by a glow. In various embodiments, the user input for the graphical representation of the physical vertical plane 243B selects the physical vertical plane.

[0073] Figure 2P The description covers user input 299P for attribute display 222. In various embodiments, user input 299P is input by the user tapping their finger or stylus on the touch-sensitive display at the location of attribute display 222. In various embodiments, user input 299P is input by the user clicking a mouse button when the cursor is displayed at the location of attribute display 222.

[0074] Figure 2Q This describes the response to detecting user input 299P for attribute display 222 and additional user input for selecting a first attribute of the physical vertical plane. Figure 2PThe GUI 201. Specifically, additional user input for selecting the first attribute includes user input associating the physical vertical plane with its attribute of being a displayable blackboard. The first attribute indicates that the physical vertical plane can be displayed on the virtual blackboard object. Similarly, the first attribute indicates that the virtual blackboard object can (including, in various embodiments, is permitted) be displayed on the physical vertical plane. In various embodiments, the displayable blackboard attribute is defined by the user of GUI 201, by the creator of the virtual blackboard object, or by the creator of GUI 201. In various embodiments, the displayable blackboard attribute is defined as the functionality of various criteria. For example, in various embodiments, criteria include height and width values ​​within a specific range. In various embodiments, criteria include associating with an object having an object type (e.g., "wall") from a specific set of object types. In various embodiments, criteria include uniform color. In various embodiments, criteria include being specified by the user as a displayable blackboard after a vertical plane is detected. In various embodiments, the functionality of the various criteria is that for a vertical plane to be determined as a displayable blackboard, it must conform to all defined criteria. In various implementations, the functionality of various standards does not need to conform to all defined standards. For example, in various implementations, a vertical plane is determined to be a displayable blackboard if it (1) has a uniform color and (2A) is associated with the object type "wall" or (2B) its height and width values ​​are greater than a certain threshold. Thus, as an example, an electronic device detects a wall as a vertical plane, assigns the object type "wall" to the vertical plane, determines that the vertical plane has a uniform color (e.g., not covered with pictures, posters, or other patterns), and therefore determines that the vertical plane is a displayable blackboard. Alternatively, as another example, an electronic device detects the writing surface of a folding whiteboard (or a blank canvas on an easel) as a vertical plane, detects the whiteboard and assigns it an object type other than "wall" (e.g., "zoning" or "vertical-other"), determines that the height and width of the vertical plane are within a certain range, determines that the vertical plane has a uniform color (e.g., not covered with pictures, posters, or other patterns), and therefore determines that the vertical plane is a displayable blackboard.

[0075] In response to the detection of user input 299N for attribute display 222 and additional user input selecting a first attribute of the first physical horizontal plane, the first physical horizontal plane is associated with the first attribute. Therefore, a textual representation of the first physical horizontal plane 233C is displayed together with a textual representation of the first attribute in asset area 212. In various specific embodiments, a graphical representation of the first physical horizontal plane 243C is displayed in view area 213 with the first attribute; for example, the first physical horizontal plane is displayed as a wall or partition instead of a general horizontal plane.

[0076] Figure 2QThis describes user input 299Q for the asset type selection indicator 231, which selects a virtual asset list. In various embodiments, user input 299Q is achieved by the user tapping their finger or using a stylus on the touch-sensitive display at the location of the asset type selection indicator 231. In various embodiments, user input 299Q is achieved by the user clicking a mouse button while the cursor is displayed at the location of the asset type selection indicator 231.

[0077] Figure 2R This describes the response to the detection of user input 299D for asset type selection indication 231. Figure 2Q The GUI. (and) Figure 2Q In contrast, Asset Zone 212 displays a list of virtual assets associated with a scene, rather than a list of physical assets associated with a scene. Figure 2R In the asset area, the text representations of the virtual professor target implementer object 232A, the virtual blackboard object 232B, the virtual paper object 232C, the virtual paper stack object 232D, and the virtual mug object 232E are displayed.

[0078] Figure 2R This describes user input 299R for the text representation of the virtual blackboard object 232B. In various embodiments, user input 299R is input by the user tapping their finger or using a stylus at the location of the text representation on the virtual blackboard object 232B onto a touch-sensitive display. In various embodiments, user input 299R is input by the user clicking a mouse button when the cursor is displayed at the location of the text representation on the virtual blackboard object 232B.

[0079] Figure 2S This describes the response to detecting user input 299R representing text for the virtual blackboard object 232B. Figure 2R The GUI 201. In response to detecting user input 299R regarding the text representation of the virtual blackboard object 232B, the text representation of the virtual blackboard object 232B is displayed differently to indicate that the virtual blackboard object has been selected. In various embodiments, the graphical representation of the virtual blackboard object 242B is also (or alternatively) displayed differently to indicate that the virtual blackboard object has been selected. For example, in various embodiments, the graphical representation of the virtual blackboard object 242B is highlighted or displayed with ambient illumination. In various embodiments, user input regarding the graphical representation of the virtual blackboard object 242B selects the virtual blackboard object.

[0080] Figure 2SThe user input 299S for the attribute display 222 is described. In various embodiments, the user input 299S is input by the user tapping their finger or stylus on the touch-sensitive display at the location of the attribute display 222. In various embodiments, the user input 299S is input by the user clicking a mouse button when the cursor is displayed at the location of the attribute display 222.

[0081] Figure 2T This describes the response to detecting user input 299S for attribute display 222 and additional user input for selecting a first attribute of the virtual blackboard object. Figure 2S The GUI 201. Specifically, additional user input for selecting the first attribute includes user input that associates the virtual blackboard object with an attribute “on” the physical vertical plane. In response to detecting user input 299S for attribute indication 222 and additional user input for selecting the first attribute of the virtual blackboard object, the virtual blackboard object is associated with the first attribute. Therefore, the text representation of the virtual blackboard object 232B is displayed in the asset area 212 together with the text representation of the first attribute, and the graphical representation of the virtual blackboard object 242B is displayed in the view area 213 having the first attribute, for example, on the graphical representation of the physical vertical plane 243B.

[0082] Figure 2T This describes the user input 299T for the text representation of the virtual professor target implementer object 232A. In various embodiments, user input 299T is input by the user tapping their finger or using a stylus at the location of the text representation of the virtual professor target implementer object 232A onto a touch-sensitive display. In various embodiments, user input 299T is input by the user clicking a mouse button while the cursor is displayed at the location of the text representation of the virtual professor target implementer object 232A.

[0083] Figure 2U This describes the response to detecting user input 299T, a text representation of the virtual professor target implementer object 232A, and additional user input that associates attributes with various virtual objects. Figure 2T GUI 201.

[0084] Additional user input includes user input to associate the virtual professor target implementer object with a first attribute indicating that the virtual professor target implementer object is "on top of" the physical floor and a second attribute indicating that the virtual professor target implementer object is "near" the virtual blackboard object. Therefore, the text representation of the virtual professor target implementer object 232A is displayed in asset area 212 along with the text representations of the first and second attributes. Additionally, a graphical representation of the virtual professor target implementer object 242A is displayed in view area 213 having the first and second attributes, for example, on top of the graphical representation of the physical floor 243A and near the graphical representation of the virtual blackboard object 242B.

[0085] Additional user input includes user input to associate each of the virtual paper object, virtual paper stack object, and virtual mug object with a first attribute that the virtual object is "on top" of the second physical horizontal plane. Therefore, the text representations of virtual paper object 232C, virtual paper stack object 232D, and virtual mug object 232E are each displayed in asset area 212 along with the text representation of the first attribute. Additionally, the graphic representations of virtual paper object 242C, virtual paper stack object 242D, and virtual mug object 242E are each displayed in view area 213 with the first attribute, for example, each graphic representation is displayed on top of the graphic representation of the second physical horizontal plane 243D.

[0086] Figure 2U This describes the user input 299U for the asset type selection indicator 231, and its selection action: the asset list. In various embodiments, user input 299U is entered by the user tapping their finger or using a stylus on the touch-sensitive display at the location of the asset type selection indicator 231. In various embodiments, user input 299U is entered by the user clicking a mouse button while the cursor is displayed at the location of the asset type selection indicator 231.

[0087] Figure 2V This describes the response to the detection of user input 299U for the asset type selection indicator 231. Figure 2U The GUI. (and) Figure 2U In contrast, asset area 212 displays a list of action assets associated with the scene, rather than a list of physical assets associated with the scene. Figure 2V Currently, no action assets are associated with the scene, and asset area 212 is blank.

[0088] Figure 2VThis describes user input 299V for the asset addition capability indicator 221. In various embodiments, user input 299V is entered by the user tapping their finger or using a stylus at the location of the asset addition capability indicator 221 on the touch-sensitive display. In various embodiments, user input 299V is entered by the user clicking a mouse button while the cursor is displayed at the location of the asset addition capability indicator 221.

[0089] Figure 2W This describes the response to the detection of user input 299V for adding an indicator 221 to an asset and additional user input for adding numerous motion assets to the scene. Figure 2V The GUI 201. In response to the detection of user input 299V for adding an indication 221 to an asset and additional user input for adding a number of motion assets to the scene, the scene is associated with a number of motion assets, some of which are displayed by text representations in the asset area 212.

[0090] The motion assets include a first motion asset described by a textual representation of a first motion asset 234A. The first motion asset describes an action that begins at the start of the scene and includes a first voice given by a virtual teacher target implementer object, which may include both audio and animation of the virtual teacher target implementer object.

[0091] The motion assets include a second motion asset described by the text representation of the second motion asset 234B. The second motion asset describes the action that begins when the virtual teacher target implementer object finishes the first speech, and includes the virtual teacher target implementer object moving from the vicinity of the virtual blackboard object to the vicinity of the virtual paper object.

[0092] The motion assets include a third motion asset described by the text representation of the third motion asset 234C. The third motion asset describes the action that begins when the virtual teacher target implementer object is near the virtual paper object, and includes the virtual teacher target implementer object giving a second voice.

[0093] The motion assets include a fourth motion asset described by the text representation of the fourth motion asset 234D. The fourth motion asset describes the action that begins when the virtual teacher target implementer object finishes the second speech, and includes moving a virtual paper object from the top of the second physical horizontal plane to inside a physical trash can or on top of a physical floor. This can be achieved by the virtual teacher target implementer object picking up the virtual paper object and throwing it into the physical trash can or onto the physical floor.

[0094] The motion assets include a fifth motion asset described by the text representation of the fifth motion asset 234E. The fifth motion asset describes the action that begins when the virtual paper object is inside a physical trash can or on top of a physical floor, and includes a virtual professor target implementer object sitting on a first physical horizontal plane, which is where the professor can sit.

[0095] Figure 2W This describes user input 299W for the preview display 229. In various embodiments, user input 299W is input by the user tapping their finger or using a stylus at the location of the preview display 229 on the touch-sensitive display. In various embodiments, user input 299W is input by the user clicking a mouse button while the cursor is displayed at the location of the preview display 229.

[0096] Figure 3A This describes the response to the detection of a user input of 299W to the preview power display 229 in the first physical environment. Figure 2W The GUI 201. In response to the detection of user input 299W for the preview display 229, the asset area 212 and the view area 213 are replaced by the preview area 301 that provides a scene preview.

[0097] The first physical environment includes a television, a table, a wastebasket, and a wooden floor. Therefore, the preview area includes a representation of the first physical environment, including a representation of the television 311, a representation of the table 312, a representation of the wastebasket 313, and a representation of the wooden floor 314.

[0098] When providing a scene preview, electronic device 110 scans a first physical environment to determine whether the first physical environment includes objects of physical assets corresponding to a scene with associated attributes of physical assets. In doing so, electronic device 110 displays a scan notification 331.

[0099] In the first physical environment, the electronic device 110 determines that the wooden floor corresponds to the physical floor of the scene, the television is capable of displaying a blackboard and corresponds to a physical vertical plane, the top of the table has appropriate size and positional attributes and corresponds to a second physical horizontal plane, and the wastepaper basket corresponds to a physical trash can.

[0100] For each physical asset in the scene's physical assets, the electronic device 110 determines whether the physical asset is needed to execute the scene or optionally enhance it. In various specific implementations, physical assets are needed if virtual assets must be displayed in association with physical assets. For example, a physical floor is needed because the virtual professor target implementer object is displayed on top of the physical object corresponding to the physical floor. Similarly, a physical vertical plane is needed because the virtual blackboard object is displayed on the physical object corresponding to the physical vertical plane. A first physical horizontal plane is required because after performing the fifth action, the virtual professor target implementer object is displayed sitting on the physical object corresponding to the first physical horizontal plane. A second physical horizontal plane is required because the virtual paper object, the virtual paper stack object, and the virtual mug object are displayed on top of the physical objects corresponding to the second physical horizontal plane. In contrast, a physical trash can is optional because although the virtual paper object can be displayed inside the physical object corresponding to the physical trash can after performing the fourth action, if the physical object corresponding to the physical trash can does not exist in the physical environment, the virtual paper object can instead be displayed on top of the physical object corresponding to the physical floor after performing the fourth action.

[0101] Therefore, electronic device 110 determines that there is no object in the first physical environment corresponding to the first physical horizontal plane, and the first physical horizontal plane is required.

[0102] Figure 3B This describes the response to determining that there are no objects in the first physical environment corresponding to the first physical horizontal plane and that the first physical horizontal plane is required. Figure 3A The GUI 201. In response to determining that no object in the first physical environment corresponds to the first physical horizontal plane and that the first physical horizontal plane is required, the preview area 301 includes a missing required object notification 332 indicating that the first physical horizontal plane is required to perform a scene missing required object notification. In various specific implementations, the missing required object notification 332 provides the user with information about the attributes of the missing object. For example, in Figure 3B In the context of missing required objects, notification 332 indicates that a teachable object is needed to perform the scenario.

[0103] The missing object notification 332 includes a rescan capability indicator 333A, which, when selected, causes the electronic device 110 to scan a first physical environment to determine whether the first physical environment includes objects of physical assets corresponding to a scene with associated attributes of physical assets. The missing object notification 332 also includes an exit capability indicator 333B, which, when selected, causes the electronic device 110 to exit the scene preview.

[0104] Figure 3C This describes what happens after the user has added the stool to the first physical environment. Figure 3BThe GUI 201. Therefore, the preview area 301, which includes a representation of the first physical environment, includes a representation of the stool 315.

[0105] Figure 3C This describes user input 399A for the rescanning power indicator 333A. In various embodiments, user input 399A is entered by the user tapping their finger or using a stylus at the location of the rescanning power indicator 333A on the touch-sensitive display. In various embodiments, user input 399A is entered by the user clicking a mouse button while the cursor is displayed at the location of the rescanning power indicator 333A.

[0106] Figure 3D This describes the response to detecting user input 399A for rescanning the display 333A. Figure 3C The GUI 201. In response to detecting user input 399A for rescanning the power display 333A, the preview area 301 includes a scan notification 331, and the electronic device 110 scans the first physical environment to determine whether the first physical environment includes objects of physical assets corresponding to a scene with associated attributes of physical assets. While scanning the first physical environment, the electronic device 110 determines that the top of the stool can be sat on by the professor and therefore can correspond to the desired first physical horizontal plane.

[0107] Figure 3E This explains the response to determining that the top of the stool can be a seat for the professor. Figure 3D The GUI 201. In response to determining that the top of the stool may be suitable for the professor to sit on, the preview area 301 includes a highlight 340 of the representation surrounding the stool 315 and a confirmation notification 334 requesting confirmation from the user that the top of the stool is suitable for the professor to sit on.

[0108] Confirmation notification 334 includes a "yes" indication 335A, which, when selected, causes the electronic device 110 to associate the top of the stool with the "teacher can sit" attribute and map the top of the stool to a first physical horizontal plane. Confirmation notification 334 also includes a "no" indication 335B, which, when selected, causes the electronic device 110 to not associate the top of the stool with the "teacher can sit" attribute, indicating that an object corresponding to the first physical horizontal plane cannot be found, and displays a "missing object" notification 332.

[0109] Figure 3E This describes user input 399B for the power indicator 335A. In various embodiments, user input 399B is input by the user tapping their finger or using a stylus at the location of the power indicator 335A on the touch-sensitive display. In various embodiments, user input 399B is input by the user clicking a mouse button while the cursor is displayed at the location of the power indicator 335A.

[0110] Figure 3F This describes the response to detecting user input 399B for the display panel 335A. Figure 3E The GUI 201. In response to detecting user input 399B for the display 335A, the electronic device 110 associates the top of the stool with the professor's sitable attribute, determines that the top of the stool corresponds to a first physical horizontal plane, determines that the first physical environment includes objects corresponding to each required physical asset of the scene, and executes the scene.

[0111] When executing a scene, the preview area 301 includes a representation of a virtual blackboard object 322 displayed above the representation of the television 311, a representation of a virtual professor goal achiever object 321 displayed on the representation of the wooden floor 314 near the representation of the virtual blackboard object 322, a representation of a virtual paper object 323 on top of the representation of the table 312, a representation of a virtual paper stack object 324 on top of the representation of the table 312, and a representation of a virtual mug object 325 on top of the representation of the table 312.

[0112] In addition, preview area 301 includes a representation of the first action asset, in which the virtual professor target implementer object gives the first voice.

[0113] Figure 3G This describes the response to the virtual professor target implementer object ending the first speech. Figure 3F GUI 201. Figure 3G In the preview area 301, there is a representation of a second motion asset in which the virtual professor target implementer object moves from the vicinity of the virtual blackboard object to the vicinity of the virtual paper object, and a representation of a third motion asset in which the virtual professor target implementer object gives a second speech. Therefore, the representation of the virtual professor target implementer object 321 is displayed near the representation of the virtual paper object 323 rather than near the representation of the virtual blackboard object 322.

[0114] Figure 3H This describes the response to the virtual professor target implementer object ending the second speech. Figure 3G GUI 201. Figure 3H In the preview area 301, a representation of the fourth action asset is included, wherein if the physical environment includes an object corresponding to a physical trash can, the virtual paper object moves from the top of the second physical horizontal plane into the interior of the physical trash can; or if the physical environment does not include an object corresponding to a physical trash can, the virtual paper object moves from the top of the second physical horizontal plane to the physical floor. Therefore, the representation of the virtual professor target implementer object 321 is shown as the representation of throwing the virtual paper object 323 into the trash can 313.

[0115] Figure 3IThis describes the response to the virtual paper object inside the physical trash can. Figure 3H The GUI. Figure 3I In the preview area 301, there is a representation of the fifth action asset, in which the virtual professor target implementer object is sitting on the first physical horizontal plane. Therefore, the representation of the virtual professor target implementer object 321 is shown as sitting on the representation of stool 315.

[0116] Figure 4A This describes the response to the detection of a user input of 299W to the preview display 229 in the second physical environment. Figure 2W The GUI 201. In response to the detection of user input 299W for the preview display 229, the asset area 212 and the view area 213 are replaced by the preview area 401 that provides a scene preview.

[0117] The second physical environment includes walls, a desk, a tiled floor, and a chair. Therefore, the preview area 401 includes a representation of the second physical environment, including a representation of the wall 411, a representation of the desk 412, a representation of the tiled floor 414, and a representation of the chair 415.

[0118] When providing a scene preview, electronic device 110 scans a second physical environment to determine whether the second physical environment includes objects of physical assets corresponding to a scene with associated attributes of physical assets. In doing so, electronic device 110 displays a scan notification 331.

[0119] In the second physical environment, the electronic device 110 determines that the tiled floor corresponds to the physical floor of the scene, the wall is a blackboard that can be displayed and corresponds to the physical vertical plane, the top of the desk has appropriate size and position attributes and corresponds to the second physical horizontal plane, and the seat of the chair is a seat that the professor can sit on and corresponds to the first physical horizontal plane.

[0120] As described above, for each physical asset in the scene's physical assets, the electronic device 110 determines whether the physical asset is needed to execute the scene or optionally to enhance it. In various specific implementations, physical assets are needed if virtual assets must be displayed in association with physical assets. For example, a physical floor is needed because the virtual professor target implementer object is displayed on top of the physical floor. Similarly, a physical vertical plane is needed because the virtual blackboard object is displayed on the physical vertical plane. A first physical horizontal plane is required because the virtual professor target implementer object is displayed sitting on the first physical horizontal plane after the fifth action is performed. A second physical horizontal plane is required because the virtual paper object, the virtual paper stack object, and the virtual mug object are displayed on top of the second physical horizontal plane. In contrast, a physical trash can is optional because although the virtual paper object can be displayed inside the physical object corresponding to the physical trash can after the fourth action is performed, if the physical object corresponding to the physical trash can does not exist in the physical environment, the virtual paper object can instead be displayed on top of the physical object corresponding to the physical floor after the fourth action is performed.

[0121] Therefore, electronic device 110 determines that there is no object corresponding to the physical trash can in the second physical environment, and that the physical trash can is optional.

[0122] Figure 4B This indicates a response to determining that no object in the second physical environment corresponds to a physical trash can, and that the physical trash can is optional. Figure 4A The GUI 201. In response to determining that no object in the second physical environment corresponds to a physical trash can and that the physical trash can is optional, preview area 401 includes a missing optional object notification 336, which indicates that a physical trash can is not required to perform the scene, but is optional to enhance the scene. In various specific implementations, the missing optional object notification 336 provides the user with information about the attributes of the missing object. For example, in Figure 3B In the context of the missing optional object notification 336, the trash can will be enhanced in the scene.

[0123] The missing optional object notification 336 includes a rescan capability indication 337A, which, when selected, causes the electronic device 110 to scan a second physical environment to determine whether the second physical environment includes objects of physical assets corresponding to a scene with associated attributes of physical assets. The missing optional object notification 336 also includes a skip capability indication 337B, which, when selected, causes the electronic device 110 to execute a scene.

[0124] Figure 4BThis describes user input 499A for skipping the power indicator 337B. In various embodiments, user input 499A is input by the user tapping their finger or using a stylus at the location of the skip power indicator 337B on the touch-sensitive display. In various embodiments, user input 499A is input by the user clicking a mouse button when the cursor is displayed at the location of the skip power indicator 337B.

[0125] Figure 4C This describes the response to detecting user input 499A for skipping the display indicator 337B. Figure 4B The GUI201. In response to detecting user input 499A for skipping the display 337B, the electronic device 110 executes the scenario.

[0126] When executing a scene, the preview area 401 includes a representation of a virtual blackboard object 322 displayed above the representation of the wall 411, a representation of a virtual professor goal achiever object 321 displayed on the representation of the tiled floor 414 near the representation of the virtual blackboard object 322, a representation of a virtual paper object 323 on top of the representation of the desk 412, a representation of a virtual paper stack object 324 on top of the representation of the desk 412, and a representation of a virtual mug object 325 on top of the representation of the desk 412.

[0127] In addition, preview area 401 includes a representation of the first action asset, in which the virtual professor target implementer object gives the first voice.

[0128] Figure 4D This describes the response to the virtual professor target implementer object ending the first speech. Figure 4C GUI 201. Figure 4D In the preview area 401, there is a representation of a second motion asset in which the virtual professor target implementer object moves from the vicinity of the virtual blackboard object to the vicinity of the virtual paper object, and a representation of a third motion asset in which the virtual professor target implementer object gives a second speech. Therefore, the representation of the virtual professor target implementer object 321 is displayed near the representation of the virtual paper object 323 rather than near the representation of the virtual blackboard object 322.

[0129] Figure 4E This describes the response to the virtual professor target implementer object ending the second speech. Figure 4D GUI 201. Figure 4EIn the preview area 401, a representation of the fourth action asset is included, wherein if the physical environment includes an object corresponding to a physical trash can, the virtual paper object moves from the top of the second physical horizontal plane into the interior of the physical trash can; or if the physical environment does not include an object corresponding to a physical trash can, the virtual paper object moves from the top of the second physical horizontal plane to the physical floor. Therefore, the representation of the virtual professor target implementer object 321 is shown as throwing the representation of the virtual paper object 323 onto the representation of the tile floor 414.

[0130] Figure 4F This describes the response to the virtual paper object on top of the physical floor. Figure 4E The GUI. Figure 4F In the preview area 401, there is a representation of the fifth action asset, in which the virtual professor target implementer object is sitting on the first physical horizontal plane. Therefore, the representation of the virtual professor target implementer object 321 is shown as sitting on the representation of chair 415.

[0131] Figure 5 This is a flowchart representation of a method 500 for displaying content according to some specific embodiments. In various embodiments, method 500 is performed by a device having a display, one or more processors, and non-transitory memory. In some embodiments, method 500 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 500 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0132] Method 500 begins in block 510, wherein a device scans a first physical environment to detect a first physical object and a second physical object within the first physical environment, wherein the first physical object conforms to at least one first object criterion, and the second physical object conforms to at least one second object criterion. For example, in Figure 3A In this process, electronic device 110 scans a first physical environment and detects the front of the television as a first physical object conforming to a physical vertical plane standard, and detects the top of the table as a second physical object conforming to a second physical horizontal plane standard. In this example, at least one first object standard includes that the front of the television is a vertical plane and can display a blackboard, and at least one second object standard includes that the top of the table is a horizontal plane, at least half a meter away from the floor, and has a width of at least one meter and a length of at least half a meter.

[0133] In various embodiments, detecting physical objects in a physical environment (e.g., detecting a first physical object in a first physical environment or detecting a second physical object in a first physical environment) includes capturing an image of the physical environment and detecting representations of the physical objects within that image. In various embodiments, capturing the image of the physical environment is performed by a camera or other image sensor. In various embodiments, detecting physical objects in a physical environment includes generating a three-dimensional model of the physical environment, such as a point cloud, and detecting representations of the physical objects within that three-dimensional model. In various embodiments, generating the three-dimensional model of the physical environment includes obtaining depth information using a depth sensor.

[0134] In various specific implementations, at least one first object criterion includes a criterion regarding the size or shape of the first physical object. For example, in Figure 2Q In this context, the second physical horizontal plane is associated with an attribute having a width greater than one meter. In various specific implementations, at least one first object criterion includes a criterion regarding the object type of the first physical object. For example, in... Figure 2Q In this context, a trash can is associated with a property that has an object type "trash can".

[0135] In various specific implementations, at least one first object criterion includes a criterion regarding the availability of the first physical object. For example, in Figure 2Q In this context, the first physical horizontal plane is associated with the "teacher can sit" attribute. In various implementations, the criterion for the usability of the first physical object is defined relative to the virtual object of the content. For example, the "teacher can sit" attribute is defined relative to the virtual teacher target implementer object. In various implementations, the criterion for the usability of the first physical object is defined relative to the fifth physical object. For example, in various implementations, the physical vertical plane can be associated with the "marker can write" attribute, allowing the corresponding object to not only display a blackboard on it but also write on it with physical chalk or a physical dry-erasable marker. In various implementations, the fifth physical object is the user of the device. For example, in various implementations, the first physical horizontal plane can be associated with the "user can sit" attribute, allowing the corresponding object to be sat on not only by the virtual teacher target implementer object but also by the user.

[0136] Method 500 continues in box 520, where the device displays, in association with a first physical environment, a virtual object moving along a first path from a first physical object to a second physical object. For example, in Figure 3G In the middle, the representation of the virtual professor target implementer object 321 moves along a first path from the representation of the television 311 (where the representation of the virtual blackboard object 322 is displayed) to the representation of the table 312 (where the representation of the virtual paper object 323 is displayed).

[0137] In various embodiments, the display is an opaque display, and the virtual object is displayed in association with a first physical environment as a composite image of the virtual object and the physical environment. Therefore, in various embodiments, displaying a virtual object in association with a first physical environment includes displaying a composite image of the virtual object and the first physical environment. In various embodiments, the display is a transparent display, and the virtual object is displayed in association with a physical environment as a projection onto a view of the first physical environment. Therefore, in various embodiments, displaying a virtual object in association with a first physical environment includes projecting the virtual object onto a view of the first physical environment.

[0138] In various specific implementations, displaying virtual objects in relation to a physical environment includes displaying the virtual objects on a representation or view of a first or second physical object. For example, in Figure 3F In this embodiment, electronic device 110 displays a representation of virtual teaching target implementer object 321 on a display of television 311. In various specific embodiments, displaying the virtual object in association with a first physical environment includes displaying the virtual object near a representation or view of a first or second physical object. For example, in... Figure 3G In the display, the electronic device 110 shows a representation of the virtual professor target implementer object 321 near the table 312, making it appear as if the virtual professor target implementer object is near the table.

[0139] In various specific implementations, displaying virtual objects in association with a first physical environment includes displaying virtual objects that interact with representations or views of the first or second physical object. For example, in Figure 3I In the middle, the electronic device 110 displays a representation of a virtual professor target implementer object 321 sitting on a stool 315.

[0140] The method continues in block 530, wherein the device scans the second physical environment to detect a third physical object in the second physical environment and a fourth physical object in the second physical environment, wherein the third physical object conforms to at least one first object criterion and the fourth physical object conforms to at least one second object criterion. For example, in Figure 4A In this process, electronic device 110 scans the second physical environment and detects the wall as a third physical object conforming to the object standard of a physical vertical plane, and detects the top of the desk as a fourth physical object conforming to the object standard of a second physical horizontal plane. As described above, in this example, at least one first object standard includes that the wall is a vertical plane and can display a blackboard, and at least one second object standard includes that the top of the desk is a horizontal plane, at least half a meter away from the floor, and has a width of at least one meter and a length of at least half a meter.

[0141] Method 500 continues in box 540, where virtual objects moving along a second path from a third physical object to a fourth physical object are displayed in association with a second physical environment, where the second path differs from the first path. For example, in Figure 4D In the middle, the representation of the virtual professor target implementer object 321 moves along a second path from the representation of the wall 411 (where the representation of the virtual blackboard object 322 is displayed) to the representation of the desk 412 (where the representation of the virtual paper object 323 is displayed).

[0142] As mentioned above, the second path differs from the first path. For example, in Figure 3G In this context, the representation of the virtual professor target implementer object 321 moves a shorter distance from the representation of the television 311 (where the representation of the virtual blackboard object 322 is displayed) to the representation of the table 312 (where the representation of the virtual paper object 323 is displayed). In contrast, in... Figure 4D In this context, the representation of the virtual professor target implementer object 321 moves a longer distance from the representation of the wall 411 (where the representation of the virtual blackboard object 322 is displayed) to the representation of the desk 412 (where the representation of the virtual paper object 323 is displayed). Therefore, in various implementations, the length of the second path differs from the length of the first path. The first and second paths can also differ in various other measurements. For example, in various implementations, the variance of the second path (e.g., measured from a straight line) differs from the variance of the first path.

[0143] In various specific implementations, the first path traverses multiple first path locations in the three-dimensional coordinate system, and the second path traverses multiple second path locations in the three-dimensional coordinate system. For example, in Figure 3G In the middle, the representation of the virtual professor target implementer object 321 is only displayed in front of the user, while... Figure 4D In this context, the virtual professor target implementer object 321 is represented further away and displayed to the left. In various implementations, the three-dimensional coordinate system is defined relative to the user, camera viewpoint, or another common feature in the physical environment (e.g., the far upper right corner of the room). In various implementations, the three-dimensional coordinate system is defined, for example, as an absolute (or global) coordinate system using GPS.

[0144] In various embodiments, the method includes determining at least one of a first path or a second path. In various embodiments, the device utilizes various pathfinding algorithms to determine at least one of the first path or the second path. In various embodiments, at least one of the first path or the second path is determined to avoid another object, which may be a virtual object or a physical object in the physical environment. In various embodiments, a virtual target implementer determines at least one of the first path or the second path, for example, based on the capabilities of the virtual target implementer. For example, a virtual dog target implementer object can use stairs to move from the floor to the bed, while a virtual fly target implementer object can fly from the floor to the bed.

[0145] Figure 6 This is a flowchart representation of a method 600 for associating tags with objects according to some specific implementations. In various implementations, method 600 is executed by a device having one or more processors and non-transitory memory. In some implementations, method 600 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some implementations, method 600 is executed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0146] Method 600 begins in box 610, where the device obtains tags associated with multiple standards. For example, in... Figure 2Q In this context, electronic device 110 has been labeled "can display blackboard" and "professor can sit," with each label associated with multiple standards.

[0147] Method 600 continues in box 620, where the device detects physical objects in the physical environment. For example, in Figure 3A In the process, electronic device 110 detects a television, a table, a wastebasket, and a wooden floor in the first physical environment. As another example, in... Figure 4A In the process, electronic device 110 detects the walls, desk, tiled floor, and chair in the second physical environment.

[0148] In various implementations, labels indicate the functionality that a physical object can support. Therefore, in various implementations, the labels are referred to as "capability labels" and the standards as "capability standards." For example, the label "displays blackboard" indicates that a physical object is capable of displaying a virtual blackboard object on said physical object. As another example, the label "professor can sit" indicates that a physical object is capable of allowing a virtual professor to sit on the physical object.

[0149] In various specific implementations, labels indicate that a physical object can be used by a virtual object to perform a function. For example, the label "displayable blackboard" indicates that a physical object can be used by a virtual blackboard object to be displayed. As another example, the label "professor can sit" indicates that a physical object can be sat on by a virtual professor target implementer object. Therefore, rather than identifying a particular object as having a specific object type (e.g., "wall") and then assigning functionality to the physical object (e.g., content can be displayed on a physical object identified as "wall"), labels are associated with the capabilities of a physical object (e.g., "displayable blackboard"), and labels are assigned only if the physical object meets the functionality indicated by multiple criteria for those capabilities.

[0150] In various specific implementations, method 600 further includes a virtual object that displays the functions performed. For example, in Figure 3I In the display, the electronic device 110 displays a representation of a virtual blackboard object 322 on a display of a television 311 that is determined to be a displayable blackboard, and displays a representation of a virtual professor target implementer object 321 sitting on a stool 315 that is determined to be a professor's seat.

[0151] In various implementations, labels indicate that a physical object can be used by the physical object to perform a function. Therefore, in various implementations, the labels are called "usability labels" and the standards are called "usability standards." For example, in various implementations, a physical vertical plane can be associated with a marker-writable attribute, such that the corresponding object can not only display a blackboard on it, but also write on it with physical chalk or a physical dry-erasable marker. As another example, in various implementations, a first physical horizontal plane can be associated with a user-seated attribute, such that the corresponding object can be sat on not only by a virtual teacher target implementer object, but also by a user. Therefore, the label "marker-writable" indicates that the physical object can be used by the physical marker to be written on. As another example, the label "user-seated" indicates that the physical object can be used by a user to sit on.

[0152] In various implementations, multiple criteria include those concerning the size or shape of a physical object. For example, in various implementations, the label "displayable blackboard" is associated with a criterion that the height and width of the physical object are within a specific range. As another example, in various implementations, the label "seatable for professors" is associated with a criterion that the height, length, and width of the physical object are within a specific range.

[0153] In various implementations, multiple standards include those concerning the color of physical objects. For example, in various implementations, the label "displayable blackboard" is associated with the standard that the physical object is a uniform color.

[0154] In various implementations, multiple criteria include those concerning the object type of the physical object. For example, in various implementations, the label "displayable blackboard" is associated with the criteria that the physical object is a vertical plane and / or that the physical object has the object type "wall". As another example, in various implementations, the label "displayable professor" is associated with the criteria that the physical object is a horizontal plane and / or that the physical object has an object type from a specific set of object types (e.g., "chair", "stool", "sofa", etc.).

[0155] In various implementations, multiple standards include user-specified standards regarding physical objects. For example, in various implementations, the label "displayable blackboard" is associated with a standard that the user designates a physical object as a displayable blackboard. As another example, in various implementations, the label "displayable professor" is associated with a standard that the user designates a physical object as a displayable professor (e.g.,...). Figure 3E The standards (conducted in the process) are related.

[0156] Method 600 continues in box 630, where the device determines that a physical object fulfills a number of criteria. For example, in various implementations, if a physical object meets a first criterion of having a uniform color, and a second criterion of having an object type “wall”, or a third criterion of its height and width values ​​being greater than a specific threshold, then the physical object is associated with the label “Displayable Blackboard.” As another example, if a physical object meets a first criterion of having an object type “chair,” or meets a second criterion of its height, length, and width values ​​being within a specific range, and a third criterion of the user designating a horizontal plane as “Professor Can Sit” after detecting a physical object with height, length, and width values ​​within the specific range, then the physical object is associated with the label “Professor Can Sit.”

[0157] In various implementations, a physical object fulfills the function of multiple criteria when it meets one or more of multiple criteria. In various implementations, a physical object fulfills the function of multiple criteria when it meets all of the multiple criteria. For example, in various implementations, if a physical object meets a first criterion of having the object type "wall" and a second criterion of having a uniform color, then the physical object is associated with the label "displayable blackboard".

[0158] In various implementations, a physical object is considered to fulfill the functionality of multiple criteria when it meets at least a threshold number of those criteria. For example, in various implementations, a physical object is associated with the label "Professor Can Sit" if it meets at least one of the following criteria: a first criterion that it has the object type "chair," a second criterion that it has the object type "stool," or a third criterion that the user designates it as a "professor can sit." In various implementations, a physical object is associated with the label if it meets at least two of three or more criteria.

[0159] In various implementations, a physical object is considered to satisfy the functionality of multiple criteria when it meets a first subset or a second subset of multiple criteria. In various implementations, the first and second subsets of multiple criteria do not include common criteria. For example, in various implementations, a physical object is associated with the label "Professor Can Sit" if it meets a first subset of criteria (including the first criterion that it has the object type "Chair"), or if it meets a second subset of criteria (including the second criterion that its height, length, and width values ​​are within a specific range, and the third criterion that the user designates the horizontal plane as "Professor Can Sit" after detecting a physical object with height, length, and width values ​​within the specific range). In various implementations, the first and second subsets of multiple criteria include one or more common criteria. For example, in various specific implementations, a physical object is associated with the label "displayable blackboard" if it conforms to a first subset of the criteria (including the first criterion that it has a uniform color and the second criterion that it has the object type "wall"), or conforms to a second subset of the criteria (including the first criterion that it has a uniform color and the third criterion that its height and width values ​​are greater than a certain threshold).

[0160] Method 600 continues at block 640, wherein the device generates a feature vector of the physical object in response to determining that the physical object meets a number of criteria, the feature vector including an object identifier and a tag for the physical object. In various embodiments, the object identifier is an assigned number, letter, or other unique tag. In various embodiments, generating the feature vector of the physical object, including the object identifier and the tag, includes adding the tag to an existing feature vector, which in various embodiments includes the object identifier.

[0161] In various specific implementations, having a physical object associated with a tag can be used to determine whether to execute a scenario that requires the physical object associated with the tag. For example, in Figure 3B In the first scenario, electronic device 110 does not execute the scenario because no physical object is associated with the label "Professor can sit". In contrast, in the second scenario... Figure 3FIn this context, electronic device 110 executes a scenario because the stool is associated with the label "Professor can sit here." Therefore, in various specific implementations, method 600 further includes executing a scenario that requires a physical object associated with the label.

[0162] Figure 7 This is a flowchart representation of method 700 according to some specific implementation scenarios. In various implementations, method 700 is executed by a device having a display, one or more processors, and non-transitory memory. In some implementations, method 700 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some implementations, method 700 is executed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0163] Method 700 begins in box 710, where the device obtains a scenario associated with one or more execution constraints. For example, in Figure 3A In this process, the electronic device 110 has obtained a scenario requiring a physical environment, which includes a physical floor, a physical vertical plane, a first physical horizontal plane, and a second physical horizontal plane, each of which has specific properties.

[0164] Method 700 continues in box 720, where the device scans the physical environment to determine whether the physical environment conforms to one or more execution constraints. For example, in Figure 3A In the process, electronic device 110 scans the first physical environment to determine whether the first physical environment meets the execution constraints of the scenario.

[0165] In various implementations, scanning the physical environment to determine whether it conforms to one or more execution constraints includes detecting physical objects within the physical environment. In various implementations, detecting physical objects in the physical environment includes capturing an image of the physical environment and detecting representations of physical objects within that image. In various implementations, capturing the image of the physical environment is performed by a camera or other image sensor. In various implementations, detecting physical objects in the physical environment includes generating a 3D model of the physical environment, such as a point cloud, and detecting representations of physical objects within that 3D model. In various implementations, generating the 3D model of the physical environment includes obtaining depth information using a depth sensor.

[0166] In various specific implementations, scanning the physical environment to determine whether the physical environment meets one or more execution constraints includes analyzing data from various sensors, such as thermometers, ambient light sensors, GPS sensors, or clocks.

[0167] In various specific implementations, one or more execution constraints include execution constraints that are met when the physical environment includes physical objects with specific attributes.

[0168] In various specific implementations, the specific attribute relates to the size or shape of the physical object. For example, in Figure 3A In this scenario, the scene requires a physical environment, which includes a second physical horizontal plane with a width greater than one meter. In various specific implementations, a particular attribute relates to the color of physical objects. For example, in... Figure 3A In this scenario, a physical environment is required, which includes a physical vertical plane that can display the blackboard. In various specific implementations, the physical vertical plane needs to have a uniform color.

[0169] In various specific implementations, a particular attribute relates to the object type of the physical object. For example, in Figure 3A In this context, the scene needs to include the physical environment as physical objects on a horizontal plane. As another example, in... Figure 3A In this scenario, the scene needs to include the physical environment of the physical floor. As another example, in Figure 3A In this scenario, a physical environment is required, which includes a physical vertical plane that can display the blackboard. In various specific implementations, the physical vertical plane needs to have an object type "wall".

[0170] In various specific implementations, the specific attribute relates to the ability to support functions. For example, in Figure 3A In this scenario, the scene requires a first physical horizontal plane that allows the virtual teacher to sit and supports the virtual teacher's target implementer object sitting on a physical object. In various specific implementations, a particular attribute relates to the capabilities of the physical object used by the virtual object to perform the function. For example, in... Figure 3A In the scenario, the teacher needs to sit on a first physical horizontal plane that can be used by the virtual teacher target implementer object.

[0171] In various implementations, a specific attribute relates to the capabilities of a physical object used by the physical object to perform a function. For example, in various implementations, a physical vertical plane can be associated with the attribute of being writable by a marker, such that the corresponding object can not only display a blackboard on it, but also write on it with physical chalk or a physical dry-erasable marker. In various implementations, a third physical object is the user of the device. For example, in various implementations, a first physical horizontal plane can be associated with the attribute of being user-seated, such that the corresponding object can be sat on not only by a virtual teacher target implementer object, but also by a user.

[0172] In various implementations, one or more execution constraints include execution constraints that are met when the physical environment has specific environmental characteristics. In various implementations, specific environmental characteristics include temperature, humidity, pressure, ambient lighting conditions, time, or location.

[0173] Method 700 continues in box 730, wherein the device executes a scenario in response to determining that the physical environment conforms to one or more execution constraints, wherein the execution scenario includes display content associated with a physical object.

[0174] In various embodiments, the display is an opaque display, and the content is displayed in association with a physical object as a composite image of the content and the physical object. Therefore, in various embodiments, displaying content in association with a physical object includes displaying a composite image of the content and the physical object. In various embodiments, the display is a transparent display, and the content is displayed in association with a physical object as a projection onto a view of the physical object. Therefore, in various embodiments, displaying content in association with a physical object includes projecting content onto a view of the physical object.

[0175] In various specific implementations, displaying content associated with physical objects includes displaying virtual objects on a representation or view of the physical object. For example, in Figure 3F In this embodiment, electronic device 110 displays a representation of virtual blackboard object 322 on the display of television 311. In various specific embodiments, displaying content associated with a physical object includes displaying the virtual object near the representation or view of the physical object. For example, in... Figure 3F In the middle, the electronic device 110 displays a representation of a virtual mug object 325 near the table 312, making it appear as if the virtual mug object 325 is on the top of the table.

[0176] In various specific implementations, the content displayed in association with physical objects includes virtual objects that display representations or views of the physical objects. For example, in Figure 3I In the middle, the electronic device 110 displays a representation of a virtual professor target implementer object 321 sitting on a stool 315.

[0177] In various implementations, the device displays an indication of non-compliance of one or more execution constraints in response to determining that the physical environment does not conform to one or more execution constraints. In various implementations, displaying an indication of non-compliance of one or more execution constraints includes displaying an indication of non-compliance of execution constraints even when no execution scenario is executed. For example, in... Figure 3B In response to determining that a first physical environment without a stool does not meet one or more execution constraints, in the absence of an execution scenario, electronic device 110 displays a missing required object notification 336, which indicates that the execution constraint, including a physical object where a professor can sit, is not met.

[0178] In various specific implementations, method 700 further includes displaying a rescanning power indicator in response to determining that the physical environment does not meet one or more execution constraints, detecting user input selecting to rescan the power indicator, and rescanning the physical environment in response to detecting user input to determine whether the physical environment meets one or more execution constraints. For example, in Figure 3B In the middle, electronic device 110 displays a rescan power indicator 333A; in Figure 3C In the middle, the user input 399A was selected to rescan the indicator display 333A; and, in Figure 3D In the middle, rescan the first physical environment.

[0179] In various implementations, the scenario is further associated with one or more execution guidelines. In various implementations, method 700 includes scanning the physical environment to determine whether the physical environment conforms to one or more execution guidelines, and in response to determining that the physical environment does not conform to one or more execution guidelines, displaying an indication of non-conforming execution guidelines from one or more execution guidelines.

[0180] For example, in Figure 4A In this context, the scene and physical environment are associated with execution guidelines, including physical objects with the object type "trash can". Figure 4A In the process, electronic device 110 scans the second physical environment to determine that the second physical environment does not include physical objects with the object type "trash can," and therefore does not comply with the implementation guidelines. Figure 4B In response to the determination that the second physical environment does not conform to the implementation guidelines, electronic device 110 displays a missing optional object notification 336.

[0181] In various specific implementations, method 700 further includes displaying a rescanning power indicator in response to determining that the physical environment does not conform to one or more execution guidelines, detecting user input selecting to rescan the power indicator, and rescanning the physical environment to determine whether the physical environment conforms to one or more execution guidelines in response to detecting user input. For example, in Figure 4B In the middle, electronic device 110 displays a rescan power indicator 337A.

[0182] In various specific implementations, method 700 further includes displaying a skip power indication in response to determining that the physical environment does not conform to one or more execution guidelines, detecting user input that selects to skip the power indication, and executing the scenario in response to detecting the user input. For example, in Figure 4B In the middle, electronic device 110 displays a skipped power indicator 337B; in Figure 4B In the middle, the user input 399A was selected to rescan the indicator display 333A; and, in Figure 4C In the execution scenario.

[0183] In various specific implementations, the device determines which of a plurality of scenarios (if any) can be executed, rather than determining whether a scenario should be executed. Therefore, in box 710, obtaining the scenarios associated with one or more execution constraints includes obtaining multiple scenarios associated with a corresponding set of one or more execution constraints. Furthermore, in box 720, scanning the physical environment to determine whether the physical environment conforms to one or more execution constraints includes scanning the physical environment to determine which execution constraint (if any) in the corresponding set of one or more execution constraints the physical environment conforms to.

[0184] In various implementations, the device displays indications (e.g., a list) of which scenarios out of multiple scenarios can be executed in association with a physical environment, for example, which scenarios out of multiple scenarios are associated with a corresponding set of one or more execution constraints conforming to the physical environment. In various implementations, the device further displays indications of which scenarios out of multiple scenarios may not be executed in association with a physical environment, for example, which scenarios out of multiple scenarios are associated with non-compliance of execution constraints. In various implementations, the device further displays indications of reasons why a scenario might not be executed, for example, in response to the selection of scenarios that may not be executed, for example, indications preventing the scenario from being executed due to non-compliance of execution constraints (or multiple non-compliance of execution constraints).

[0185] Figure 8 This is a flowchart representation of a method 800 according to several specific implementation scenarios. In various implementations, method 800 is executed by a device having a display, one or more processors, and non-transitory memory. In some implementations, method 800 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some implementations, method 800 is executed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0186] Method 800 begins in box 810, where the device displays a representation of the scene. For example, in Figures 2A to 2W In the view area 213, the device displays a representation of the scene.

[0187] Method 800 continues in box 820, where the device receives user input that associates multiple physical assets with the scene. For example, in Figures 2E to 2J In this context, electronic device 110 receives user input 299E and other inputs that interact with asset addition display 221 and other user interface elements to associate physical floor, physical vertical plane, first physical horizontal plane, second physical horizontal plane and physical trash can with the scene.

[0188] In various specific implementations, method 800 includes a device displaying multiple representations of multiple physical assets within a scene representation. For example, in Figure 2JIn the view area 213, the electronic device 110 displays a representation of the physical floor 243A, a representation of the physical vertical plane 243B, a representation of the first physical horizontal plane 243C, a representation of the second physical horizontal plane 243D, and a representation of the physical trash can 243E.

[0189] Method 800 continues in box 830, where the device receives user input that associates multiple virtual assets with the scene. For example, in Figures 2A to 2D In this context, electronic device 110 receives user input 299A and other inputs that interact with asset addition display 221 and other user interface elements to associate virtual teacher target implementer objects, virtual blackboard objects, virtual paper objects, virtual paper stack objects, and virtual mug objects with the scene.

[0190] In various specific implementations, method 800 includes a device displaying corresponding multiple representations of multiple virtual assets within a scene representation. For example, in Figure 2D In the view area 213, the electronic device 110 displays a graphical representation of the virtual professor target implementer object 242A, the virtual blackboard object 242B, the virtual paper object 242C, the virtual paper stack object 242D, and the virtual mug object 242E.

[0191] In various implementations, multiple virtual assets include one or more invisible marker objects, which are represented in the scene's representation but not displayed when the scene is executed. For example, see reference... Figures 2A to 2W In various specific implementations, the scene is associated with a virtual asset of a first virtual invisible marker object having the attribute of being located on top of the physical floor, and a virtual asset of a second virtual invisible marker object having the first attribute of being located on top of the floor and the second attribute of being far away from the first virtual invisible marker object. Furthermore, the scene is associated with the motion asset of a virtual teacher target implementer object that repeatedly moves between the top of the first virtual invisible marker object and the top of the second virtual invisible marker object. Therefore, when the scene is executed, the representation of the virtual teacher target implementer object 321 appears to be pacing back and forth in the room.

[0192] Method 800 continues in block 840, wherein the device receives user input that associates a first attribute with a first virtual asset among a plurality of virtual assets, wherein the first attribute is a spatial relationship between the first virtual asset and a first physical asset among a plurality of physical assets. For example, in Figure 2SIn this process, electronic device 110 detects user input 299S interacting with attribute display 222 and other user input interacting with other user interface elements to associate a first attribute with a virtual blackboard object, the first attribute being the spatial relationship between the virtual blackboard object and the virtual blackboard object in a physical vertical plane.

[0193] Method 800 continues in box 850, where the device displays a representation of a first virtual asset that is spatially related to the representation of the first physical asset within the representation of the scene. For example, in Figure 2T In the process, the electronic device 110 displays a graphical representation of the virtual blackboard object 242B in terms of spatial relationships on a physical vertical plane 243B.

[0194] In various specific implementations, method 800 includes receiving user input that associates a second attribute with a second physical asset among a plurality of physical assets. For example, in Figures 2K to 2M In this process, electronic device 110 detects user input 299K and other user input interacting with attribute display representation 222 and other user interface elements to associate attributes with a second physical horizontal plane. In various specific embodiments, method 800 further includes displaying a representation of the second physical asset in the scene representation as having (or exhibiting) a second attribute. For example, in... Figure 2M In this process, the electronic device 110 displays a representation of the second physical horizontal plane as having a height of at least half a meter, a length of at least one meter, and a width of half a meter.

[0195] In various specific implementations, method 800 further includes receiving user input that associates a second attribute with a second physical asset among a plurality of physical assets, wherein the user input includes selection of an attribute associated with the second virtual asset among a plurality of virtual assets. For example, in Figure 2N In this process, electronic device 110 detects user input 299N interacting with attribute display 222 and other user input interacting with other user interface elements, in order to associate the professor seating attribute with a first physical horizontal plane by selecting the professor seating attribute associated with the virtual professor target implementer object.

[0196] In various specific implementations, the attributes associated with the second virtual asset are the capabilities of the second physical asset used by the second virtual asset to perform functions. For example, the "teacher can sit" attribute is the ability of a first physical horizontal plane on which a virtual teacher target implementer object can sit.

[0197] In various specific implementations, method 800 further includes receiving user input that associates multiple action assets with a scene. For example, in Figure 2VIn this context, electronic device 110 detects user input 299V and other user inputs that interact with the asset addition display representation and other user inputs that interact with other user interface elements, in order to associate the first action asset, the second action asset, the third action asset, the fourth action asset, and the fifth action asset with the scene.

[0198] In various implementations, action assets describe actions performed in response to specific triggers. In various implementations, triggers include the start of a scene's execution. For example, in... Figure 2W In this context, the first action asset (represented by the text representation of the first action asset 234A) is triggered by initiating the execution scene. In various specific implementations, the trigger includes objects with specific spatial relationships. For example, in... Figure 2W In this context, when the virtual teacher target implementer object is near the virtual paper object, a third action asset (represented by the text representation of third action asset 234C) is triggered. In various specific implementations, the trigger includes a user who is in a specific location or spatial relationship relative to the virtual object or another physical object.

[0199] In various specific implementations, actions include changes in the spatial relationships between objects. For example, in Figure 2W In this context, the second action asset (represented by the textual representation of the second action asset 234B) indicates the change of the virtual teacher target implementer object from its proximity to the virtual blackboard object to its proximity to the virtual paper object. In various implementations, the action includes ending a scene and executing different scenes.

[0200] In various implementations, method 800 further includes determining, for each of a plurality of physical assets, whether the physical asset is necessary or optional for the execution scenario. In various implementations, if a virtual asset is necessarily displayed in association with a physical object corresponding to the physical asset, then it is determined that a physical asset is required. For example, in Figure 3A In this context, a physical floor is required because the virtual professor target implementer object is displayed on top of the physical object corresponding to the physical floor. Similarly, a physical vertical plane is required because the virtual blackboard object is displayed on the physical object corresponding to the physical vertical plane. A first physical horizontal plane is required because after performing the fifth action, the virtual professor target implementer object is displayed sitting on the physical object corresponding to the first physical horizontal plane. A second physical horizontal plane is required because the virtual paper object, virtual paper stack object, and virtual mug object are displayed on top of the physical objects corresponding to the second physical horizontal plane. In contrast, a physical trash can is optional because although the virtual paper object can be displayed inside the physical object corresponding to the physical trash can after performing the fourth action, if a physical object corresponding to the physical trash can does not exist in the physical environment, the virtual paper object can instead be displayed on top of the physical object corresponding to the physical floor after performing the fourth action.

[0201] In various specific implementations, the device determines whether each of a number of physical assets is necessary or optional for scenario execution before attempting to execute the scenario, and stores this information in association with the scenario. This type of preprocessing can reduce latency in scenario execution.

[0202] In various implementations, the device determines whether each of a plurality of physical assets is required or optional based on the initial attributes of the virtual assets (e.g., a second physical horizontal plane is required because the virtual physical mug is displayed on top of the physical object corresponding to the second physical horizontal plane) and the attributes of the virtual assets modified by the action assets (e.g., a first physical horizontal plane is required because after performing the fifth action, the virtual professor target implementer object is displayed sitting on the physical object corresponding to the first physical horizontal plane). Therefore, in various implementations, determining whether a physical asset is required or optional for the execution scenario for each of a plurality of physical assets is based on the initial attributes of the virtual assets and the attributes of the virtual assets, such as those modified by the action assets.

[0203] In various implementations, the device further determines whether each physical asset among the physical assets is required or optional based on heuristics stored by the device. Therefore, in various implementations, determining whether a physical asset is required or optional for the execution scenario for each of multiple physical assets is heuristic-based. These heuristics allow the device to creatively conform to execution constraints. For example, relative to… Figures 2A to 2W In various implementations, a first physical horizontal plane is required because after the fifth action is performed, the virtual professor target implementer object is displayed sitting on a physical object corresponding to the first physical horizontal plane. Therefore, the scene is associated with the execution constraint that the physical environment includes a professor-seated physical object. In various implementations, the device includes a heuristic that if the physical environment does not include a professor-seated physical object, the device generates a virtual chair object (and displays a representation of the virtual chair object when executing the scene), the virtual chair object being interpreted as a professor-seated physical object when attempting to execute the scene and during scene execution. Therefore, in various implementations, the heuristic includes a heuristic for generating a virtual object that is interpreted as a physical object.

[0204] In various implementations, the device includes a heuristic that changes the fifth action asset from including a virtual professor target implementer object sitting on a first physical horizontal plane (which is where the professor can sit) to including a virtual professor target implementer object sitting on (1) the first physical horizontal plane (which is where the professor can sit) or (2) the physical floor. Therefore, in various implementations, the heuristic includes a heuristic for modifying the action asset to include alternative actions. This renders an optional first physical horizontal plane, rather than the one required for executing the scene.

[0205] As described above, in various implementations, the device determines whether each of a plurality of physical assets is required or optional based on the initial attributes of the virtual asset and its attributes as modified by the action asset. In various implementations, the action asset is, for example, specified as necessary or unnecessary by the user, and the device determines whether each of the plurality of physical assets is required or optional based on the initial attributes of the virtual asset and its attributes as modified by the action asset specified as necessary. Therefore, in various implementations, determining whether a physical asset is necessary or optional for the execution scenario for each of the plurality of physical assets is based on the initial attributes of the virtual asset and its attributes as modified by the action asset specified as necessary.

[0206] For example, refer to Figures 2A to 2W In various implementations, the scene includes a sixth action asset, which includes a virtual professor goal implementer object kicking a physical trash can in frustration. If the device determines whether each of a plurality of physical assets is required or optional based on the initial attributes of the virtual asset and the attributes of the virtual asset as modified by the action asset, then this will render the physical trash can required for executing the scene. However, if the device determines whether each of a plurality of physical assets is required or optional based on the initial attributes of the virtual asset and the attributes of the virtual asset as modified by the action asset designated as non-required (and the sixth action asset is designated as non-required), then the physical trash can remains optional for executing the scene. Therefore, in various implementations, when executing a scene, the device can selectively render action assets based on the physical environment and the physical objects present therein.

[0207] In various specific implementations, method 800 further includes receiving user input to provide a scene preview, and executing the scene in response to receiving the user input to provide a scene preview by displaying a representation of a virtual asset in association with the physical environment. For example, in Figure 2W In the middle, electronic device 110 receives and interacts with user input 299W with preview display 229, and in Figures 3F to 3I and Figures 4C to 4F In this context, scenarios are executed by displaying representations of virtual assets in relation to the physical environment.

[0208] In various specific implementations, the execution scenario includes scanning the physical environment to determine whether it contains physical objects corresponding to each physical asset required. For example, in Figure 3A In the middle, electronic device 110 scans the first physical environment, and... Figure 4AIn this process, electronic device 110 scans the second physical environment. In various specific implementations, the execution scenario further includes displaying a representation of a virtual asset associated with each physical object in response to determining that the physical environment includes physical objects corresponding to each physical asset required. For example, in... Figure 3F In this context, electronic device 110 displays a representation of virtual assets in association with a representation of physical objects in the first physical environment, and... Figure 4C In this context, electronic device 110 displays the representation of virtual assets in association with the representation of physical objects in the second physical environment.

[0209] Figure 9 This is a block diagram of an electronic device 900 according to some specific embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, electronic device 900 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more XR displays 912, one or more optional internal and / or external image sensors 914, memory 920, and one or more communication buses 904 for interconnecting these components and various other components.

[0210] In some embodiments, the one or more communication buses 904 include circuitry for communication between interconnecting system components and control system components. In some embodiments, one or more I / O devices and sensors 906 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0211] In some embodiments, one or more XR displays 912 are configured to present XR content to a user. In some embodiments, the one or more displays 912 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some embodiments, the one or more XR displays 912 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, electronic device 900 includes a single XR display. In another example, electronic device 900 includes an XR display for each of the user's eyes. In some embodiments, the one or more XR displays 912 are capable of presenting AR, MR, and / or VR content.

[0212] In various embodiments, one or more XR displays 912 are video pass-through displays that display at least a portion of the physical environment as an image captured by a scene camera. In various embodiments, one or more XR displays 912 are optical pass-through displays that are at least partially transparent and allow light emitted or reflected by the physical environment to pass through.

[0213] In some embodiments, one or more image sensors 914 are configured to acquire image data corresponding to at least a portion of a user's face (including the user's eyes) (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 914 are configured to face forward in order to acquire image data corresponding to a scene that the user would see when the electronic device 900 is not present (and may thus be referred to as a scene camera). One or more optional image sensors 914 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, etc.

[0214] Memory 920 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 920 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 920 optionally includes one or more storage devices remotely located to one or more processing units 902. Memory 920 includes a non-transitory computer-readable storage medium. In some embodiments, memory 920 or its non-transitory computer-readable storage medium stores programs, modules, and data structures, or subsets thereof, including optional operating system 930 and XR rendering module 940.

[0215] Operating system 930 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, XR presentation module 940 is configured to present XR content to a user via one or more XR displays 912. Therefore, in various implementations, XR presentation module 940 includes a data acquisition unit 942, a scene composition unit 944, an XR presentation unit 946, and a data transmission unit 948.

[0216] In some embodiments, the data acquisition unit 942 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.). Data can be acquired from one or more processing units 902 or another electronic device. Therefore, in various embodiments, the data acquisition unit 942 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0217] In some implementations, the scene-forming unit 944 is configured to provide an interface for constructing a scene to be executed in association with a physical environment. To this end, in various implementations, the scene-forming unit 944 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0218] In some implementations, the XR rendering unit 946 is configured to render XR content via one or more XR displays 912. For example, in various implementations, the XR rendering unit 946 is configured to execute a scene in association with a physical environment. To this end, in various implementations, the XR rendering unit 946 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0219] In some implementations, the data transmission unit 948 is configured to transmit data (e.g., presentation data, location data, etc.) to one or more processing units 902, memory 920, or another electronic device. Therefore, in various implementations, the data transmission unit 948 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0220] Although the data acquisition unit 942, scene composition unit 944, XR presentation unit 946 and data transmission unit 948 are shown residing on a single electronic device 900, it should be understood that in other specific implementations, any combination of the data acquisition unit 942, scene composition unit 944, XR presentation unit 946 and data transmission unit 948 may reside in a separate computing device.

[0221] also, Figure 9 This serves more as a functional description of various features that may exist in a particular specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 9 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0222] While various aspects of specific embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described embodiments can be embodied in a wide variety of forms, and any particular structure and / or function described above are merely illustrative. Based on this disclosure, those skilled in the art will understand that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Furthermore, such an apparatus and / or such a method can be implemented using other structures and / or functions besides or different from one or more aspects set forth herein.

[0223] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0224] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term “comprising” as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0225] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method for displaying content, the method comprising: Scan a first physical environment to detect a first physical object and a second physical object in the first physical environment, wherein the first physical object conforms to at least one first object criterion and the second physical object conforms to at least one second object criterion; A virtual object is displayed in association with the first physical environment, moving along a first path from the first physical object to the second physical object; Scanning a second physical environment to detect a third physical object and a fourth physical object within the second physical environment, wherein the third physical object conforms to at least one first object criterion, and the fourth physical object conforms to at least one second object criterion; and The virtual object is displayed in association with the second physical environment, moving along a second path from the third physical object to the fourth physical object, wherein the second path is different from the first path.

2. The method of claim 1, wherein the at least one first object criterion includes a criterion regarding the size or shape of the first physical object.

3. The method of claim 1, wherein the at least one first object criterion includes a criterion regarding the object type of the first physical object.

4. The method of claim 1, wherein the at least one first object criterion includes a criterion regarding the availability of the first physical object.

5. The method of claim 4, wherein the criterion for the availability of the first physical object is defined relative to the virtual object of the content.

6. The method of claim 4, wherein the criterion for the availability of the first physical object is defined relative to the fifth physical object.

7. The method of claim 6, wherein the fifth physical object is a user of the device performing the method.

8. The method of claim 1, wherein displaying the virtual object in association with the first physical environment includes displaying a composite image of the virtual object and the first physical environment.

9. The method of claim 1, wherein displaying the virtual object in association with the first physical environment includes projecting the virtual object onto a view of the first physical environment.

10. The method of claim 1, wherein displaying the virtual object in association with the first physical environment includes displaying the virtual object on a representation or view of the first physical object or the second physical object.

11. The method of claim 1, wherein displaying the virtual object in association with the first physical environment includes displaying the virtual object near a representation or view of the first physical object or the second physical object.

12. The method of claim 1, wherein displaying the virtual object in association with the first physical environment includes displaying the virtual object that interacts with a representation or view of the first physical object or the second physical object.

13. The method of claim 1, wherein the length of the second path is different from the length of the first path.

14. The method of claim 1, wherein the variance of the second path measured relative to a straight line is different from the variance of the first path measured relative to a straight line.

15. The method of claim 1, wherein at least one of the first path or the second path is determined to avoid another object, wherein the other object is a virtual object or a physical object in a corresponding physical environment.

16. A device for displaying content, the device comprising: monitor; Non-transitory memory; as well as One or more processors, said one or more processors being used for: Scan a first physical environment to detect a first physical object and a second physical object in the first physical environment, wherein the first physical object conforms to at least one first object criterion and the second physical object conforms to at least one second object criterion; A virtual object is displayed in association with the first physical environment, moving along a first path from the first physical object to the second physical object; Scanning a second physical environment to detect a third physical object and a fourth physical object within the second physical environment, wherein the third physical object conforms to at least one first object criterion, and the fourth physical object conforms to at least one second object criterion; and The virtual object is displayed in association with the second physical environment, moving along a second path from the third physical object to the fourth physical object, wherein the second path is different from the first path.

17. The device of claim 16, wherein the at least one first object criterion includes a criterion regarding the size or shape of the first physical object.

18. The device of claim 16, wherein the at least one first object criterion includes a criterion regarding the object type of the first physical object.

19. The device of claim 16, wherein the at least one first object criterion includes a criterion regarding the availability of the first physical object.

20. A non-transitory memory storing one or more programs, said one or more programs causing the device to: Scan a first physical environment to detect a first physical object and a second physical object in the first physical environment, wherein the first physical object conforms to at least one first object criterion and the second physical object conforms to at least one second object criterion; A virtual object is displayed in association with the first physical environment, moving along a first path from the first physical object to the second physical object; Scanning a second physical environment to detect a third physical object and a fourth physical object within the second physical environment, wherein the third physical object conforms to at least one first object criterion, and the fourth physical object conforms to at least one second object criterion; and The virtual object is displayed in association with the second physical environment, moving along a second path from the third physical object to the fourth physical object, wherein the second path is different from the first path.

Images