Position-based virtual component modality in 3D content

Abstract

The present disclosure relates to position-based virtual element modalities in three-dimensional content. Various specific implementations disclosed herein include devices, systems, and methods that enable a device to provide views of virtual elements and a physical environment, wherein the presentation of the virtual elements is based on positioning relative to the physical environment. In one example, the device is configured to detect changes in the positioning of the virtual elements, for example, when adding, moving virtual elements, or changing the physical environment around the virtual elements. The position of the virtual element in the physical environment is used to detect the properties of the physical environment on which the presentation of the virtual element depends. Therefore, the device is further configured to detect the properties of the physical environment (e.g., surface, table, mid-air, etc.) based on the placement of the virtual element, and present the virtual element based on the detected properties.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese patent application with the application date of September 2, 2019, application number 201910823345.X, and name “Position-based virtual element modality in dimensional content”. Technical Field

[0003] The present disclosure relates generally to display of computer generated reality (CGR) environments, and more particularly to systems, methods, and apparatus for displaying virtual elements in three-dimensional (3D) content, wherein the virtual elements in the content change modality. Background Art

[0004] Various devices are used to provide a computer generated reality (CGR) environment to a user. The 3D content of such an experience may be experienced by a user using a mobile device, a head mounted device ("HMD"), or other device that presents the visual or audio features of the content. The experience may be, but need not be, immersive, e.g., providing most or all of the visual or audio content experienced by the user. The content may include pass-through video (e.g., where real-world content is captured by a camera and displayed on a display with additional content) or optical see-through (e.g., where real-world content is viewed directly or through glass and supplemented with displayed additional content).

[0005] Some devices and software programs facilitate displaying 3D content including virtual objects / elements. However, these devices and software programs do not provide sufficient features to adapt or customize the modality of such virtual objects / elements. Summary of the invention

[0006] Various embodiments disclosed herein include devices, systems, and methods that enable a device to present a view of a virtual element and a physical environment, wherein the presentation of the virtual element is based on properties of the physical environment at the location of the virtual element in the view. In one example, the device is configured to detect a change in the view, such as when a virtual element is placed at a location in the view relative to the physical environment and the location of the virtual element in the view is used to detect properties of the physical environment. Therefore, the device is further configured to detect properties of the physical environment (e.g., a surface, a table, etc.) based on the placement of the virtual element, and present the virtual element based on the detected properties.

[0007] Some specific implementations as shown in the above examples and elsewhere herein therefore enable the presentation of virtual elements based on the position of the virtual elements relative to the properties of the physical environment. Creators of virtual elements (including creators of applications that include virtual elements) can provide views of virtual elements with appearance, functionality, and interactive features that are adapted to the representation of the physical environment in which they are used. The end-user experience of using views that include such adaptive virtual elements can be significantly better. The end-user can see views of virtual elements presented based on the position of the virtual elements relative to the physical environment, and can have great freedom to move the virtual elements to experience the virtual elements in their different presentations.

[0008] In some implementations, a non-transitory computer-readable storage medium stores computer-executable instructions to perform or cause the performance of any of the methods described herein. According to some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. In some implementations, the methods described herein are performed at a device having one or more processors and a computer-readable storage medium, such as a desktop, laptop computer, tablet computer, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary embodiments, some of which are illustrated in the accompanying drawings.

[0010] Figure 1 A 3D environment provided on a device is shown, wherein the device displays a first virtual element based on a positioning of the first virtual element relative to the environment, according to some implementations.

[0011] Figure 2 According to some specific implementations, Figure 1 A 3D environment is provided, wherein the device is a head-mounted device.

[0012] Figure 3 A 3D environment provided on a device is shown, wherein the device displays a second virtual element based on a positioning of the second virtual element relative to the environment, according to some implementations.

[0013] Figure 4 According to some specific implementations, Figure 3 A 3D environment is provided, wherein the device is a head-mounted device.

[0014] Figure 5 It is implemented according to some specific Figure 1A block diagram of a device that includes providing virtual elements to an operating system for inclusion in a 3D environment by multiple applications.

[0015] Figure 6 is a block diagram illustrating device components of an example device according to some implementations.

[0016] Figure 7 A flowchart representation of a method for presenting a view including a virtual element, wherein the presentation of the virtual element is based on a positioning relative to a physical environment shown in the view.

[0017] Figure 8 A flowchart representation of a method for presenting 3D content on a display, wherein a virtual element changes modality based on a position of the virtual element relative to an environment shown in the 3D content.

[0018] Fig. 9 is a flowchart representation of a method for presenting content on a display in which a virtual element changes modality based on a position of the virtual element relative to an environment shown in the content.

[0019] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0020] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0021] See also Figure 1 and Figure 2, an example operating environment 5 for implementing various aspects of the present disclosure is shown. Generally speaking, the operating environment 5 represents a device 10 involving the display of 3D content, including but not limited to a computer generated reality (CGR) environment. A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that conforms to at least one physical law. For example, a CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of features of virtual objects in the CGR environment can be performed in response to a representation of physical movement (e.g., a voice command).

[0022] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0023] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0024] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0025] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0026] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. As used herein, an HMD in which at least some light of the physical environment can pass through a transparent or translucent display is referred to as an "optical see-through" HMD.

[0027] Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on the opaque display. The person uses the system to indirectly view the physical environment via the image or video of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display.

[0028] Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0029] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0030] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0031] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment.

[0032] The head-mounted system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project virtual objects into a physical environment, such as a hologram or onto a physical surface.

[0033] In some implementations, the device 10 is a handheld electronic device (e.g., a smartphone or tablet) configured to present CGR content 20 to a user. In some implementations, the device 10 is a head mounted device (HMD) worn by a user. Such an HMD may surround the user's field of view. In some implementations, the HMD is worn in a manner such that one or more screens are positioned to display CGR content in the user's field of view. In some implementations, two devices (e.g., an HMD and a controller device that wirelessly communicates with the HMD) communicate with each other to provide CGR content (e.g., the HMD uses cameras and sensors to compile information about a real-world scene 5 and has one or more displays to display CGR content, and the controller device processes the camera / sensor data and other data to provide CGR content to the HMD).

[0034] exist Figure 1 and Figure 2 In the embodiment of the present invention, 3D content (e.g., CGR environment 15) combines a depiction of a real-world scene 20 (e.g., via a locally captured image of the real-world scene 20) with a depiction of virtual elements. Various specific implementations enable the device 10 to view or edit the CGR environment in different viewing modes (e.g., monocularly, stereoscopically, etc.). The CGR environment can be a pass-through video (e.g., where the real-world content is captured by a camera and displayed on a display) or an optical see-through (e.g., viewing the real-world content directly or through glass). For example, a CGR system can provide a user with a pass-through video on a display of a consumer's mobile phone by integrating the rendered 3D graphics into a real-time video stream captured by an on-board camera. For another example, a CGR system can provide a user with optical see-through CGR by superimposing the rendered 3D graphics into a wearable see-through head-mounted display ("HMD"), thereby electronically enhancing the user's optical view of the real world with the help of the superimposed 3D model.

[0035] The modality of a virtual element in content is based on the position of the virtual element relative to the content. In some implementations, the modality of the virtual element is determined when the virtual element is first inserted into the content, for example, based on the initial position of the virtual element. In some implementations, the modality of the virtual element is determined and changed when the virtual element moves in the content. In some implementations, changes made to the content affect the modality of the virtual element. For example, a user of device 10 can move or change the properties of the content associated with the virtual element, for example, eliminating the virtual surface on which the virtual element is "resting" so that the virtual element remains in open 3D space (e.g., "in mid-air").

[0036] exist Figure 1 and Figure 2In the example of , the device 10 presents a CGR environment 15. The CGR environment 15 observed on the device 10 depicts a real world scene 20 with real objects, such as a coffee table 25, a wall 30, a floor 35, and an open 3D space 40. One or more virtual elements are automatically or manually included in the CGR environment 15. In one example, the virtual element is automatically selected based on the real objects in the CGR environment 15. As another example, the user provides input to select one or more virtual elements to be included in the CGR environment 15. In some specific implementations, the CGR environment 15 includes virtual elements from a single source, such as virtual elements from a single software application. In other specific implementations, the CGR environment 15 includes an arrangement of virtual elements from multiple sources. For example, the CGR environment 15 can provide a CGR "desktop" for real or virtual elements, including multiple virtual elements from multiple separately executed software applications. For example, such an environment can display a depiction of a table of a real environment, which has one or more virtual elements positioned on the top surface of the table, on the wall, or on the floor, etc., and such virtual elements may have been provided by a single, separately executed application.

[0037] exist Figure 1 and Figure 2 In the example of , the CGR environment 15 illustrates flat surfaces such as coffee table 25, wall 30, floor 35, but in some implementations, the CGR environment 15 may alternatively illustrate non-flat surfaces (e.g., surfaces with complex geometry), including real-world non-flat surfaces and virtual or synthetic non-flat surfaces. In addition, some implementations may use planes, triangular mesh elements, or volume representations to capture or represent the real-world scene 20, and generate the CGR environment 15. In some implementations, a reference computer-aided drawing ("CAD") model may be used to identify or represent an item, such as coffee table 25.

[0038] In some implementations, real items of the real world scene 20 are identified based on transitions or differences in texture (e.g., color) within the real world scene 20 or the CGR environment 15. For example, a real item (e.g., coffee table 25) may include a black area, and an adjacent area of ​​the real world scene 20 or the CGR environment 15 may be white. Therefore, the boundary of the real item can be determined by identifying a texture transition area between light and dark colors. In some implementations, real items can be identified based on a structure from motion ("SfM") method, a simultaneous localization and mapping ("SLAM") method, a computer vision passive stereo method, a computer vision active stereo method including structured light, a photometric stereo method, a shape from shading method, a shape from defocus method, a geometric model fitting method, a machine learning method, or a monocular depth estimation method.

[0039] The devices, systems, and methods disclosed herein enable displaying virtual elements based on the relative position of the virtual elements to physical or virtual attributes in the CGR environment 15. When a user of the device 10 places or repositions the virtual element within the CGR environment 15 or changes the CGR environment 15 around the virtual element, the modality of the virtual element changes or adapts. Specifically, the modality of the virtual element is determined based on the position of the virtual element relative to one or more attributes of nearby CGR content. Such modalities may include appearance, functionality, or interactive aspects of the virtual element.

[0040] In addition, the CGR environment 15 may include coplanar virtual elements, such as a virtual monitor displayed on a depiction of a table in the real environment, having one or more virtual elements located on a virtual screen of the virtual monitor. In such implementations, a first coplanar virtual element (e.g., a virtual monitor) may act as a receiver for placing one or more secondary coplanar virtual elements (e.g., a weather application) and may define boundaries for placing the secondary coplanar virtual elements. In other implementations, the first coplanar virtual element may provide a modal context for the secondary coplanar virtual element, i.e., the modality of the secondary coplanar virtual element may change or adapt based on the first coplanar virtual element.

[0041] A virtual element (e.g., a weather object) is included in the CGR environment 15 in different modes 45, 50, 55, depending on the different locations of the virtual element in the CGR environment 15. To illustrate this, Figure 1 In the CGR content, multiple instances of a virtual element are depicted in different modalities at different locations. These instances of the virtual element depict the modality of the virtual element at different locations, for example, because the virtual element may be positioned at different points in time during the use of the CGR content. Therefore, the different modalities 45, 50, 55 are not necessarily displayed at the same time. If multiple different instances of the virtual element are included in the CGR content at the same time, the different modalities 45, 50, 55 will only be displayed at the same time, which is possible in some specific implementations.

[0042] Figure 1 It shows how the modality of a virtual element depends on its position relative to one or more attributes of the CGR environment 15 (e.g., on a surface, a vertical surface, a horizontal surface, a wall, a floor, a ceiling, a table, in mid-air, etc.). In this example, the modality of the virtual element depends on whether the virtual element is located on the coffee table 25, on the wall 30, on the floor 35, or in open 3D space 40.

[0043] The modality (e.g., appearance, functionality, and interaction features) of a virtual element may be configured by a virtual element creator, for example, a virtual element creator may create a different modality state definition for each of a plurality of position states (e.g., on a horizontal surface, on a vertical surface, in mid-air, etc.) associated with a plurality of CGR content attributes (e.g., surface, horizontal surface, vertical surface, wall, floor, table, ceiling, etc.).

[0044] Figure 1 A virtual element displayed in a vertical surface mode 45 is shown. Based on the virtual element being positioned on the wall 30 and thus associated with a vertical surface attribute, the virtual element is displayed in a vertical surface mode 45 (e.g., as a weather sign). In this example, the vertical surface mode 45 of the virtual element displays a 2D image or symbol representing the weather and the high and low predicted daily temperatures. The functionality and interactivity of the virtual element may depend on its modality. For example, the vertical surface mode 45 of the virtual element may provide modality-specific functionality, such as the display of a Doppler radar map. As another example, the vertical surface mode 45 of the virtual element may provide modality-specific user interactivity, such as a user-controllable animation. For example, the vertical surface mode 45 of the virtual element may be configured to provide a weather animation when the virtual element or an area near the virtual element is clicked or touched on the touch screen of the device 10. In another example, the vertical surface mode 45 of the virtual element may be configured to change its appearance, such as the information it displays, when the virtual element is hovered, touched, or clicked. As another example, the vertical surface mode 45 of the virtual element may be configured to expand its appearance based on the user's interaction.

[0045] Figure 1 Also shown is a virtual element displayed in an open space modal 50. In this example, the appearance, functionality, and interactivity of the open space modal 50 of the virtual element differ from the appearance, functionality, and interactivity of the vertical surface modal 45. The open space modal 50 has a 3D appearance as opposed to the 2D appearance of the vertical surface modal 45. The open space modal 50 provides a 3D representation of current weather or forecasted weather conditions in the user's current geographic location, for example, showing a floating sun, rain clouds, a tornado, etc.

[0046] Figure 1Also shown is a virtual element displayed in a horizontal surface mode 55. In this example, the appearance, functionality, and interactivity of the horizontal surface mode 55 of the virtual element are different from the appearance, functionality, and interactivity of the vertical surface mode 45 and the open space mode 50. The virtual element in this example is positioned relative to the horizontal flat surface of the depiction of the real-world coffee table 25. Based on this position, the virtual element is displayed in a horizontal surface mode 55, for example, as a decorative snow globe, where the current or predicted weather is displayed in 3D within the sphere. In addition, the exemplary horizontal surface mode 55 presents the weather using time-varying content (e.g., snow or rain in the sphere) according to the predicted weather. The interactivity of the horizontal surface mode 55 (e.g., snow globe) can include starting or stopping the animation of the sphere when the user clicks or touches the sphere via the interface of the device 10. As adjusted by those of ordinary skill in the art, the appearance, functionality, and interactivity of the virtual element can include any number of features, for example, displaying additional forecast information when the snow globe is clicked or touched by the device 10.

[0047] Figure 2 Shown by Figure 1 An environment is provided, wherein the device is an HMD configured to be worn on the head of a user 65. Such an HMD may surround the field of view of the user 65. The HMD may include one or more screens or other displays configured to display a CGR environment 15. In some implementations, the HMD includes one or more screens or other displays to display virtual elements with real-world content in the field of view of the user 65. In some implementations, the HMD is worn in a manner in which one or more screens are positioned to display a CGR environment 15 with real-world content of a real-world scene 20 in the field of view of the user 65. In some implementations, the device 10 providing the CGR environment 15 is a chamber, housing, or room that is configured to present the CGR environment 15, wherein the user 65 does not wear or hold the device 10.

[0048] Figure 3 and Figure 4 An exemplary operating environment 70 is shown in which CGR content includes virtual elements that provide time information (such as the current time). In the exemplary operating environment 70, the real world scene 20 includes a plurality of real world objects and associated attributes, such as walls 30, floor 35, table 75, and open 3D space 40. Figure 1 and Figure 2 As in the example, Figure 3 and Figure 4The CGR environment 15 depicted in the example of includes virtual elements having different modalities when placed at different positions relative to the attributes of the CGR content. The modality (e.g., appearance, functionality, and interaction features) of the virtual element can be configured by the virtual element creator, for example, the virtual element creator can create different modality state definitions for each of multiple position states (e.g., on a horizontal surface, on a vertical surface, in mid-air, etc.) associated with multiple CGR content attributes (e.g., surface, horizontal surface, vertical surface, wall, floor, table, ceiling, etc.).

[0049] Figure 3 The virtual element is shown displayed in a vertical surface modality 45. In the exemplary operating environment 70, the user can position the virtual element near a wall 30 in the CGR environment 15. When positioned near the wall 30, the virtual element has a vertical surface modality 80. In the vertical surface modality 80, the virtual element has the appearance, functionality, and interactivity of a wall clock.

[0050] The virtual element may be configured to be customized by the end user in one or more of its modalities. For example, when the virtual element is placed near a wall 30, the virtual element may be displayed in a vertical surface modality 80, and the user may be presented with modality-specific display options, such as whether the user may choose to display the time in analog or digital format in that particular vertical surface modality 80.

[0051] Figure 3 Also shown is a virtual element displayed in a floor mode 85. As an alternative to placing the virtual element along the wall 30, the user can place the virtual element near the floor 35. In the exemplary illustration, the floor form 85 of the virtual element is a standing grandfather clock. The floor mode 85 of the virtual element can provide modality-specific functions and interactive features. For example, the floor mode 85 can include a pendulum, a cuckoo bird that appears from the face of the grandfather clock when the user provides user interaction of the device 10 (e.g., voice command, gesture, gaze, etc.).

[0052] Figure 3 Also shown is a virtual element displayed in a horizontal surface mode 55. For example, if a user positions the virtual element near a horizontal surface of a table 75 in the CGR environment 15, the virtual element may be displayed in a horizontal surface mode 90. In the horizontal surface mode 90, the virtual element has the appearance of a clock radio. In this example, the horizontal surface mode 90 provides the user with the ability to interact through interactive features 105. Specifically, the horizontal surface mode 90 of the virtual element (e.g., a clock radio) includes a knob or button to control a radio feature of the virtual element by tuning the radio or adjusting the volume of the clock radio.

[0053] like Figure 4As shown, device 10 may be an HMD configured to be worn on the head of user 65 .

[0054] In some implementations, CGR content, including descriptions of real-world objects and virtual elements, is provided by a single application. In other implementations, an application such as an operating system provides CGR content that includes a shared virtual space in which content from separate sources is combined. In one example, an operating system provides a shared virtual space that includes virtual elements provided by one or more separate applications. Figure 5 Examples of such specific implementations are provided.

[0055] Figure 5 is a block diagram of a device 10 including a plurality of applications 115a-n that provide virtual elements, such as virtual element 120, to an operating system 130 for inclusion in a CGR environment 15, according to some implementations. In this example, application 115a provides virtual element 120 that includes information regarding placement and modality of virtual element 120. Such modality information may include information regarding how virtual element 120 should appear, function, and interact in different modes. The modality information may include one or more 2D or 3D models of virtual element 120. In some implementations, the 3D model of virtual element 120 identifies a collection of points in 3D space, connected by various geometric entities, such as triangles, lines, curves, surfaces, etc. In Figure 5 , the virtual element 120 is added to the CGR environment 15 and is shown on top of the table 125 in the view 200 of the CGR environment 15. The virtual element 120 is provided to the operating system 130 by the application 115a and includes modality information indicating that the virtual element 120 will change modality based on placement.

[0056] Figure 61 is a block diagram illustrating device components according to some implementations of device 10. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 10 includes one or more processing units 142 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 146, one or more communication interfaces 148 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 150, one or more displays 152, one or more internally or externally facing image sensors 154, a memory 160, and one or more communication buses 144 for interconnecting these and various other components.

[0057] In some implementations, one or more communication buses 144 include circuits that interconnect and control communication between system components. In some implementations, one or more I / O devices and sensors 146 include at least one of a touch screen, soft keys, keyboard, virtual keyboard, buttons, knobs, joysticks, switches, dials, inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.). In some implementations, movement, rotation, or position of device 10 detected by one or more I / O devices and sensors 146 provides input to device 10.

[0058] In some implementations, one or more displays 152 are configured to present a user interface. In some implementations, one or more displays 152 correspond to holographic, digital light processing ("DLP"), liquid crystal display ("LCD"), liquid crystal on silicon ("LCoS"), organic light emitting field effect transistor ("OLET"), organic light emitting diode ("OLED"), surface conduction electron emitter display ("SED"), field emission display ("FED"), quantum dot light emitting diode ("QD-LED"), microelectromechanical system ("MEMS"), or similar display types. In some implementations, one or more displays 152 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, device 10 includes a single display. As another example, device 10 includes a display for each eye. In some implementations, one or more displays 152 are capable of presenting CGR content.

[0059] In some implementations, the one or more image sensor systems 154 are configured to obtain image data corresponding to at least a portion of a local scene of the device 10. The one or more image sensor systems 154 can include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (“CMOS”) image sensor or a charge coupled device (“CCD”) image sensor), a monochrome camera, an IR camera, or an event-based camera, etc. In various implementations, the one or more image sensor systems 154 also include an illumination source that emits light, such as a flash.

[0060] Memory 160 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 160 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 160 optionally includes one or more storage devices located away from one or more processing units 142. Memory 160 includes non-transitory computer-readable storage media. In some specific implementations, memory 160 or non-transitory computer-readable storage media of memory 160 stores the following programs, modules and data structures, or a subset thereof, including an optional operating system 130 and one or more application programs 115a-n. Operating system 130 includes processes for handling various basic system services and for performing hardware-related tasks.

[0061] In some implementations, the operating system 130 includes a CGR content creation unit 132 for generating CGR content (e.g., a view of a CGR environment) using data from multiple sources (e.g., real world data from a camera and virtual elements from one or more applications 138 (e.g., 115a-n)). The operating system 130 also includes a CGR modality selector unit 134 used by the one or more applications 138 to determine a modality. The CGR modality selector unit 134 is configured to identify a modality based on a position of a virtual element relative to the CGR content. The operating system 130 also includes an application interface unit 136 for receiving information from the applications 138, such as virtual element information, and exchanging information about placement of virtual elements and modality selection with the applications 115a-n, as interpreted by the CGR content creation unit 132 and the CGR modality selector unit 134. In some implementations, the functionality for adjusting the modality based on position is built into the applications 115a-n instead of the operating system 130. In some implementations, a single application, module, or other functional unit defines the virtual element and includes functionality for creating CGR content and including the virtual element in the CGR content according to a modality selected based on the location of the virtual element in the CGR content.

[0062] Figure 6 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 6 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0063] Figure 7 700 is a flowchart representation of a method for presenting a view including a virtual element, wherein the presentation of the virtual element is based on a position relative to a physical environment depicted in the view. In some implementations, the method 700 is performed by a device (e.g., Figures 1 to 6 The method 700 can be performed on a mobile device, an HMD, a desktop computer, a laptop, or a server device. In some specific implementations, the method 700 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 700 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0064] At block 710, the device uses an image sensor to acquire image data representing a physical environment. The image may be captured by a local or remote recorder / camera, and in some implementations, the recorder / camera may be a standalone device, may be incorporated into a mobile phone or video game controller, or may be included in an HMD. Light reflected by a real-world scene may be captured by one or more camera images. In some implementations, the method may combine one or more camera images of a real-world scene to form a single image or 3D model of the environment.

[0065] At block 720, method 700 provides a view depicting one or more virtual elements and a physical environment on a display of the device. The view may include pass-through video (e.g., where real-world content is captured by a camera and displayed on the display along with the virtual elements) or optical see-through content (e.g., where real-world content is viewed directly or through glass and supplemented with displayed images of virtual elements).

[0066] At box 730, method 700 detects a change in the view. The change may include placing a virtual element in the view at a position relative to the physical environment depicted in the view. In addition, the device may receive the change by inputting the virtual element at the position. For example, a user of the device may provide keyboard input, mouse input, touch input, voice input, or other input to change the properties or characteristics of the view including the virtual element. For example, the user may change the size, color, texture, direction, etc. of the virtual element or 3D model, add a 3D model or a part of the 3D model, delete a 3D model or a part of the 3D model, etc. In some specific implementations, the changes are integrated or merged to improve system efficiency. For example, the change can involve detecting multiple changes between the initial state and the final state of the 3D model, and identifying the difference between the initial state and the final state of the 3D model. For example, if the virtual element first moves 10 units to the left and then moves 5 units to the right, a single change of the virtual element moving 5 units to the left can be identified.

[0067] At box 740, method 700 detects properties in the physical environment depicted in the view based on the position of the virtual element in the view. In some specific implementations, detecting the property includes detecting a surface, an adjacent surface or element, or the type of surface on which the virtual element is located in the physical environment depicted in the view, for example, determining whether the virtual element is located in free space or on a floor, table, wall, or ceiling. In some specific implementations, detecting the property includes using a model such as a neural network or other machine learning model to identify objects and other properties of the physical environment. In some specific implementations, the machine learning model is trained to identify walls, ceilings, floors, tables, ramps, flat planes, curved surfaces, round surfaces, textured surfaces, surfaces of specific colors or textures, etc. In some specific implementations, detecting the property includes using a classifier to identify a classification of a portion of the physical environment depicted in the view or to identify the amount of the portion on which the virtual element can be positioned, for example, detecting that the virtual element is located in mid-air (e.g., the virtual element is not positioned on a surface). In addition, in some specific implementations, method 700 determines a 3D model of the physical environment having a coordinate system and identifies the property based on the position of the virtual element relative to the property in the 3D model. In some implementations, method 700 determines whether to allow the element to be displayed at the location based on the attribute.

[0068] At block 750, the method 700 presents the virtual element based on the attribute. In some implementations, the method 700 selects a modality (eg, appearance, functionality, or interactivity) of the virtual element based on the attribute. Figures 1 to 4 Examples are provided for adjusting a modality (eg, appearance, functionality, or interactivity) of a virtual element based on a position of the virtual element relative to an environmental attribute.

[0069] In a specific implementation, method 700 updates a view on a display of a device so that the view includes a virtual element according to a selected modality. Method 700 may send a notification to a separate application that provides the virtual element. The notification may identify the properties of the view corresponding to the location of the virtual element, and the application may respond with specific information that updates the appearance, functionality, or interactivity of the virtual element based on the modality.

[0070] Figure 8 8 is a flowchart representation of a method 800 for presenting 3D content on a display, wherein the 3D content (e.g., a virtual element) changes modality based on a position of the 3D content relative to a CGR environment. In some implementations, the method 800 is performed by a device (e.g., Figures 1 to 6The method 800 can be performed on a mobile device, an HMD, a desktop computer, a laptop, or a server device. In some specific implementations, the method 800 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 800 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0071] At block 810, method 800 presents a view of a CGR environment on a display of a device. The CGR environment may include one or more 3D content items (e.g., physical or virtual elements). The 3D content may include pass-through video (e.g., where real-world content is captured by a camera and displayed on a display along with virtual elements) or optical see-through content (e.g., where real-world content is viewed directly or through glass and supplemented with displayed images of virtual elements).

[0072] At block 820, method 800 detects a change associated with the 3D content. The change may include positioning the virtual element at a location relative to the rest of the environment. In addition, the device may receive the change by inputting to position the virtual element at the location. For example, a user of the device may provide hand detection input, keyboard input, mouse input, touch input, voice input, or other input to change the properties of the 3D content, including the virtual element. For example, the user may change the size, color, texture, orientation, etc. of the virtual element or 3D model, add a 3D model or a portion of a 3D model, delete a 3D model or a portion of a 3D model, etc.

[0073] At box 830, method 800 detects attributes based on the location of the 3D content. In some specific implementations, detecting attributes includes detecting adjacent surfaces or elements, or the type of surface on which the virtual element is positioned in the environment depicted in the 3D content, for example, determining whether the virtual element is located in free space or on a floor, table, wall, ceiling, or other virtual element. In some specific implementations, detecting attributes includes using a classifier to identify a classification of a portion of the environment depicted in the 3D scene and / or identifying the amount of available portion on which the virtual element can be located, for example, detecting that the virtual element is located in mid-air or within the boundary of another coplanar virtual element. In addition, in some specific implementations, method 800 determines a 3D model of a CGR environment with a coordinate system and identifies attributes based on the location of the virtual element relative to the attribute in the CGR environment. In some specific implementations, method 800 determines whether to allow the element to be displayed at the location based on the attribute.

[0074] Attributes are used to describe components of a CGR environment, including real-world and / or virtual components, as well as virtual objects. In addition, attributes can represent intra-class variations (e.g., an ocean may or may not have waves) as well as inter-class relationships (e.g., a pool and an ocean may both have waves). Examples of attributes may include surface properties (e.g., shiny), functional or affordance representations (e.g., bounce, shimmer), spatial envelope properties (e.g., closed, symmetric), material properties (e.g., aluminum, wood), or any other attribute-based representation of 3D content.

[0075] At block 840, the method 800 selects a modality of the virtual element based on the attribute. In some implementations, the method 800 selects an appearance, functionality, and / or interactivity of the virtual element based on the attribute.

[0076] At block 850, method 800 updates the 3D content on a display of the device. The 3D content may be updated to include the virtual element according to the selected modality. Method 800 may send a notification to a separate application providing the virtual element. The notification may identify a property corresponding to the location of the virtual element, and the application may respond with modality-specific information that updates the appearance, functionality, or interactivity of the virtual element.

[0077] Fig. 9 900 is a flowchart representation of a method 900 for presenting a CGR environment on a display, wherein a virtual element changes modality based on a position of the virtual element relative to the rest of the CGR environment. In some implementations, the method 900 is performed by a device (e.g., Figures 1 to 6 The method 900 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. In some specific implementations, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 900 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0078] At block 910, method 900 presents a view of a CGR environment on a display of a device. The CGR environment may include pass-through video (e.g., where real-world content is captured by a camera and displayed on a display along with virtual elements) or optical see-through content (e.g., where real-world content is viewed directly or through glass and supplemented with displayed images of virtual elements).

[0079] At box 920, method 900 detects changes associated with the CGR environment. The change may include positioning virtual elements and / or physical objects at positions relative to the physical environment and / or virtual environment / elements depicted in the CGR environment. In addition, the device may receive the change by inputting to position the virtual element at the position. For example, a user of the device may provide hand detection input, keyboard input, mouse input, touch input, voice input, or other input to change the properties of the CGR environment including the virtual element. For example, the user may change the size, color, texture, orientation, etc. of the virtual element or 3D model, add a 3D model or a part of a 3D model, delete a 3D model or a part of a 3D model, etc. For another example, the user may change the position of a physical object in the real world environment, which in turn may change the relative position of the virtual element depicted in the CGR environment relative to the physical object.

[0080] At box 930, the method 900 detects a surface depicted in the CGR environment (or the lack of a surface in the case of a standalone or floating item) based on the location of the virtual element. In some implementations, detecting the surface includes detecting adjacent surfaces or elements, or the type of surface on which the virtual element is positioned in the environment depicted in the CGR content, for example, determining whether the virtual element is located in free space or on a floor, table, wall, ceiling, or other virtual element. In some implementations, detecting the surface includes using a classifier to identify a classification of a portion of the environment depicted in the CGR environment and / or identifying an amount of available portion on which the virtual element can be located, for example, detecting that the virtual element is located in mid-air or within the boundary of another coplanar virtual element. In addition, in some implementations, the method 900 determines a 3D model of the CGR environment having a coordinate system and identifies the surface based on the location of the virtual element relative to the location of the surface in the 3D model.

[0081] In some implementations, the modality of a virtual object is based on the position of the virtual object relative to one or more other virtual objects. For example, the modality of a virtual object may depend on another virtual object in which it is located (e.g., whether the virtual object is on a virtual floor or a virtual wall, on a virtual table or in mid-air, etc.).

[0082] In some embodiments, a method provides a view of an environment depicting multiple virtual objects or other elements. The method detects changes in the view. The change may include automatically or based on user input placing a virtual element in the view relative to one or more other virtual elements depicted in the view. In addition, the device may receive the change by inputting to position the virtual element at the position. For example, a user of the device may provide keyboard input, mouse input, touch input, voice input, or other input to change the properties or characteristics of the view including the virtual element. For example, the user may change the size, color, texture, orientation, etc. of the virtual element or the 3D model of the virtual element, add a 3D model or a portion of a 3D model, delete a 3D model or a portion of a 3D model, etc.

[0083] The method detects the properties of one or more other virtual elements depicted in the view based on the position of the virtual element in the view. In some specific implementations, detecting the property includes detecting a virtual surface, an adjacent virtual surface or a virtual element, or the type of virtual surface where the virtual element is located, for example, determining whether the virtual element is located on a virtual floor, a virtual desktop, a virtual wall, or a virtual ceiling. In some specific implementations, the method determines whether to allow the virtual element to be displayed at the position based on the property. The method selects or changes the modality of the virtual element based on the properties of one or more other virtual elements. In some specific implementations, the method selects the appearance, function, or interactivity of the virtual element based on the property, and updates the view to include the virtual element according to the selected modality. The method can send a notification to a separate application that provides the virtual element. The notification can identify the properties of the view corresponding to the position of the virtual element, and the application can respond with modality-specific information that updates the appearance, function, or interactivity of the virtual element.

[0084] Examples of CGR environment objects that may include surfaces include, but are not limited to, another virtual element, a table, a floor, a wall, a desk, a book, a body of water, a mountain, a field, a vehicle, a counter, a human face, a human hand, a human hair, another human body part, an entire human body, an animal or other organism, clothing, a piece of paper, a magazine, a book, a car, a machine, or other man-made objects with a flat surface, and any other real-world or virtual object or scene that may present a flat surface over time. In addition, CGR content may include any number of flat surfaces. In addition, the term "surface" is not intended to limit the surface of the present disclosure to a specific object or part of the CGR environment. One or more surfaces may be identified in the CGR environment, and therefore, in some specific implementations, a single surface may be identified, and in other specific implementations, all surfaces in the CGR environment may be identified. In addition, texture information may be associated with a surface, such as color or appearance. Information about a surface or multiple surfaces in a CGR environment may be used for a variety of purposes, including but not limited to purposes of rendering involving generating or synthesizing light sources, shadows, or otherwise interpreting virtual elements.

[0085] At block 940, the method 900 selects a modality of the virtual element based on the surface. In some implementations, the method 900 selects an appearance, functionality, and / or interactivity of the virtual element based on the surface.

[0086] At block 950, method 900 updates the CGR environment on a display of the device so that the CGR environment includes the virtual element according to the selected modality. Method 900 may send a notification to a separate application that provides the virtual element. The notification may identify the properties of the CGR environment corresponding to the location of the virtual element, and the application may respond with modality-specific information that updates the appearance, functionality, or interactivity of the virtual element.

[0087] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0088] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0089] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the disclosed subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0090] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0091] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0092] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0093] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts or their groupings.

[0094] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0095] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At a device having a processor and a computer readable medium: providing a view, the view comprising: a plurality of virtual elements from a plurality of individual separately executed applications; and a representation of a physical environment; wherein the plurality of virtual elements each have a set of interactive features; detecting a change in the view, the change comprising placing a virtual element of a plurality of the virtual elements at a changed position in the view relative to the physical environment; detecting a property of the physical environment at the changed location in the view; changing the set of interaction features of the virtual element having the changed position to a different set of interaction features based on the detected attributes; Based on the different set of interactive features and the changed position, changing at least one of a size, shape, design, and color attribute of the virtual element at the changed position; and An updated view of the plurality of virtual elements and the physical environment is presented.

2. The method according to claim 1, wherein: The device includes an operating system including an application interface for exchanging information with the application regarding placement of the virtual elements and selection of interactive features. 3 . The method of claim 1 , further comprising sending a notification to a first application providing a first virtual component, wherein the notification identifies the detected attribute.

4. The method according to claim 3, further comprising: In response to receiving the notification, information is sent by the first application, the information updating a first set of interaction characteristics of the first virtual element to a second set of interaction characteristics different from the first set of interaction characteristics based on the detected attribute.

5. The method of any preceding claim, wherein providing the view comprises: acquiring an image of the physical environment using an image sensor; generating a representation of the physical environment using the acquired image; as well as The plurality of virtual elements are presented with the representation of the physical environment.

6. The method of claim 1 , wherein providing the view comprises: using an image sensor to identify features in the physical environment; as well as In response to identifying the features, the plurality of virtual elements are presented on the optical see-through display.

7. The method of any one of the preceding claims, wherein changing the set of interaction features of the virtual element having the changed position to a different set of interaction features based on the detected properties comprises: updating the first virtual element to have a first visual appearance based on determining that the attribute is a first attribute; as well as Based on determining that the attribute is a second attribute, the first virtual element is updated to have a second visual appearance, wherein the first visual appearance is different from the second visual appearance.

8. The method of any one of the preceding claims, wherein changing the interaction feature set of the virtual element having the changed position to a different interaction feature set based on the detected attribute comprises: According to determining that the attribute is the first attribute, updating the first virtual component to have a first function; as well as According to determining that the attribute is a second attribute, the first virtual component is updated to have a second function, wherein the first function is different from the second function.

9. The method of any preceding claim, wherein detecting the change in the view comprises receiving a user input indicative of a change in the position of the first virtual element.

10. The method of any of the preceding claims, wherein changing the set of interaction features of the virtual element having the changed position to a different set of interaction features based on the detected properties comprises determining that the first virtual element is located on a plane.

11. The method of any of the preceding claims, wherein changing the set of interaction features of the virtual element having the changed position to a different set of interaction features based on the detected properties comprises determining that the first virtual element is located on a horizontal plane.

12. The method of any of the preceding claims, wherein changing the set of interaction characteristics of the virtual element having the changed position to different interaction characteristics based on the detected properties comprises determining that the first virtual element is located on a vertical plane.

13. The method of any preceding claim, wherein detecting the attribute comprises: identifying, using a classifier, a classification of a portion of the physical environment; as well as An amount of available portion on which the first virtual element can be positioned is identified.

14. The method of any one of the preceding claims, wherein presenting an updated view of the plurality of virtual elements and the physical environment comprises: displaying the virtual element with the changed position at a second position in response to determining that the attribute is the first attribute; as well as Based on determining that the attribute is a second attribute, the virtual element is displayed with the changed position away from the second position, wherein the first attribute is different from the second attribute.

15. A computer-readable medium storing computer-executable program instructions to perform operations comprising: A view is provided, the view comprising: Multiple virtual components from multiple individual separately executed applications; and representations of the physical environment; wherein a first virtual element of the plurality of virtual elements is located at a first position in the view relative to the physical environment and has a first set of interaction features; detecting a change in the view, the change comprising placing the first virtual element at a second position in the view relative to the physical environment; detecting a property of the physical environment at the second position of the first virtual element in the view; selecting a second set of interactive features based on the detected attributes; changing at least one of a size, a shape, a design, and a color attribute of the first virtual element based on the selected set of interaction features and the second position of the first virtual element in the view; changing the first interaction feature group to the second interaction feature group; as well as An updated view of the physical environment and a plurality of virtual elements including the first virtual element is presented.

16. A system, comprising: An electronic device having a display and an image sensor; processor; as well as A computer readable medium comprising instructions that, when executed by the processor, cause the system to perform operations comprising: providing a view including a plurality of virtual elements from a plurality of individual separately executing applications and a representation of a physical environment, wherein a first virtual element of the plurality of virtual elements is located at a first position in the view relative to the physical environment and has a first set of interaction features; detecting a change in the view, the change comprising placing the first virtual element at a second position in the view relative to the physical environment; detecting a property of the physical environment at the second position of the first virtual element in the view; selecting a second set of interactive features based on the detected attributes; changing at least one of a size, a shape, a design, and a color attribute of the first virtual element based on the selected set of interaction features and the second position of the first virtual element in the view; changing the first set of interactive features to the second set of interactive features; and An updated view of the plurality of virtual elements including the first virtual element and the physical environment is presented.

Images