Generating content for physical elements

Through environmental sensors and processors, they can identify physical elements and generate augmented reality content associated with them, which solves the problem of insufficient interaction capabilities of existing devices in physical elements interaction, and achieves rich augmented reality experience and user interaction effects.

CN118981250APending Publication Date: 2024-11-19APPLE INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411013126.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-05-29
Filing Date
2020-05-26
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing augmented reality devices lack sufficient attributes in physical element interactions, limiting the ability of users to interact with them.

Method used

By combining environmental sensors, displays, non-transitory memory and processors, identifying and detecting physical elements, generating augmented reality content associated with them, and generating new augmented reality content based on the combination of these contents, displaying them on the display, Allows users to manipulate these contents.

Benefits of technology

It realizes a rich augmented reality experience in the interaction of physical elements, enhances the interaction between users and the virtual environment, and improves the display and manipulation effect of augmented reality content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981250A_ABST
    Figure CN118981250A_ABST
Patent Text Reader

Abstract

The invention relates to generating content for physical elements. In response to identifying a first physical element of a first object type, first augmented reality (ER) content may be acquired. The first ER content may be associated with a first object type. Second ER content may be retrieved in response to identifying a second physical element of a second object type. The second ER content may be associated with a second object type. The second physical element may be detected as being within a threshold distance of the first physical element. Third ER content may be generated based on a combination of the first ER content and the second ER content. The third ER content may be associated with a third object type different from the first object type and the second object type. The third ER content may be displayed on a display.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 034477, international application date May 26, 2020, date of entry into the Chinese national phase November 26, 2021, Chinese national application number 202080039380.3, and invention name “Generating Content for Physical Elements”.

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 853,852, filed May 29, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure generally relates to generating content for physical elements. Background Art

[0005] Some devices may be capable of generating and presenting augmented realities. Some augmented realities may include virtual scenes, which are simulated replacements for real-world scenes. Some augmented realities may include augmented scenes, which are modified versions of real-world scenes. Some devices that present augmented realities may include mobile communication devices, such as smart phones, head-mounted displays (HMDs), glasses, head-up displays (HUDs), and optical projection systems. Some devices that present augmented realities may present augmented reality (ER) representations of physical elements. Some devices that present ER representations of physical elements may not provide enough properties for users to interact with them. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0007] Figures 1A to 1E is an illustration of an example operating environment according to some implementations.

[0008] Figure 2 is a flowchart representation of a method of generating ER content according to some specific implementations.

[0009] Figure 3 is a block diagram of a device enabled with various components for generating ER content according to some implementations.

[0010] FIG. 4A to FIG. 4C is a block diagram of an exemplary operating environment according to some implementations.

[0011] Figure 5 is a flowchart representation of a method of generating ER content according to some specific implementations.

[0012] Figure 6 is a block diagram of a device enabled with various components for generating ER content according to some implementations.

[0013] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0014] Various embodiments disclosed herein include devices, systems, and methods for generating augmented reality (ER) content for physical elements. In various embodiments, a device may include an environmental sensor, a display, a non-transitory memory, and one or more processors coupled to the non-transitory memory. In response to identifying a first physical element of a first object type, first augmented reality (ER) content may be obtained. The first ER content may be associated with the first object type. In response to identifying a second physical element of a second object type, second ER content may be obtained. The second ER content may be associated with the second object type. The second physical element may be detected as being within a threshold distance of the first physical element. A third ER content may be generated based on a combination of the first ER content and the second ER content. The third ER content may be associated with a third object type different from the first object type and the second object type. The third ER content may be displayed on the display.

[0015] In some implementations, a device may include an environmental sensor, a display, a non-transitory memory, and one or more processors coupled to the non-transitory memory. ER content may be displayed within or extended from an ER representation of a physical element. An affordance may be compounded in association with a portion of the ER content. The affordance may allow manipulation of the portion of the ER content. An input involving the affordance may be detected. In response to detecting the input involving the affordance, manipulation of the portion of the ER content may be displayed.

[0016] According to some specific implementations, a device may include one or more processors, non-volatile memory, and one or more programs. In some specific implementations, one or more programs may be stored in a non-volatile memory and may be executed by one or more processors. In some specific implementations, one or more programs may include instructions for executing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium may store instructions that, when executed by one or more processors of a device, may cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device may include one or more processors, non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0017] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0018] Various examples of electronic systems and techniques for using such systems in connection with various augmented reality techniques are described.

[0019] A physical set refers to a world that individuals can sense and / or interact with without the use of electronic systems. A physical set, such as a physical park, includes physical elements, such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with a physical set, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0020] In contrast to a physical set, an augmented reality (ER) set refers to a fully (or partially) computer-generated set that various people can sense and / or otherwise interact with using electronic systems. In ER, the movement of a person is monitored in part, and in response thereto, at least one property corresponding to at least one virtual object in the ER set is changed in a manner consistent with one or more laws of physics. For example, in response to the ER system detecting that the person is looking up, the ER system may adjust various audio and graphics presented to the person in a manner consistent with the manner in which such sounds and appearances would change in a physical set. Adjustments to properties of virtual objects in the ER set may also be made, for example, in response to representations of movement (e.g., voice commands).

[0021] A person may sense and / or interact with ER objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER scenery, a person may sense and / or interact with only audio objects.

[0022] Virtual reality (VR) is an example of ER. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0023] Mixed reality (MR) is another example of ER. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0024] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0025] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0026] AR scenery may also refer to an augmented scenery in which a representation of a physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery can be graphically modified (e.g., enlarged) so that the modified portion can still represent the originally captured image (but not an exact copy). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0027] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0028] Various systems allow people to sense and / or interact with the ER scenery. For example, a head-mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into the head-mounted system. The head-mounted system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or translucent display may also be included in the head-mounted system. A translucent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eyes. The display may also include an LED, an OLED, a silicon-based liquid crystal, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. A transparent or translucent display may, for example, selectively transition between a transparent / translucent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0029] The present disclosure provides methods, systems, and / or devices for detecting physical elements and acquiring and / or presenting augmented reality (ER) content based on one or more properties of the physical elements. In some implementations, when two physical elements with associated ER content are within a threshold distance of each other, ER content is generated based on the ER content associated with the physical elements. In some implementations, contextual ER content is displayed with ER representations of the physical elements. In some implementations, affordances are compounded in association with the ER content. The user can manipulate the ER content.

[0030] Figures 1A to 1E 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 may include a controller 102 and an electronic device 104. In some implementations, the electronic device 104 includes a smart phone, a tablet computer, a laptop computer, etc. The electronic device 104 can be carried or worn by the user 10.

[0031] like Figures 1A to 1E As shown, in various implementations, the electronic device 104 presents an ER scenery 106. In some implementations, the ER scenery 106 is generated by the controller 102 and / or the electronic device 104. In some implementations, the ER scenery 106 includes a virtual scenery that is a simulated replacement for a physical scenery. For example, in some implementations, the ER scenery 106 is simulated by the controller 102 and / or the electronic device 104. In some implementations, the ER scenery 106 may be different from the physical scenery on which the electronic device 104 is located.

[0032] In some implementations, the ER scenery 106 includes an enhanced scenery that is a modified version of the physical scenery. For example, in some implementations, the controller 102 and / or the electronic device 104 modifies (e.g., enhances) a representation of the physical scenery in which the electronic device 104 is located to generate the ER scenery 106. In some implementations, the controller 102 and / or the electronic device 104 can generate the ER scenery 106 by adding items to the representation of the physical scenery in which the electronic device 104 is located.

[0033] In some implementations, the controller 102 and / or the electronic device 104 generates the ER scenery 106 by simulating a replica of the physical scenery in which the electronic device 104 is located. In some implementations, the controller 102 and / or the electronic device 104 can generate the ER scenery 106 by removing items and / or adding items from the simulated replica of the physical scenery in which the electronic device 104 is located.

[0034] ER scenery 106 may include an augmented reality (ER) representation of a real object (e.g., a physical element). For example, ER scenery 106 may include an ER representation of a cereal box 108 (hereinafter referred to as "cereal box 108" for brevity). In some implementations, when controller 102 and / or electronic device 104 recognizes the physical element as a cereal box, controller 102 and / or electronic device 104 obtains ER content 118. In some implementations, ER content 118 includes the ER representation of cereal box 108. The ER representation of cereal box 108 may be rendered in a cross-sectional view. Rendering the cross-sectional view may facilitate using the interior of cereal box 108 in ER scenery 106. In other words, in some implementations, ER content 118 is displayed within the ER representation of cereal box 108. More generally, in various implementations, ER content 118 is displayed in association with the ER representation of cereal box 108.

[0035] ER scene 106 may include ER representations of other physical elements, such as Figure 1B The ER representation of the dental floss container 110 shown (hereinafter referred to as "dental floss container 110" for brevity) and Figure 1D12 (hereinafter referred to as "lipstick container 112" for brevity). In some implementations, when the controller 102 and / or the electronic device 104 identifies the physical element as a dental floss container, the controller 102 and / or the electronic device 104 obtains the ER content 120. In some implementations, the ER content 120 includes the ER representation of the dental floss container 110. In some implementations, when the controller 102 and / or the electronic device 104 identifies the physical element as a lipstick container, the controller 102 and / or the electronic device 104 obtains the ER content 122. In some implementations, the ER content 122 includes the ER representation of the lipstick container 112. The ER representation can be displayed at a location corresponding to the location of the associated physical element. For example, if the dental floss container is placed near one side of a cereal box in the physical setting, the ER representation of the dental floss container 110 can be displayed near the corresponding side of the ER representation of the cereal box 108.

[0036] Figure 1B An ER set 106 is shown with an ER representation of another physical element, such as a dental floss container 110. The controller 102 and / or the electronic device 104 may identify the actual dental floss container in the physical set, for example, based on environmental data corresponding to the physical set. The environmental data may include, for example, depth information, photographic information, and / or video information. Figure 1B Dental floss container 110 is shown at a distance D1 from cereal box 108 that is greater than a threshold distance T. Threshold distance T may be a fixed distance, may vary, or may be selected based on the size of the first physical element and / or the second physical element.

[0037] like Figure 1B As shown, when the distance D1 between the dental floss container 110 and the cereal box 108 is greater than the threshold distance T, the controller 102 and / or the electronic device 104 obtains (e.g., retrieves, synthesizes, or generates) and displays the ER content 120 associated with the dental floss container 110. For example, in some implementations, the ER content 120 is retrieved from a content data store. In some implementations, the ER content 120 is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the ER content 120 is generated based on one or more characteristics of the physical element.

[0038] Figure 1CThe dental floss container 110 is shown at a distance D2 from the cereal box 108, which is less than a threshold distance T. In some implementations, when the dental floss container 110 is closer to the cereal box 108 than the threshold distance T, ER content 130 is generated based on a combination of ER content 118 and ER content 120, as disclosed in more detail herein. In some implementations, the ER content 130 is associated with the ER representations of both physical elements (e.g., the cereal box 108 and the dental floss container 110). For example, if the physical element is identified as a dental floss container, the controller 102 and / or the electronic device 104 can generate an ER representation of a rope.

[0039] Figure 1D An ER set 106 is shown with an ER representation of another physical element, such as a lipstick container 112. The controller 102 and / or the electronic device 104 may identify the lipstick container in the physical set, for example, based on environmental data corresponding to the physical set. Figure 1D As shown, when the distance D3 between the lipstick container 112 and the cereal box 108 is greater than the threshold distance T, the controller 102 and / or the electronic device 104 obtains (e.g., retrieves, synthesizes, or generates) and displays the ER content 122 associated with the ER representation of the lipstick container 112. Figure 1D As shown, since the distance D3 is greater than the threshold distance T, the ER content 122 is displayed separately from the ER content 130 .

[0040] Figure 1E Lipstick container 112 is shown at a distance D4 from cereal box 108, which is less than a threshold distance T. In some implementations, when lipstick container 112 is closer to cereal box 108 than threshold distance T, ER content 140 is generated based on a combination of ER content 122 and ER content 130, as disclosed in more detail herein. In some implementations, ER content 140 is not associated with a physical element. For example, if the physical element is identified as a lipstick container, controller 102 and / or electronic device 104 may include an ER representation of a power source, such as a battery in ER content 140.

[0041] like Figure 1E As shown, the controller 102 and / or the electronic device 104 determines that the distance D4 between the cereal box 108 and the lipstick container 112 is within (e.g., less than or equal to) the threshold distance T. In some implementations, the controller 102 and / or the electronic device 104 obtains environmental data (e.g., including depth data, image data, and / or video) to make this determination.

[0042] In some implementations, a head mounted device (HMD) worn by the user 10 presents (e.g., displays) an ER scenery 106 according to various implementations. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the ER scenery 106. In some implementations, the HMD includes a head mounted shell. In various implementations, the head mounted shell includes an attachment area to which another device having a display can be attached. For example, in some implementations, the electronic device 104 can be attached to the head mounted shell. In various implementations, the head mounted shell is shaped to form a receiver for receiving another device including a display (e.g., the electronic device 104). For example, in some implementations, the electronic device 104 slides / snaps onto the head mounted shell or is otherwise attached to the head mounted shell. In some implementations, the display of the device attached to the head mounted shell presents (e.g., displays) the ER scenery 106. In various implementations, examples of the electronic device 104 include smart phones, tablet computers, media players, laptop computers, etc.

[0043] Figure 2 2 is a flowchart representation of a method 200 for generating ER content according to some specific implementations. In various specific implementations, the method 200 is performed by a device (e.g., Figure 1A The method 200 is performed by the controller 102 and / or the electronic device 104 shown in the figure. In some specific implementations, the method 200 is performed by a processing logic, which may include hardware, firmware, software, or a combination thereof. In some specific implementations, the method 200 is performed by a processor executing code, which is stored in a non-transitory computer-readable medium (e.g., a memory).

[0044] As represented by block 202, in some implementations, method 200 includes acquiring (e.g., retrieving, synthesizing, or generating) first ER content in response to identifying a first physical element of a first object type. For example, in some implementations, method 200 includes retrieving the first ER content from a content data store. In some implementations, method 200 includes synthesizing the first ER content based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, method 200 includes generating the first ER content based on the one or more characteristics of the physical element.

[0045] For example, Figure 1AAs shown, the controller 102 and / or the electronic device 104 identifies a cereal box. In some implementations, the controller 102 and / or the electronic device 104 identifies the cereal box based on environmental data corresponding to the physical set. The environmental data may include, for example, depth information, image data, and / or video. In some implementations, in response to identifying the cereal box, the controller 102 and / or the electronic device 104 displays the ER representation of the cereal box 108 and obtains the ER content 118.

[0046] As represented by block 202a, in some implementations, the first ER content is associated with the first physical element. For example, ER content 118 can be associated with a real cereal box represented by ER cereal box 108 (e.g., ER content 118 can include a video provided by a cereal manufacturer). In some implementations, method 200 includes obtaining the first ER content based on one or more features of the physical element. For example, in some implementations, method 200 includes determining a set of features of the physical element, using the set of features to form a search query to an ER content data store, and receiving the first ER content as a search result (e.g., if the cereal in the cereal box is in the shape of an animal figure, receiving an ER representation of an animal). In some implementations, method 200 includes providing the set of features as input to a neural network system that generates the first ER content.

[0047] As shown in block 202b, in some implementations, the first ER content includes a video component and / or an audio component. In some implementations, method 200 includes obtaining the video component and the audio component from different sources. For example, in some implementations, method 200 includes obtaining the video component from a video content storage and obtaining the audio component from a music streaming service.

[0048] As shown in block 202c, the first ER content includes an ER representation of a first set of goal achievers. In some implementations, the first set of goal achievers represent items within the first physical element. For example, each goal achiever in the first set of goal achievers includes a character goal achiever that represents a piece of cereal in a cereal box. For example, if some of the cereal pieces in the cereal box are in the shape of animals, then at least some of the goal achievers in the first set of goal achievers will mimic the behavior of the animals.

[0049] As represented by block 204, in some implementations, the controller 102 and / or the electronic device 104 obtains the second ER content in response to identifying the second physical element of the second object type. For example, in some implementations, the method 200 includes retrieving the second ER content from a content data store. In some implementations, the method 200 includes synthesizing the second ER content based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the method 200 includes generating the second ER content based on the one or more characteristics of the physical element.

[0050] For example, Figure 1B As shown, the controller 102 and / or the electronic device 104 obtains the ER content 120 in response to identifying the physical dental floss container represented by the ER dental floss container 110. In some implementations, the controller 102 and / or the electronic device 104 identifies the dental floss container, for example, based on environmental data corresponding to the physical scene. The environmental data may include, for example, depth information, photographic information, and / or video information. In some implementations, the method 200 includes using an environmental sensor to obtain the environmental data. In some implementations, the environmental sensor is or includes, for example, a camera or a depth sensor.

[0051] Based on the identification, the controller 102 and / or the electronic device 104 can generate the ER content 120, as shown in block 204a. In some implementations, the controller 102 and / or the electronic device 104 retrieves the ER content 120 from a data store based on the identification. In some implementations, the ER content 120 is associated with the dental floss container 110. In some implementations, the ER content 120 is not associated with the dental floss container 110. In some implementations, the ER content 120 is obtained based on a property (e.g., a physical, chemical, electrical, structural, or functional property) of the dental floss container 110.

[0052] In some specific implementations, such as Figure 1B As shown, the controller 102 and / or the electronic device 104 determines that the distance D1 between the cereal box 108 and the dental floss container 110 is greater than the threshold distance T. In some specific implementations, the controller 102 and / or the electronic device 104 obtains environmental data (e.g., including depth information, photographic information, and / or video information) to make this determination. The threshold distance T can be a fixed distance. The threshold distance T can vary or can be selected based on the size of the first physical element or the second physical element. For example, the threshold distance T can be a percentage of the size of the first physical element or a percentage of the size of the second physical element.

[0053] As represented by block 206, in some implementations, the controller 102 and / or the electronic device 104 detects that the second physical element is within a threshold distance of the first physical element. For example, in some implementations, the controller 102 and / or the electronic device 104 determines that the distance D2 between the cereal box 108 and the dental floss container 110 is within (e.g., less than or equal to) the threshold distance T. Figure 1C shown.

[0054] As represented by block 206a, in some implementations, the controller 102 and / or the electronic device 104 obtains environmental data to make this determination. The environmental data may include, for example, depth information, image data, and / or video. In some implementations, the threshold distance T is a fixed distance.

[0055] As represented by block 206b, the threshold distance T may vary or may be selected based on the size of the first physical element or the second physical element. For example, the threshold distance T may be a percentage of the size of the first physical element or a percentage of the size of the second physical element.

[0056] As represented by block 208, in some implementations, method 200 includes generating third ER content based on a combination of the first ER content and the second ER content. The third ER content can be associated with a third object type. The third object type can be different from the first object type and / or the second object type. In some implementations, the third ER content can include a portion of the first ER content, which can be selected based on one or more properties associated with the first object type. The third ER content can include a portion of the second ER content, which can be selected based on one or more properties associated with the second object type. For example, in some implementations, in response to determining that a first physical element (e.g., cereal box 108) and a second physical element (e.g., dental floss container 110) are within a threshold distance T of each other, controller 102 and / or electronic device 104 generates ER content 130, such as Figure 1C shown.

[0057] In some implementations, ER content 130 is associated with ER content 118 and / or ER content 120. For example, controller 102 and / or electronic device 104 may obtain (e.g., retrieve, synthesize, or generate) an ER rope extending from the ER representation of cereal box 108. In some implementations, ER content 130 is retrieved from a content data store. In some implementations, ER content 130 is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of a physical element. In some implementations, ER content 130 is generated based on one or more characteristics of a physical element. In some implementations, ER scenery 106 may include goal achievers representing characters from different fictional materials (e.g., movies, television shows, games, comics, and / or novels). In various implementations, goal achievers may represent tangible objects (e.g., mimic their behavior). For example, in some implementations, goal achievers may represent equipment (e.g., machinery such as airplanes, tanks, robots, cars, etc.). In some implementations, a goal achiever can emulate the behavior of (e.g., represent) a fictional entity (e.g., a thing such as equipment from fictional material). In some implementations, a goal achiever can emulate entities from physical entities (e.g., a goal achiever can represent things from the real world), including entities located inside and / or outside of ER scenery 106. In some implementations, ER content can be related to a goal of a goal achiever.

[0058] In various implementations, the goal achiever performs one or more actions to achieve (e.g., complete, satisfy and / or achieve) one or more goals. In some implementations, the goal achiever performs a series of actions. In some implementations, the controller 102 and / or the electronic device 104 determine the action that the goal achiever will perform. In some implementations, the action of the goal achiever is similar to a certain extent to the action that the entity (e.g., role or thing) corresponding to the goal achiever can perform in the fictional material from which the goal achiever is derived. In some implementations, the goal achiever is associated with a specific goal, and the goal achiever can perform the action that can improve the possibility of satisfying the specific goal.

[0059] In some implementations, a goal achiever may be referred to as an object representation, for example, because a goal achiever may represent a variety of objects (e.g., real-world objects or fictional objects). In some implementations, a goal achiever that represents a character may be referred to as a character goal achiever. In some implementations, a character goal achiever may perform actions to achieve a character goal. In some implementations, a goal achiever that represents an equipment may be referred to as an equipment goal achiever. In some implementations, an equipment goal achiever may perform actions to achieve an equipment goal. In some implementations, a goal achiever that represents an environment may be referred to as an environment goal achiever. In some implementations, an environment goal achiever may perform environment actions to achieve an environment goal. In some implementations, ER content is associated with the actions of a goal achiever.

[0060] The third ER content may include a mix of audio and / or video from the first ER content and / or the second ER content. In some implementations, as represented by block 208a, the third ER content includes a video component from the first ER content and / or the second ER content. In some implementations, as represented by block 208b, the third ER content includes an audio component from the first ER content and / or the second ER content. For example, the third ER content may include audio from the first ER content and video from the second ER content. As another example, the third ER content may include video from the first ER content and audio from the second ER content.

[0061] In some implementations, as represented by block 208c, the third ER content includes an ER representation of a subset of goal achievers. For example, in some implementations, the third ER content is generated at least in part by importing one or more goal achievers from one ER content into another ER content. One or more goal achievers can be imported from the first ER content into the second ER content. As another example, one or more goal achievers can be imported from the second ER content into the first ER content. Any imported goal achievers can interact with goal achievers in the ER content into which they are imported. For example, goal achievers imported from the second ER content can interact with goal achievers in the first ER content. Similarly, goal achievers imported from the first ER content can interact with goal achievers in the second ER content.

[0062] As represented by block 210, in some implementations, method 200 includes displaying third ER content. For example, the third ER content can be displayed using a display that can be integrated as part of electronic device 104. For example, user 10 may be able to see content displayed on a display that forms part of electronic device 104. The third ER content can include virtual reality content, mixed reality content, or both. In some implementations, the third ER content can be displayed using an external display.

[0063] As represented by box 210a, in some embodiments, the third ER content is displayed within the boundaries of the first physical element (e.g., cereal box 108) and / or the second physical element (e.g., dental floss container 110 or lip balm container 112), for example, within the boundaries of the ER representation of the first physical element and / or the second physical element.

[0064] In some implementations, method 200 includes detecting a user input interacting with a portion of the third ER content, and displaying a manipulation of the portion of the third ER content in response to detecting the user input.

[0065] In some implementations, method 200 includes compounding an enable representation in association with a portion of a third ER content. In some implementations, the enable representation allows manipulation of the portion of the third ER content. In some implementations, method 200 includes detecting input involving the enable representation. In some implementations, method 200 includes displaying manipulation of the portion of the third ER content in response to detecting input involving the enable representation. In some implementations, the enable representation includes at least one of a visible enable representation, an invisible enable representation, an opaque enable representation, a semi-transparent enable representation, or a transparent enable representation.

[0066] In some implementations, the portion of the third ER content includes an ER representation of a goal achiever. In some implementations, displaying the manipulation of the portion of the third ER content includes animating the ER representation of the goal achiever to provide the appearance that the ER representation of the goal achiever is performing an action.

[0067] In some implementations, the portion of the third ER content includes an ER object. In some implementations, displaying the manipulation of the portion of the third ER content includes moving the ER object (e.g., moving from a first display position to a second display position) or changing a state of the ER object (e.g., switching between an open state and a closed state).

[0068] It should be understood that Figure 2The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with reference to other methods described herein (e.g., method 500) also apply in a similar manner to the details of the above referenced methods. Figure 2 For example, the first ER content, the second ER content, the third ER content, etc. described above with reference to method 200 optionally have one or more features of the ER content, affordances, inputs, and manipulations, etc. described herein with reference to other methods described herein (e.g., method 500). For the sake of brevity, these details are not repeated here.

[0069] Figure 3 is a block diagram of a device 300 (e.g., a server system) according to some implementations. In some implementations, the device 300 implements Figure 1A Controller 102 and / or electronic device 104 are shown. Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, device 300 includes one or more processing units (CPUs) 302, network interfaces 304, programming interfaces 306, memory 308, and one or more communication buses 310 for interconnecting these components and various other components.

[0070] In some implementations, a network interface 304 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, the one or more communication buses 310 include circuits for interconnecting system components and controlling communications between system components. The memory 308 may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices. The memory 308 may include one or more storage devices remote from the one or more CPUs 302. The memory 308 may include a non-transitory computer-readable storage medium.

[0071] In some implementations, the memory 308 or the non-transitory computer-readable storage medium of the memory 308 may store one or more of the following programs, modules, and data structures, or a subset thereof, which may include an operating system 312, a data acquirer 314, a physical element detector 316, an ER content generator 318, and / or an ER content renderer 320. The data acquirer 314 may include instructions 314a and / or heuristics and metadata 314b for acquiring data, for example, from an environmental sensor. The physical element detector 316 may include instructions 316a and / or heuristics and metadata 316b for detecting physical elements based on the acquired data. The ER content generator 318 may include instructions 318a and / or heuristics and metadata 318b for generating ER content based on the detected physical elements. The ER content renderer 320 may include instructions 320a and / or heuristics and metadata 320b for displaying or rendering the generated ER content.

[0072] In various implementations, the device 300 performs Figure 2 In some implementations, the data acquirer 314 performs the following steps: Figure 2 The operations described in blocks 202 and 204 are shown. In some implementations, the physical element detector 316 performs Figure 2 In some implementations, the ER content generator 318 performs operations related to Figure 2 The operations described in block 208 are shown. In some implementations, the ER content renderer 320 performs the operations described in relation to block 210 .

[0073] FIG. 4A to FIG. 4C 4 is a block diagram of an exemplary operating environment 400 according to some implementations. The operating environment 400 may include a controller 402 and an electronic device 404. In some implementations, the electronic device 404 includes a smart phone, a tablet computer, a laptop, etc. In some implementations, the electronic device 404 includes a head mounted device (HMD). The electronic device 404 may be carried or worn by the user 40.

[0074] like FIG. 4A to FIG. 4C As shown, in various implementations, the electronic device 404 presents an ER scenery 406. In some implementations, the ER scenery 406 is generated by the controller 402 and / or the electronic device 404. In some implementations, the ER scenery 406 includes a virtual environment that is a simulated replacement for a physical scenery. For example, in some implementations, the ER scenery 406 is simulated by the controller 402 and / or the electronic device 404. In some implementations, the ER scenery 406 may be different from the real world environment in which the electronic device 404 is located.

[0075] In some implementations, the ER scenery 406 includes an augmented environment that is a modified version of the physical scenery. For example, in some implementations, the controller 402 and / or the electronic device 404 modifies (e.g., augments) a representation of the physical scenery in which the electronic device 404 resides in order to generate the ER scenery 406. In some implementations, the controller 402 and / or the electronic device 404 may generate the ER scenery 406 by adding items to the representation of the physical scenery in which the electronic device 404 resides.

[0076] In some implementations, the controller 402 and / or the electronic device 404 generates the ER scenery 406 by simulating a replica of the real-world scene in which the electronic device 404 is located. In some implementations, the controller 402 and / or the electronic device 404 can generate the ER scenery 406 by removing items and / or adding items from the simulated replica of the real-world scene in which the electronic device 404 is located.

[0077] ER scenery 406 may include an ER representation of a real-world object (e.g., a physical element). For example, ER scenery 406 may include an ER representation of a cereal box 408 (hereinafter referred to as "ER cereal box 408" for brevity). In some implementations, when controller 402 and / or electronic device 404 identifies the physical element as a cereal box, controller 402 and / or electronic device 404 obtains ER content 410. In some implementations, ER content 410 includes the ER representation of cereal box 408. The ER representation of cereal box 408 may be rendered in a cross-sectional view. Rendering the cross-sectional view may facilitate working with the interior of cereal box 408 in ER scenery 406. In other words, in some implementations, ER content 410 is displayed within the ER representation of cereal box 408. More generally, in various implementations, ER content 410 is displayed in association with the ER representation of cereal box 408.

[0078] like Figure 4A As shown, affordance 412 can be composited in association with a portion of ER content 410. In some implementations, affordance 412 is visible. In some implementations, affordance 412 is invisible. In some implementations, affordance 412 is opaque. In some implementations, affordance 412 is translucent. In some implementations, affordance 412 is transparent. In some implementations, affordance 412 allows manipulation of portions of ER content 410.

[0079] Figure 4BInput 414 is shown that relates to affordance 412. In some implementations, input 414 includes user input provided by user 40. In some implementations, input 414 is generated by controller 402 and / or electronic device 404. When controller 402 and / or electronic device 404 detects input 414, manipulation of a portion of ER content 410 can be displayed.

[0080] For example, Figure 4C As shown, controller 402 and / or electronic device 404 may display the modified or changed portion of the ER content, as represented by modified ER content 410a.

[0081] Figure 5 is a flowchart representation of a method 500 for manipulating ER content according to some implementations. In various implementations, the method 500 can be performed by a device (e.g., Figure 4A The method 500 may be performed by a controller 402 and / or an electronic device 404 as shown. In some implementations, the method 500 may be performed by a processing logic that may include hardware, firmware, software, or a combination thereof. In some implementations, the method 500 may be performed by a processor that executes code that may be stored in a non-transitory computer-readable medium (e.g., a memory).

[0082] As represented by block 502, in some implementations, method 500 includes displaying ER content within or extending ER content from a physical element, such as a Figure 4A Cereal box 408, or an ER representation of a physical element. In some implementations, method 500 includes displaying the ER representation of the physical element. The ER content can be displayed in whole or in part within the boundaries of the ER representation of the physical element.

[0083] As represented by block 502a, in some implementations, method 502a includes detecting a physical element. For example, in some implementations, controller 402 and / or electronic device 404 identifies the physical element as cereal box 408 based on environmental data corresponding to the physical scenery. The environmental data may include, for example, depth information, image data, and / or video. In some implementations, method 500 includes acquiring environmental data using an environmental sensor. The environmental sensor may be or may include, for example, a camera or a depth sensor.

[0084] As represented by block 502b, in some implementations, method 500 includes obtaining ER content. Obtaining ER content may include retrieving, synthesizing, or generating ER content. For example, in some implementations, ER content is retrieved from a content data store. In some implementations, ER content is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of a physical element. In some implementations, ER content is generated based on one or more characteristics of a physical element. Figure 4A As shown, the controller 402 and / or the electronic device 404 may obtain ER content 410 .

[0085] As represented by block 504, in some implementations, method 500 includes compounding an affordance 412 in association with a portion of the ER content. Affordance 412 may allow manipulation of the portion of the ER content. As represented by block 504a, in some implementations, affordance 412 is visible. In some implementations, affordance 412 is invisible. In some implementations, affordance 412 is opaque. In some implementations, affordance 412 is translucent. In some implementations, affordance 412 is transparent.

[0086] In some implementations, as represented by block 504b, the portion of the ER content includes ER representations of goal achievers (such as, for example, character goal achievers, equipment goal achievers, and / or environment goal achievers). Activating the affordance representation can trigger the ER representation of the goal achiever to perform an action. The action can increase the likelihood of satisfying a particular goal that may be associated with the goal achiever. The action can be somewhat similar to an action that an entity (e.g., a character or thing) corresponding to the goal achiever can perform in the fictional material from which the goal achiever is derived.

[0087] As represented by block 506, in some implementations, method 500 includes detecting input 414 related to an affordance. In some implementations, as represented by block 506a, input 414 includes user input provided by user 40. For example, user 40 may click or otherwise select or interact with affordance 412. In some implementations, as represented by block 506b, method 500 includes detecting input generated by controller 402 and / or electronic device 404.

[0088] As represented by block 508, in some implementations, method 500 includes displaying a manipulation of the portion of the ER content in response to detecting the input. For example, as represented by block 508a, the affordance may be visible, and controller 402 and / or electronic device 404 may display an ER representation of a goal achiever (e.g., an ER avatar of a person) that performs the action (e.g., activates the affordance). As another example, controller 402 and / or electronic device 404 may display the portion of the ER content that is modified or changed, such as Figure 4C The modified ER content 410a is represented by.

[0089] In some implementations, as indicated by block 508b, method 500 includes an ER representation of a goal achiever that displays a series of actions. In some implementations, controller 402 and / or electronic device 404 may determine the actions that the goal achiever will perform. In some implementations, the actions of the goal achiever may be similar to a certain extent to the actions that a character or thing corresponding to the goal achiever may perform in the fictional material from which the goal achiever is derived. In some implementations, the goal achiever may be associated with a specific goal, and the goal achiever may perform actions that may increase the likelihood of meeting the specific goal.

[0090] In some implementations, the portion of the ER content includes an ER object. In some implementations, displaying the manipulation of the portion of the ER content includes displaying movement of the ER object from a first display location to a second display location.

[0091] In some implementations, the portion of the ER content includes an ER object. In some implementations, displaying the manipulation of the portion of the ER content includes displaying a change of the ER object from a first state to a second state (e.g., switching between an open state and a closed state).

[0092] In some embodiments, detecting the input includes detecting that the user's ER indicates that the enable indication has been activated.

[0093] In some implementations, detecting input includes detecting that an ER representation of a goal achiever has activated an affordance. For example, in some implementations, detecting input includes detecting that an ER character has activated an affordance without an explicit command from a user.

[0094] It should be understood that Figure 5The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with reference to other methods described herein (e.g., the method 200) also apply in a similar manner to the details of the processes described above with reference to Figure 5 For example, the ER content, affordances, inputs, and manipulations, etc., described above with reference to method 500 optionally have one or more features of the first ER content, second ER content, third ER content, etc., described herein with reference to other methods described herein (e.g., method 200). For the sake of brevity, these details are not repeated here.

[0095] Figure 6 4. It is a block diagram of a device 600 (e.g., a server system) according to some specific implementations. In some specific implementations, the device 600 implements the controller 402 and / or the electronic device 404 shown in FIG. 4. Although some specific features are shown, a person of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 600 may include one or more processing units (CPUs) 602, a network interface 604, a programming interface 606, a memory 608, and one or more communication buses 610 for interconnecting these components and various other components.

[0096] In some implementations, a network interface 604 may be provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, the one or more communication buses 610 may include circuits for interconnecting system components and controlling communications between system components. The memory 608 may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices. The memory 608 may include one or more storage devices remote from the one or more CPUs 602. The memory 608 may include a non-transitory computer-readable storage medium.

[0097] In some implementations, the memory 608 or the non-transitory computer-readable storage medium of the memory 608 may store one or more of the following programs, modules, and data structures, or a subset thereof, which may include an operating system 612, a data acquirer 614, a physical element detector 616, an ER content generator 618, an ER content renderer 620, and / or an affordance representation engine 622. The data acquirer 614 may include instructions 614a and / or heuristics and metadata 614b for acquiring data, for example, from an environmental sensor. The physical element detector 616 may include instructions 616a and / or heuristics and metadata 616b for detecting physical elements based on the acquired data. The ER content generator 618 may include instructions 618a and / or heuristics and metadata 618b for generating ER content based on the detected physical elements. The ER content renderer 620 may include instructions 620a and / or heuristics and metadata 620b for displaying or rendering the generated ER content. The affordance engine 622 may include instructions 622a and / or heuristics and metadata 622b for compounding affordances and / or detecting inputs involving affordances.

[0098] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0099] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0100] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and in the appended claims, the singular forms of "a", "an" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "include" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0101] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device including an environmental sensor, a display, a non-transitory memory, and one or more processors coupled to the non-transitory memory: displaying CGR content within or extending said CGR content from a computer generated reality CGR representation of a physical item; composite an affordance in association with a portion of the CGR content, wherein the affordance enables manipulation of the affordance portion of the CGR content; detecting an input involving the affordance; as well as A manipulation of the portion of the CGR content is displayed in response to detecting the input involving the affordance. 2 . The method of claim 1 , further comprising detecting the physical item based on environmental data.

3. The method according to any one of claims 1-2 also includes displaying a CGR representation of the physical item.

4. A method according to any one of claims 1-3, wherein the enable representation includes at least one of a visible enable representation, an invisible enable representation, an opaque enable representation, a semi-transparent enable representation or a transparent enable representation.

5. The method of any one of claims 1-3, wherein the portion of the CGR content comprises a CGR representation of an entity.

6. The method according to claim 5, further comprising: In response to detecting the input involving the enabling representation, displaying the CGR representation of the entity performing the operation.

7. The method according to any one of claims 1 to 6, further comprising: In response to detecting the input involving the affordance, a CGR avatar of the person activating the affordance is displayed.

8. The method of any one of claims 1-7, wherein the portion of the CGR content comprises a CGR object, and wherein displaying the manipulation of the portion of the CGR content comprises: The movement of the CGR object from the first display position to the second display position is displayed.

9. The method of any one of claims 1-8, wherein the portion of the CGR content comprises a CGR object, and wherein displaying the manipulation of the portion of the CGR content comprises: A change of the CGR object from a first state to a second state is displayed.

10. The method of any one of claims 1-9, wherein retrieving the input comprises detecting a user's CGR representation activating the enable representation.

11. A method according to any one of claims 1-9, wherein retrieving the input includes detecting that a CGR representation of an agent activates the enable representation.

12. A device comprising: one or more processors; non-transitory memory; one or more displays; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: displaying CGR content within or extending said CGR content from a computer generated reality CGR representation of a physical item; composite an affordance in association with a portion of the CGR content, wherein the affordance enables manipulation of the affordance portion of the CGR content; detecting an input involving the affordance; as well as A manipulation of the portion of the CGR content is displayed in response to detecting the input involving the affordance.

13. The device of claim 12, wherein the one or more programs further cause the device to: detect the physical item based on environmental data.

14. The device of any one of claims 12-13, wherein the one or more programs further cause the device to: display a CGR representation of the physical item.

15. An apparatus according to any one of claims 12-14, wherein the enable representation includes at least one of a visible enable representation, an invisible enable representation, an opaque enable representation, a semi-transparent enable representation, or a transparent enable representation.

16. The apparatus of any of claims 12-15, wherein the portion of the CGR content comprises a CGR representation of an entity.

17. The device of claim 16, wherein the one or more programs further cause the device to: In response to detecting the input involving the enabling representation, displaying the CGR representation of the entity performing the operation.

18. The apparatus of any one of claims 12-17, wherein the one or more programs further cause the apparatus to: In response to detecting the input involving the affordance, a CGR avatar of the person activating the affordance is displayed.

19. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device having a display, cause the device to: displaying CGR content within or extending said CGR content from a computer generated reality CGR representation of a physical item; composite an affordance in association with a portion of the CGR content, wherein the affordance enables manipulation of the affordance portion of the CGR content; detecting an input involving the affordance; as well as A manipulation of the portion of the CGR content is displayed in response to detecting the input involving the affordance.

20. The non-transitory memory of claim 19, wherein the one or more programs further cause the device to: detect the physical item based on environmental data.

Citation Information

Patent Citations

  • Clustering process for objects using link counts

    US20200034477A1