Device, method and graphical user interface for forming a CGR file

Abstract

In one embodiment, a method of generating a computer generated reality (CGR) file includes receiving, via one or more input devices, user input to generate a computer generated reality (CGR) scene, user input to associate an anchor with the CGR scene, user input to associate one or more CGR objects with the CGR scene, and user input to associate a behavior with the CGR scene, wherein the CGR object is to be displayed in association with the anchor, wherein the behavior includes one or more triggers and an action, and wherein the action is performed in response to detecting any of the triggers. The method includes generating a CGR file including data about the CGR scene, the CGR file including data about the anchor, the CGR objects, and the behavior.

Description

Technical Field

[0001] This document generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that compose computer-generated reality (CGR) files. Background Art

[0002] Programming CGR applications can be a difficult and time-consuming process, requiring specialized knowledge in, for example, 3D object design and application coding. This creates a high barrier to the generation of high-quality CGR applications. Summary of the invention

[0003] Therefore, there is a need for electronic devices with faster, more efficient methods and interfaces for composing computer generated reality (CGR) files (such as CGR files that can execute CGR applications or can be read by CGR applications to provide a CGR experience to a user). Such methods and interfaces optionally supplement or replace conventional methods for composing CGR files. Such methods and interfaces reduce the cognitive burden imposed on users and produce more efficient human-computer interfaces. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0004] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0005] According to some embodiments, a method is performed at a device having one or more processors, a non-volatile memory, and a display. The method includes displaying a representation of a CGR scene on the display, including displaying corresponding representations of a plurality of CGR objects of the CGR scene. The method includes determining that two CGR objects of the plurality of CGR objects overlap in the CGR scene. The method includes displaying an indicator indicating that two CGR objects of the plurality of CGR objects overlap in the CGR scene in association with at least one corresponding representation of the two CGR objects of the plurality of CGR objects on the display.

[0006] According to some embodiments, a method is performed at a device having one or more processors, a non-volatile memory, a display, and one or more input devices. The method includes displaying a representation of a CGR scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. The method includes receiving user input pointing to a specific representation of one or more CGR objects via one or more input devices. The method includes providing a first manipulation mode associated with a first set of conformal space manipulations of the corresponding CGR object in response to receiving user input pointing to a specific representation of one or more CGR objects. The method includes receiving user input switching from the first manipulation mode to a second manipulation mode associated with a second set of conformal space manipulations of the corresponding CGR object via one or more input devices. The method includes providing a second manipulation mode in response to receiving user input switching from the first manipulation mode to the second manipulation mode.

[0007] According to some embodiments, a method is performed at a device having one or more processors, non-volatile memory, a display, and one or more input devices. The method includes displaying a representation of a computer-generated reality (CGR) scene on the display, including displaying corresponding representations of one or more CGR objects of the CGR scene. The method includes receiving user input for spatially manipulating a specific CGR object in one or more CGR objects via one or more input devices, wherein the specific CGR object is associated with an intuitive spatial manipulation point. The method includes changing the spatial properties of the specific CGR object based on the user input and the intuitive spatial manipulation point.

[0008] According to some embodiments, a method is performed at a device having one or more processors, a non-volatile memory, a display, and one or more input devices. The method includes displaying a representation of a CGR scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. The method includes receiving, via one or more input devices, a user input pointing to a representation of a specific CGR object in one or more CGR objects, the specific CGR object being associated with a parameter. The method includes displaying multiple representations of a specific CGR object on a display from a first perspective, wherein each of the multiple representations of the specific CGR object is associated with a different corresponding value of the parameter. The method includes receiving, via one or more input devices, a user input for changing a first perspective to a second perspective. The method includes displaying multiple representations of a specific CGR object on a display from a second perspective in response to receiving a user input for changing a first perspective to a second perspective.

[0009] According to some embodiments, a method is performed at a device having one or more processors, non-volatile memory, a display, and one or more input devices. The method includes displaying a representation of a computer-generated reality (CGR) scene on the display, including displaying corresponding representations of one or more CGR objects of the CGR scene. The method includes receiving user input to add a behavior to the CGR scene via one or more input devices. The method includes receiving user input of a first trigger defining a behavior via one or more input devices. The method includes receiving user input of a first action defining a behavior via one or more input devices, wherein when presenting the CGR scene, the first action is performed in response to detecting the first trigger.

[0010] According to some embodiments, a method is performed at a device having one or more processors, a non-volatile memory, a display, and one or more input devices. The method includes receiving user input for generating a computer generated reality (CGR) scene via one or more input devices. The method includes receiving user input for associating an anchor with a CGR scene via one or more input devices. The method includes receiving user input for associating one or more CGR objects with a CGR scene via one or more input devices, wherein one or more CGR objects will be displayed in association with the anchor. The method includes receiving user input for associating a behavior with a CGR scene via one or more input devices, wherein the behavior includes one or more triggers and one or more actions, and wherein one or more actions will be performed in response to detecting any one of the one or more triggers. The method includes generating a CGR file including data about a CGR scene, wherein the CGR file includes data about an anchor, one or more CGR objects, and a behavior.

[0011] According to some embodiments, an electronic device includes a display, one or more input devices, one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein, which, when executed by one or more processors of an electronic device having a display and one or more input devices, cause the device to perform or cause the execution of the operations of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display, one or more input devices, a non-transitory memory, and one or more processors configured to execute one or more programs stored in the non-transitory memory includes one or more elements of the elements displayed in any of the above methods, which are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display, one or more input devices; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display and one or more input devices includes a device for performing or causing the execution of the operations of any of the methods described herein.

[0012] Thus, electronic devices having displays and input devices (such as touch-sensitive surfaces) are provided with faster, more efficient methods and interfaces for composing CGR files, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for composing CGR files. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0014] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0015] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0016] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0017] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0018] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0019] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0020] Figure 5 is a block diagram of an exemplary operating architecture according to some embodiments.

[0021] Figure 6 is a block diagram of an exemplary controller according to some embodiments.

[0022] Figure 7 is a block diagram of an exemplary head mounted device (HMD) according to some embodiments.

[0023] Figures 8A to 8DF An exemplary user interface for generating a CGR file is shown according to some embodiments.

[0024] Fig. 9 is a flowchart representation of a method for presenting a CGR scene using backup anchors according to some embodiments.

[0025] Fig.10 is a flowchart representation of a method of configuring a CGR object according to some embodiments.

[0026] Fig.11 is a flowchart representation of a method of resolving overlap of two CGR objects according to some embodiments.

[0027] Fig.12 is a flowchart representation of a method for spatially manipulating a CGR object in different spatial manipulation modes according to some embodiments.

[0028] Fig.13 is a flowchart representation of a method for spatially manipulating a CGR object using intuitive spatial manipulation points, according to some embodiments.

[0029] Fig.14 is a flowchart representation of a method of configuring a CGR object according to some embodiments.

[0030] Fig.15 is a flowchart representation of a method of presenting a CGR scene according to some embodiments.

[0031] Fig.16is a flowchart representation of a method of associating behaviors with CGR scenarios according to some embodiments.

[0032] Fig.17 is a flowchart representation of a method of creating a CGR file according to some embodiments. DETAILED DESCRIPTION

[0033] According to various embodiments, a graphical user interface (GUI) is provided to simplify the generation of a CGR file that includes data about the CGR content and also includes data describing how to render the CGR content.

[0034] under, Figure 1A to Figure 1B , Figures 2 to 3 as well as FIG. 4A to FIG. 4B A description of an exemplary CGR object generation apparatus is provided. Figure 5 , Figure 6 and Figure 7 A description of an exemplary CGR object rendering device is provided. Figures 8A to 8DF An exemplary user interface for generating a CGR file is shown. Figures 8A to 8DF The user interface in Figures 9 to 17 process.

[0035] Exemplary CGR file composition device

[0036] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0037] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0038] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0039] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0040] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices are optionally used, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0041] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0042] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0043] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0044] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0045] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0046] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0047] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0048] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0049] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0050] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0051] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0052] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0053] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

[0054] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0055] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0056] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0057] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0058] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0059] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0060] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0061] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0062] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 163 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or an electromechanical device that converts energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). One or more tactile output generators 163 receive tactile feedback generation instructions from the tactile feedback module 133 and generate tactile outputs on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front portion of the device 100 .

[0063] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver for obtaining information about the location of device 100.

[0064] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in FIG. 1A and FIG. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0065] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0066] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in Apple Inc. (Cupertino, California). iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0067] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touch pad.

[0068] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0069] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0070] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0071] Haptic feedback module 133 includes various software components for generating instructions used by one or more tactile output generators 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0072] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0073] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0074] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0075] Contacts module 137 (sometimes called address book or contact list);

[0076] Telephone module 138;

[0077] Video conferencing module 139;

[0078] Email client module 140;

[0079] Instant messaging (IM) module 141;

[0080] Fitness support module 142;

[0081] A camera module 143 for still images and / or video images;

[0082] Image management module 144;

[0083] Browser module 147;

[0084] Calendar module 148;

[0085] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0086] A widget creator module 150 for forming a user-created widget 149-6;

[0087] Search module 151;

[0088] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0089] Memo module 153;

[0090] Map module 154; and / or

[0091] Online video module 155.

[0092] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0093] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0094] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0095] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0096] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0097] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0098] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0099] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0100] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0101] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0102] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0103] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0104] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0105] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0106] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0107] In conjunction with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc. according to user instructions.

[0108] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0109] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0110] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0111] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0112] A predefined set of functions that are performed exclusively through a touch screen and / or a trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a main desktop menu, or a root menu. In such embodiments, a trackpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0113] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0114] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0115] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0116] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0117] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0118] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0119] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0120] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0121] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0122] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0123] Event distributor module 174 distributes the event information to event recognizers (e.g., event recognizer 180). In some embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0124] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0125] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0126] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0127] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (for example, from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0128] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0129] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0130] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0131] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0132] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0133] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0134] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0135] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0136] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0137] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0138] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user breaks contact with the one or more graphics, selection of the one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some embodiments or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0139] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0140] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0141] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169). The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0142] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0143] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0144] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0145] Signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0146] Time 404;

[0147] Bluetooth indicator 405;

[0148] Battery status indicator 406;

[0149] A tray 408 with icons for commonly used applications, such as:

[0150] o an icon 416 labeled “Phone” of the phone module 138, which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0151] o an icon 418 of the email client module 140 labeled “Mail”, which optionally includes an indicator 410 of the number of unread emails;

[0152] o An icon 420 labeled "Browser" of the browser module 147; and

[0153] o an icon 422 labeled “iPod” of the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0154] Icons for other apps, such as:

[0155] o Icon 424 labeled “Text” of IM module 141;

[0156] o Icon 426 labeled “Calendar” of calendar module 148;

[0157] o Icon 428 labeled “Photos” of the image management module 144;

[0158] o An icon 430 labeled “Camera” of the camera module 143;

[0159] o Icon 432 labeled “Online Video” of the online video module 155;

[0160] o Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0161] o an icon 436 labeled “Map” of the map module 154;

[0162] o Icon 438 labeled “Weather” of the Weather widget 149-1;

[0163] o Icon 440 labeled as “Clock” of the alarm clock desktop applet 169-6;

[0164] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0165] o an icon 444 labeled “Memo” of the memo module 153; and

[0166] o Icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0167] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, in some embodiments, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0168] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 451 and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0169] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0170] In addition, although the following examples are given primarily with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0171] Exemplary CGR file rendering device

[0172] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0173] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0174] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0175] Examples of CGR include virtual reality and mixed reality.

[0176] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0177] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0178] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0179] Examples of mixed reality include augmented reality and augmented virtuality.

[0180] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0181] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0182] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0183] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0184] Figure 5 5 is a block diagram of an exemplary operational architecture 500 according to some embodiments. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, as a non-limiting example, the operational architecture 500 includes an electronic device 520A.

[0185] In some embodiments, the electronic device 520A is configured to present CGR content to the user. In some embodiments, the electronic device 520A includes a suitable combination of software, firmware and / or hardware. According to some embodiments, the electronic device 520A presents CGR content to the user via a display 522 when the user is physically present in a physical environment 503, which includes a table 507 within a field of view 511 of the electronic device 520A. Therefore, in some embodiments, the user holds the electronic device 520A in his / her hand. In some embodiments, when providing CGR content, the electronic device 520A is configured to display a virtual object (e.g., a virtual cylinder 509) and to implement video pass-through of the physical environment 503 (e.g., including a representation 517 of the table 507) on the display 522.

[0186] In some embodiments, the electronic device 520A corresponds to a head mounted device (HMD), and the operating architecture 500 (eg, the scene) includes a controller (eg, Figure 6 In some embodiments, the controller is configured to manage and coordinate the presentation of CGR content to the user. In some embodiments, the controller includes a suitable combination of software, firmware, and / or hardware. Figure 6 The controller is described in more detail. In some embodiments, the controller is a computing device that is in a local or remote position relative to the scene. For example, the controller is a local server located in the scene. In another example, the controller is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene. In some embodiments, the controller is communicatively coupled to the HMD via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, the controller is included in the housing of the HMD.

[0187] In some embodiments, the HMD is configured to present CGR content to the user. In some embodiments, the HMD includes a suitable combination of software, firmware, and / or hardware. Figure 7 The HMD is described in more detail. In some embodiments, the functionality of the controller is provided by and / or integrated with the HMD.

[0188] According to some embodiments, the HMD presents CGR content to the user while the user is virtually and / or physically present within the scene.

[0189] In some embodiments, the user wears the HMD on his head. Thus, the HMD includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the HMD surrounds the user's field of view. In some embodiments, such as in Figure 5 In some embodiments, the electronic device 520A is a handheld device (such as a smartphone or tablet) and is configured to present CGR content, and the user no longer wears the HMD but instead holds the device, with the display directed toward the user's field of view and the camera directed toward the scene. In some embodiments, the handheld device can be placed in a housing that can be worn on the user's head. In some embodiments, the HMD is replaced with a CGR cabin, housing, or chamber configured to present CGR content, in which the user no longer wears or holds the HMD.

[0190] Figure 6 600 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the controller 600 includes one or more processing units 602 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 606, one or more communication interfaces 608 (e.g., universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, etc. type interfaces), one or more programming (e.g., I / O) interfaces 610, a memory 620, and one or more communication buses 604 for interconnecting these components and various other components.

[0191] In some embodiments, one or more communication buses 604 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices 606 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0192] The memory 620 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 620 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 620 or the non-transitory computer-readable storage medium of the memory 620 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 630 and a CGR content module 640.

[0193] The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR content module 640 is configured to manage and coordinate the presentation of CGR content for one or more users (e.g., a single set of CGR content for one or more users, or multiple sets of CGR content for corresponding groups of one or more users). To this end, in various embodiments, the CGR content module 640 includes a data acquisition unit 642, a tracking unit 644, a coordination unit 646, and a data transmission unit 648.

[0194] In some embodiments, the data acquisition unit 642 is configured to at least Figure 5 The electronic device 520A acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For this purpose, in various embodiments, the data acquisition unit 642 includes instructions and / or logic for instructions and heuristics and metadata for heuristics.

[0195] In some embodiments, the tracking unit 644 is configured to map the physical environment 503 and to track at least Figure 5 The electronic device 520A is relative to Figure 5 The location / position of the physical environment 503. For this purpose, in various embodiments, the tracking unit 644 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0196] In some embodiments, the coordination unit 646 is configured to manage and coordinate the CGR content presented to the user by the electronic device 520A. For this purpose, in various embodiments, the coordination unit 646 includes instructions and / or logic for instructions and heuristics and metadata for heuristics.

[0197] In some embodiments, the data transmission unit 648 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device 520A. For this purpose, in various embodiments, the data transmission unit 648 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0198] Although the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 are shown as residing on a single device (e.g., controller 600), it should be understood that in other embodiments, any combination of the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 may be located in separate computing devices.

[0199] also, Figure 6 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 6 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0200] Figure 77 is a block diagram of an exemplary head mounted device (HMD) 700 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the HMD 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more CGR displays 712, one or more optional internal-facing and / or external-facing image sensors 714, memory 720, and one or more communication buses 704 for interconnecting these and various other components.

[0201] In some embodiments, one or more communication buses 704 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 706 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones 707A, one or more speakers 707B, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0202] In some embodiments, one or more CGR displays 712 are configured to display CGR content to the user. In some embodiments, one or more CGR displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more CGR displays 712 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, HMD 700 includes a single CGR display. In another example, HMD 700 includes a CGR display for each eye of the user.

[0203] In some embodiments, one or more image sensors 714 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (either sensor may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 714 are configured to face forward so as to acquire image data corresponding to the scene that the user would see in the absence of the HMD 700 (and may be referred to as a scene camera). The one or more optional image sensors 714 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0204] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some embodiments, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules and data structures or a subset thereof, including an optional operating system 730 and a CGR rendering module 740.

[0205] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR presentation module 740 is configured to present CGR content to the user via one or more CGR displays 712 and / or I / O devices and sensors 706 (such as one or more speakers 707B). For this purpose, in various embodiments, the CGR presentation module 740 includes a data acquisition unit 742, a CGR content presentation unit 744, and a data transmission unit 746.

[0206] In some embodiments, the data acquisition unit 742 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller. In various embodiments, the data acquisition unit acquires a CGR file. For this purpose, in various embodiments, the data acquisition unit 742 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0207] In some embodiments, the CGR content presentation unit 744 is configured to present CGR content to a user. In various embodiments, the CGR content presentation unit 744 presents the CGR content of the CGR file according to the rules set forth in the CGR file. To this end, in various embodiments, the CGR content presentation unit 744 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0208] In some embodiments, the data transfer unit 746 is configured to transfer data (e.g., presentation data, position data, etc.) to at least the controller. For this purpose, in various embodiments, the data transfer unit 746 includes instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0209] Although the data acquisition unit 742, the CGR content presentation unit 744, and the data transmission unit 746 are shown as residing on a single device (e.g., Figure 5 , but it should be understood that in other embodiments, any combination of the data acquisition unit 742, the CGR content presentation unit 744 and the data transmission unit 746 may be located in a separate computing device.

[0210] also, Figure 7 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0211] In various embodiments, CGR content is presented to a user from a CGR file that includes data about the CGR content and also includes data describing how to present the CGR content. In various embodiments, the CGR file includes data about one or more CGR scenes. The CGR file also includes data about triggers (e.g., detectable events) for presenting various CGR scenes. As an example, in various embodiments, the CGR file includes data about a CGR scene representing a game board and also includes data indicating that a CGR scene representing a game board is to be presented when a horizontal flat surface is detected in the field of view of a scene camera.

[0212] The CGR file also includes data about one or more CGR objects associated with the corresponding CGR scene. The CGR file also includes data about triggers related to the actions of the CGR objects. As an example, in various embodiments, the CGR file includes data about multiple CGR objects associated with a CGR scene representing a game board, each CGR object in the multiple CGR objects represents a game piece. The CGR file also includes data indicating the actions of the multiple CGR objects in response to the detected triggers, for example, when a collision of two game pieces is detected, the display of the game piece is stopped.

[0213] Creating such CGR files can be a difficult and time-consuming process, requiring expertise in, for example, 3D object design and application coding. This creates a high barrier to the generation of high-quality CGR applications. Therefore, in various embodiments, a graphical user interface (GUI) is provided to simplify the generation of CGR files that can be read (or executed) by a user device to render a CGR application that includes the CGR content described by the CGR file.

[0214] User interface and associated processes

[0215] Attention now turns to embodiments of a user interface ("UI") and related processes that may be implemented on an electronic device, such as a portable multifunction device (PMD) 100 having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, or a device 300 having one or more processors, non-volatile memory, a display, and an input device.

[0216] Figures 8A to 8DF An exemplary user interface for generating a CGR file according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including Figures 9 to 17 Although some of the examples that follow will be given with reference to input on a touch screen display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface 451 that is separate from the display 450, such as Figure 4B as shown in .

[0217] Fig. 8A A CGR file composition user interface 801 is shown displayed by a portable multifunction device 100 (hereinafter "device 100"). In various embodiments, the CGR file composition user interface 801 is displayed by a CGR file composition application that is executed by the device 100.

[0218] exist Fig. 8AIn the figure, the CGR file constitutes a user interface 801 including a welcome user interface 802A having a new project enable indication 802AA for creating a new CGR project (which can be saved as and / or compiled into a CGR file) and a load project enable indication 802AB for loading a CGR project for editing.

[0219] Fig. 8A User input 899A is shown directed to new item enable representation 802AA. In various embodiments, user input 899A corresponds to a contact (e.g., a tap) detected at the location of new item enable representation 802AA.

[0220] Figure 8B A CGR file composition user interface 801 is shown in response to detecting user input 899A directed to a new project affordance 802AA. Figure 8B In the embodiment of the present invention, the CGR file composition user interface 801 includes an anchor selection user interface 802B for selecting an anchor of a first CGR scene of the project. When a CGR application corresponding to the project is presented and an anchor is detected in a scene camera image of the real environment, the corresponding CGR scene is presented in association with the anchor (e.g., visibly bound to or anchored to the anchor).

[0221] The anchor selection user interface 802B includes multiple anchor selection enable indications 802BA to 802BE. The multiple anchor selection enable indications 802BA to 802BE include a horizontal plane anchor selection enable indication 802BA for creating a CGR project, which includes a CGR scene anchored to a horizontal plane. The multiple anchor selection enable indications 802BA to 802BE include an image anchor selection enable indication 802BB for creating a CGR project, which includes a CGR scene anchored to a selected or captured image. The multiple anchor selection enable indications 802BA to 802BE include a face anchor selection enable indication 802BC for creating a CGR project, which includes a CGR scene anchored to a human face. The multiple anchor selection enable indications 802BA to 802BE include a vertical plane anchor selection enable indication 802BD for creating a CGR project, which includes a CGR scene anchored to a vertical plane. The plurality of anchor selection-enable representations 802BA to 802BE include an object anchor selection-enable representation 802BE for creating a CGR project including a CGR scene anchored to a selected or scanned object.

[0222] Figure 8B User input 899B is shown pointing to image anchor selection-enabling representation 802BB. In various embodiments, user input 899B corresponds to a contact (e.g., a tap) detected at the location of image anchor selection-enabling representation 802BB.

[0223] Figure 8C The CGR file composition user interface 801 is shown in response to detecting user input 899B pointing to image anchor selection affordance 802BB (and possibly additional user input selecting or capturing an image of a playing card). Figure 8C In the example, the CGR file constitutes a user interface 801 including a project management user interface 802C, which includes a plurality of project management user interface areas 803A to 803E.

[0224] The plurality of project management user interface areas 803A to 803E include a toolbar area 803A. The toolbar area 803A includes a plurality of CGR content adding enable indications 803AA to 803AC. The plurality of CGR content adding enable indications 803AA to 803AC include a CGR scene adding enable indication 803AA for adding a CGR scene to a project, a CGR object adding enable indication 803AB for adding a CGR object to a currently selected CGR scene, and a behavior adding enable indication 803AC for adding a behavior to a currently selected CGR scene.

[0225] The toolbar area 803A includes a preview affordance 803AD for presenting a preview of the item in a CGR environment.

[0226] The toolbar area 803A includes multiple view enable representations 803AE to 803AF, which include an object view enable representation 803AE and a CGR view enable representation 803AF, wherein the object view enable representation is used to view the currently selected CGR scene in the view area 803E (further described below) in scene mode (where user input directed to the view area 803E changes the perspective of the view of the currently selected CGR scene), and the CGR view enable representation is used to view the currently selected CGR scene in the view area 803A in CGR mode (where movement of the device 100 changes the perspective of the view of the currently selected CGR scene).

[0227] The toolbar area 803A includes a plurality of setting enable indications 803AG to 803AJ for presenting different settings in the settings area 803D (described further below). The settings enable indications 803AG to 803AJ include a project setting enable indication 803AG for presenting project settings for a project in the settings area 803D, a scene setting enable indication 803AH for presenting scene settings for a currently selected CGR scene in the settings area 803D, an object setting enable indication 803AI for presenting object settings for a currently selected CGR object in the settings area 803D, and a behavior setting enable indication 803AJ for presenting behavior settings for a currently selected behavior in the settings area 803D.

[0228] The plurality of project management user interface regions 803A through 803E include a scene list region 803B that includes a respective affordance for each CGR scene of the project. Figure 8C In the case where the project includes only the first CGR scene, the scene list area 803B includes only the first CGR scene enable representation 803BA corresponding to the first CGR scene. In various embodiments, the first CGR scene enable representation 803BA (and other scene enable representations) include the name of the CGR scene (e.g., "New Scene 01"), which is displayed below the graphical representation of the CGR scene.

[0229] The plurality of project management user interface regions 803A through 803E include a behavior region 803C that includes respective affordances for each behavior of the currently selected CGR scenario. Figure 8C In the example of FIG. 800 , in the case where no behavior has been defined for the first CGR scene, the behavior area 803C is blank.

[0230] The plurality of project management user interface regions 803A to 803E include a settings region 803D for presenting a plurality of settings manipulation affordances. Figure 8C , when scene setting enable representation 803AH is selected (as indicated by the different display of scene setting enable representation 803AH compared to other setting enable representations 803AG, 803AI and 803AJ), setting area 803D includes multiple scene setting manipulation enable representations (including Figure 8C Only a single scene setting manipulation capability indication is shown, namely, the scene name manipulation capability indication 803DAA) for changing the name of the currently selected CGR scene.

[0231] The scene setting menus 803DA to 803DE include: a scene attribute menu 803DA, which includes scene setting manipulation enable indications for changing the scene attributes of the currently selected CGR scene; an anchor attribute menu 803DB, which includes scene setting manipulation enable indications for changing the anchor attributes of the currently selected CGR scene; a global physical attribute menu 803DC, which includes scene setting manipulation enable indications for changing the physical attributes of the currently selected CGR scene (such as whether CGR objects are interactive and / or exist and / or the gravity strength); a global audio A properties menu 803DD, which includes scene setting manipulation affordances for changing the audio properties of the currently selected CGR scene (such as sounds to be played when presenting the CGR scene, e.g., soundtracks, or audio effects to be applied to detected real sounds when presenting the CGR scene, e.g., reverb or attenuation); and a global illumination properties menu 803DE, which is used to change the illumination properties of the currently selected CGR scene (such as directional or omnidirectional lights to be rendered when presenting the CGR scene, or how real lights affect the display of CGR objects of the CGR scene).

[0232] Multiple project management user interface areas 803A to 803E include a view area 803E, which is used to view the currently selected CGR scene, receive user input for manipulating the perspective (e.g., virtual camera position) for viewing the currently selected CGR scene in object view mode, and manipulate CGR objects of the currently selected CGR scene.

[0233] The view area 803E includes a representation of an anchor 803EA associated with the currently selected CGR scene (in Figure 8C , a representation of a selected image (eg, block A)) and representations of CGR objects associated with the currently selected CGR scene as described below.

[0234] Figure 8C User input 899C directed to scene name manipulation affordance 803DAA is shown. In various embodiments, user input 899C corresponds to a contact (e.g., a tap) detected at the location of scene name manipulation affordance 803DAA.

[0235] Fig.8D CGR file composition user interface 801 is shown in response to detecting user input 899C directed to scene name manipulation affordance 803DAA (and possibly additional user input defining a new name for the first CGR scene).

[0236] exist Fig.8D, the scene name manipulation affordance 803DAA has changed to indicate the new name of the first CGR scene (e.g., "interactive card"). Similarly, the first CGR scene affordance 803BA has changed to indicate the new name of the first CGR scene.

[0237] Fig.8D A user input 899D pointing to the anchor property menu 803DB is shown. In various embodiments, the user input 899D corresponds to a contact (eg, a tap) detected at the location of the anchor property menu 803DB.

[0238] Fig. 8E The CGR file composition user interface 801 is shown in response to detecting user input 899D pointing to the anchor property menu 803DB. Fig. 8E , the anchor properties menu 803DB is expanded (within the settings area 803D) to present the primary anchor for changing the currently selected CGR scene (previously accessed via Fig.8D The primary anchor indication 803DBA (selected in the anchor selection user interface 802B), the anchor image indication 803DBB for changing the anchor image (displayed when the primary anchor is an image), and the secondary anchor indication 803DBC for selecting or changing the secondary anchor.

[0239] In various embodiments, when a CGR application corresponding to the project is presented and no primary anchor is detected in a scene camera image of the real environment, but a secondary anchor is detected in the scene camera image (optionally, after a threshold amount of time in which the primary anchor has not been detected), the corresponding CGR scene is presented in association with (e.g., explicitly bound to or anchored to) the secondary anchor.

[0240] Fig. 8E User input 899E is shown directed to secondary anchor affordance 803DBC. In various embodiments, user input 899E corresponds to a contact (e.g., a tap) detected at the location of secondary anchor affordance 803DBC.

[0241] Fig.8F CGR file composition user interface 801 is shown in response to detecting user input 899E pointing to secondary anchor affordance 803DBC (and possibly additional user input selecting a horizontal plane as a secondary anchor). Fig.8F , the anchor properties menu 803DB is further expanded (within the settings area 803D) to include a third-level anchor affordance 803DBD for selecting a third-level anchor.

[0242] In various embodiments, when a CGR application corresponding to the project is presented and neither a primary anchor nor a secondary anchor is detected in a scene camera image of the real environment, but a tertiary anchor is detected in the scene camera image (optionally, after a threshold amount of time in which both the primary and secondary anchors have failed to be detected), the corresponding CGR scene is presented in association with (e.g., explicitly bound to or anchored to) the tertiary anchor.

[0243] Fig.8F User input 899F pointing to CGR object add enable representation 803AB is shown. In various embodiments, user input 899F corresponds to a contact (e.g., a tap) detected at the location of CGR object add enable representation 803AB.

[0244] Figure 8G A CGR file composition user interface 801 is shown in response to detecting user input 899F pointing to a CGR object to add an affordance 803AB. Figure 8G In the example, the CGR file composition user interface 801 includes a media library user interface 804 in the form of a pop-up window.

[0245] The media library user interface 804 includes a plurality of specific CGR object addition enable representations 805A to 805H for adding corresponding CGR objects to the currently selected CGR scene. In various embodiments, each of the plurality of specific CGR object addition enable representations 805A to 805H corresponds to a specific CGR object. In various embodiments, each of the plurality of specific object addition enable representations 805A to 805H corresponds to a specific CGR object associated with a specific CGR object file stored in a local or remote media library (e.g., a media library stored on the device 100 or stored at a remote location associated with the device 100 (such as stored in the cloud)).

[0246] Multiple specific CGR object adding enable representations 805A to 805H include: a cube adding enable representation 805A for adding a cube to the currently selected CGR scene, a diamond adding enable representation 805B for adding a diamond to the currently selected CGR scene, a sphere adding enable representation 805C for adding a sphere to the currently selected CGR scene, a tree adding enable representation 805D for adding a tree to the currently selected CGR scene, a building adding enable representation 805E for adding a building to the currently selected CGR scene, a rock adding enable representation 805F for adding a rock to the currently selected CGR scene, a dish adding enable representation 805G for adding a dish to the currently selected CGR scene, and a cup adding enable representation 805H for adding a cup to the currently selected CGR scene.

[0247] The media library user interface 804 includes a search bar 804A for receiving user input including text. In various embodiments, the text is compared with keywords or other metadata associated with CGR object files in the media library to present a subset of specific CGR object add-in affordances 805A to 805H.

[0248] The media library user interface 804 includes an add to library affordance 804B for adding a CGR object file to the media library. In various embodiments, when a user input pointing to the add to library affordance 804B is detected, an add to library user interface is presented that provides an option for searching other CGR object file repositories for a CGR object file (or multiple CGR object files) to select and add to the media library.

[0249] The media library user interface 804 includes a cancel enable indication 804C, which is used to remove (stop displaying) the media library user interface 804, thereby returning the CGR file composition user interface 801 to the Fig.8F Status shown.

[0250] Figure 8G User input 899G directed to diamond add affordance 805B is shown. In various embodiments, user input 899G corresponds to a contact (e.g., a tap) detected at the location of diamond add affordance 805B.

[0251] Figure 8H A CGR file composition user interface 801 is shown in response to detecting user input 899G pointing to a diamond add affordance 805B. Figure 8H In the media library user interface 804, the display stops. Figure 8H , view area 803E includes a representation of a diamond CGR object 806A displayed at a particular location above a representation of anchor 803EA. In connection therewith, first CGR scene affordance 803BA has changed to indicate that the first CGR scene includes a diamond CGR object.

[0252] Figure 8H User input 899H is shown pointing to preview enable representation 803AD. In various embodiments, user input 899H corresponds to a contact (e.g., a tap) detected at the location of preview enable representation 803AD.

[0253] Figure 8I A CGR file composition user interface 801 is shown in response to detecting user input 899H pointing to a preview affordance 803AD. Figure 8IIn the example, the CGR file composition user interface 801 includes a preview user interface 802D. The preview user interface 802D includes a toolbar area 802DA and a preview display area 802DB. The toolbar area 802DA includes a completion indicator 802DAA, which is used to exit the preview user interface 802D and return to (for example, Figure 8H The preview display area 802DB includes a preview image displayed by the device 100 (for example, using a scene camera or a device 100 that is connected to the preview image). Figure 8I 802DBB and the Ace of Stars 802DBC.

[0254] Figure 8J The CGR file composition user interface 801 is shown in response to the device 100 detecting that a portion of the scene camera image (e.g., a representation of the Ace of Diamonds playing card 802DBB) matches an anchor image (e.g., AofD.jpg). Figure 8J In the embodiment, the preview display area 802DB includes a diamond CGR object 802DBD that is displayed as anchored to a representation of the Ace of Diamonds card 802DBB. Specifically, the diamond CGR object 802DBD is displayed as anchored to a representation of the Ace of Diamonds card 802DBB at a specific position corresponding to the position at which the diamond CGR object 802DBD is anchored. Figure 8H A specific location of a representation of a diamond CGR object 806A is shown above the representation of the anchor 803EA.

[0255] Figure 8K The CGR file showing the Ace of Diamonds card changing position in the physical environment constitutes the user interface 801. Figure 8K , the preview display area 802DB includes a representation of the Ace of Diamonds playing card 802DBB displayed at a different position. The preview display area 802DB also includes a diamond CGR object 802DBD at a different position, which is still displayed as anchored to the representation of the Ace of Diamonds playing card 802DBB.

[0256] Figure 8L The CGR file showing the response to the Ace of Diamonds card moving out of the field of view of the scene camera constitutes the user interface 801. Figure 8L , the preview display area 802DB does not include representations of the Ace of Diamonds playing card 802DBB or the Diamonds CGR object 802DBD.

[0257] Figure 8MA CGR file composition user interface 801 is shown in response to the device 100 detecting a representation of a horizontal plane in a portion of a scene camera image (and in various embodiments, in response to failing to detect that a portion of the scene camera image matches an anchor image for at least a threshold amount of time, e.g., 3 seconds). Figure 8M , because no primary anchor—a representation matching the anchor image (e.g., AofD.jpg)—is detected in the scene image, but a secondary anchor—a representation of the horizontal plane—is detected, the diamond CGR object 802DBD is displayed as anchored to the representation of the horizontal plane (e.g., anchored to the representation of the table 802DBA).

[0258] Figure 8M User input 899I is shown pointing to completion enable indication 803DAA. In various embodiments, user input 899I corresponds to a contact (e.g., a tap) detected at the location of completion enable indication 803DAA.

[0259] Figure 8N A CGR file composition user interface 801 is shown in response to detecting user input 899I pointing to a completion affordance 803DAA. Figure 8N In the example, the CGR file constitutes the user interface 801 including the project management user interface 802C (e.g., in Figure 8H status shown).

[0260] Figure 8N User input 899J directed to CGR scene add affordance 803AA is shown. In various embodiments, user input 899J corresponds to a contact (e.g., a tap) detected at the location of CGR scene add affordance 803AA.

[0261] Fig.8O A CGR file composition user interface 801 is shown in response to detecting user input 899J directed to adding an affordance 803AA to a CGR scene. Fig.8O In the example, the CGR file constitutes the user interface 801 including the anchor selection user interface 802B.

[0262] Fig.8O User input 899K is shown directed to horizontal plane anchor selection-enablement representation 802BA. In various embodiments, user input 899K corresponds to a contact (e.g., a tap) detected at the location of horizontal plane anchor selection-enablement representation 802BA.

[0263] Figure 8P A CGR file composition user interface 801 is shown in response to detecting user input 899K pointing to a horizontal plane anchor selection affordance 802BA. Figure 8PIn the example, the CGR file constitutes the user interface 801 including the project management user interface 802C. Figure 8P In the figure, in addition to the first CGR scene enable representation 803BA, the scene list area 803B also includes a second CGR scene enable representation 803BB corresponding to the second CGR scene.

[0264] The second CGR scene enable representation 803BB is displayed in a different manner than the first CGR scene enable representation 803BA to indicate that the second CGR scene is the currently selected CGR scene. Figure 8P In the embodiment, the second CGR scene enable representation 803BB is displayed with a thicker border than the first CGR scene enable representation 803BA. In various embodiments, the second CGR scene enable representation 803BB is displayed with highlighting or other indications, which indicate that the second CGR scene is the currently selected CGR scene.

[0265] exist Figure 8P , the settings area 803D includes a scene name manipulation enable indication 803DAA within the scene properties menu 803DA indicating the name of the currently selected CGR scene (e.g., "New Scene 02").

[0266] exist Figure 8P , the view area 803E includes a representation of an anchor 803EA associated with the currently selected CGR scene with a grid 803EB overlaid thereon (in Figure 8P , horizontal plane representation).

[0267] Figure 8P User input 899L directed to scene name manipulation affordance 803DAA is shown. In various embodiments, user input 899C corresponds to a contact (e.g., a tap) detected at the location of scene name manipulation affordance 803DAA.

[0268] Figure 8Q CGR file composition user interface 801 is shown in response to detecting user input 899L directed to scene name manipulation affordance 803DAA (and possibly additional user input defining a new name for a second CGR scene).

[0269] exist Figure 8Q , the scene name manipulation affordance 803DAA has changed to indicate the new name of the second CGR scene (e.g., "Tea Time"). Similarly, the second CGR scene affordance 803BB has changed to indicate the new name of the second CGR scene.

[0270] Figure 8QUser input 899M directed to CGR object add enable indication 803AB is shown. In various embodiments, user input 899M corresponds to a contact (e.g., a tap) detected at the location of CGR object add enable indication 803AB.

[0271] Figure 8R A CGR file composition user interface 801 is shown in response to detecting user input 899M pointing to a CGR object to add an affordance 803AB. Figure 8R In the example, the CGR file composition user interface 801 includes a media library user interface 804 in the form of a pop-up window.

[0272] Figure 8R A user input 899N pointing to a dish adding an affordance 805G is shown (in various embodiments, the type of user input is similar to Figure 8G In various embodiments, user input 899N corresponds to an extended and / or intense contact (e.g., a tap and hold and / or a force touch) detected at the location of dish addition indicator 805G.

[0273] Figure 8S The CGR file composition user interface 801 is shown in response to detecting user input 899N pointing to the dish addition user interface 805G. Figure 8S , the media library user interface 804 includes an add selected affordance 804D for adding one or more selected CGR objects to the CGR scene and a deselect all affordance 804E for deselecting any selected CGR objects. Figure 8S As shown, adding a selected indication 804D and a deselect all indication 804E to replace the cancel indication 804C (for example, as Figure 8R In various embodiments, the add selected enable indication 804D and the deselect all enable indication 804E are displayed simultaneously with the deselect enable indication 804C.

[0274] exist Figure 8S In the figure, the dish addition indication 805G includes a selection indicator 805GA indicating that the dish CGR object is selected.

[0275] and Figure 8G and Figure 8H By contrast, in both figures, a first type of user input (e.g., a tap) pointing to a particular CGR object to add an affordance adds the corresponding CGR object to the scene. Figure 8R and Figure 8S, a second type of user input (e.g., tapping and holding) pointing to a particular CGR object to add an enable representation selects the corresponding object (e.g., as indicated by a selection indicator displayed in conjunction with the particular CGR object to add an enable representation).

[0276] Figure 8S User input 899O is shown directed to cup add affordance 805H. In various embodiments, user input 899O corresponds to a contact (e.g., a tap) detected at the location of cup add affordance 805H.

[0277] Figure 8T The CGR file composition user interface 801 is shown in response to detecting user input 899O directed to the cup add user interface 805H. Figure 8T In the figure, the cup addition enable indication 805H includes a selection indicator 805HA indicating that the cup CGR object is selected.

[0278] Figure 8T User input 899P directed to add selected affordance 804D is shown. In various embodiments, user input 899P corresponds to a contact (e.g., a tap) detected at the location of add selected affordance 804D.

[0279] Figure 8U The CGR file composition user interface 801 is shown in response to detecting user input 899P directed to add a selected affordance 804D. Figure 8U In the media library user interface 804, the display stops. Figure 8U , view area 803E includes a representation of a dish CGR object 806B displayed at a first position above a representation of anchor 803EA and a representation of a cup CGR object 806C displayed at a second position above the representation of anchor 803EA. In connection therewith, second CGR scene affordance 803BB has changed to indicate that the second CGR scene includes the dish CGR object and the cup CGR object.

[0280] In various embodiments, the first position of the representation of the dish CGR object 806B and the second position of the representation of the cup CGR object 806C are selected so that the bottom of the representation of the dish CGR object 806B is adjacent to the top of the representation of the anchor 803EA and so that the bottom of the representation of the cup CGR object 806C is adjacent to the top of the representation of the anchor 803EA.

[0281] In various embodiments, the first position of the representation of the dish CGR object 806B and the second position of the representation of the cup CGR object 806C are selected to ensure that the representation of the dish CGR object 806B and the representation of the cup CGR object 806C do not intersect.

[0282] In various embodiments, a first position of the representation of the dish CGR object 806B and a second position of the representation of the cup CGR object 806C are selected based on the grid 803EB. For example, in various embodiments, the first position is selected so that a spatial manipulation point (e.g., an edge, a midpoint, a center of mass, an intuitive spatial manipulation point, or a custom spatial manipulation point) of the representation of the dish CGR object 806B is vertically aligned with a first grid point of the grid 803EB, and the second position is selected so that a spatial manipulation point of the representation of the cup CGR object 806C is vertically aligned with a second grid point of the grid 803EB.

[0283] Figure 8U A user input 899Q is shown directed to the representation of the cup CGR object 806C. In various embodiments, the user input 899Q corresponds to a contact (eg, a tap) detected at the location of the representation of the cup CGR object 806C.

[0284] Figure 8V The CGR file composition user interface 801 is shown in response to detecting user input 899Q pointing to a representation of a cup CGR object 806C. Figure 8V , the view area 803E includes a first type of object selection indicator 807 surrounding the representation of the cup CGR object 806C. The first type of object selection indicator 807 displayed surrounding the representation of the cup CGR object 806C indicates that the cup CGR object is selected.

[0285] exist Figure 8V , when object setting enable representation 803AI is selected (as indicated by the different display of object setting enable representation 803AI compared to other setting enable representations 803AG, 803AH and 803AJ), setting area 803D includes multiple object setting manipulation enable representations (including Figure 8V 806C, object name manipulation affordance 813DAA that displays the name of the cup CGR object (e.g., “cup01”) indicates that the cup CGR object is selected.

[0286] The object setting menus 813DA to 813DD include: an object property menu 813DA, which includes scene setting manipulation enable representations for changing scene properties of the currently selected CGR object; a display property menu 813DB, ​​which includes display setting manipulation enable representations for changing the display (or rendering) properties of the currently selected CGR object (such as the shape, color, or optical transmission of the CGR object); a physical property menu 813DC, which includes object setting manipulation enable representations for changing the physical properties of the currently selected CGR object (such as the light reflectivity of the CGR object or the elasticity of the CGR object [for example, how the CGR object interacts with other CGR objects]); and an audio property menu 813DD, which includes object setting manipulation enable representations for changing the audio properties of the currently selected CGR object (such as omnidirectional sound or ambient sound to be played when the CGR object is presented, or directional sound to be emitted by the CGR object).

[0287] Figure 8V User input 899R is shown pointing to display properties menu 813DB. In various embodiments, user input 899Q corresponds to a contact (e.g., a tap) detected at the location of the representation of display properties menu 813DB.

[0288] Figure 8W The CGR file composition user interface 801 is shown in response to detecting user input 899R directed to display attribute menu 813DB. Figure 8W , the display properties menu 813DB is expanded (within the settings area 803D) to present: a configurator enable indication 813DBA for presenting a configurator user interface (described below); a style enable indication 813DBB for changing the shape, rendering wireframe, or display mesh of the currently selected CGR object; and a pattern enable indication 813DBC for changing the pattern, color, or texture of the currently selected CGR object.

[0289] Figure 8W User input 899S is shown directed to configurator-enabled representation 813DBA. In various embodiments, user input 899S corresponds to a contact (e.g., a tap) detected at the location of configurator-enabled representation 813DBA.

[0290] Figure 8X A CGR file composition user interface 801 is shown in response to detecting user input 899S pointing to the configurator affordance 813DBA. Figure 8X In the example, the CGR file composition user interface 801 includes a configurator user interface 808 in the form of a pop-up window.

[0291] Configurator user interface 808 includes a style bar 808E including a plurality of style affordances 808EA to 808ED for changing a currently selected style of style bar 808E, and a pattern bar 808F including a plurality of pattern affordances 808FA to 808FD for changing a currently selected pattern of pattern bar 808F. Style bar 808E includes a mug affordance 808EA for changing a currently selected style to a mug style, a teacup affordance 808EB for changing a currently selected style to a teacup style, an espresso affordance 808EC for changing a currently selected style to an espresso style, and a beer mug affordance 808ED for changing a currently selected style to a beer mug style. The pattern bar 808F includes: a white enable indication 808FA for changing the currently selected pattern to a white pattern, a glass enable indication 808FB for changing the currently selected pattern to a glass pattern, a star enable indication 808FC for changing the currently selected pattern to a star pattern, and an ivy enable indication 808FD for changing the currently selected pattern to an ivy pattern.

[0292] The configurator user interface 808 includes a view window 808D that presents a representation of a currently selected CGR object 808DA having the style currently selected in the style bar 808E and the pattern currently selected in the pattern bar 808F (as well as the name 808C of the currently selected CGR object above the view window 808D).

[0293] The configurator user interface 808 includes multiple view enable representations 808A to 808B, which include an object view enable representation 808A and a CGR view enable representation 808B, wherein the object view enable representation is used to view the currently selected CGR object in the view window 808D in a scene mode (wherein user input directed to the view window 808D changes the perspective of the view of the currently selected CGR object), and the CGR view enable representation is used to view the currently selected CGR object in the view window 808D in a CGR mode (wherein movement of the device 100 changes the perspective of the view of the currently selected CGR object).

[0294] The configurator user interface 808 includes a confirmation enable indication 808G, which is used to change the style of the currently selected CGR object to the currently selected style in the style bar 808E, change the pattern of the currently selected CGR object to the currently selected pattern in the pattern bar 808F, and cancel (stop displaying) the configurator user interface 808. The configurator user interface 808 includes a cancel enable indication 808H, which is used to cancel (stop displaying) the configurator user interface 808, thereby returning the CGR file composition user interface 801 to Figure 8W The state shown does not change the style and / or pattern of the currently selected CGR object.

[0295] In various embodiments, when a particular style is the currently selected style of the style column 808E, a corresponding subset of patterns is presented for selection in the pattern column 808F. In various embodiments, the corresponding subset is based on one or more selection rules that define how various styles and patterns can be combined. In various embodiments, the selection rules are stored in the corresponding CGR object file.

[0296] For example, in Figure 8X 808FD are presented for selection. Specifically, star indicator 808FC and ivy indicator 808FD are presented for selection, as indicated by the second display style (e.g., black) of mug indicator 808EA, which contrasts with the second display styles (e.g., white) of other style indicators 808EB through 808ED. Additionally, white pattern is the currently selected pattern of pattern bar 808F, as indicated by the first display style of white indicator 808FA. Based on the currently selected style being mug style, a subset of pattern indicators 808FA through 808FD are presented for selection. Specifically, star indicator 808FC and ivy indicator 808FD are presented for selection, as indicated by the second display style, while glass indicator 808FB is not presented for selection, as indicated by the third display style (e.g., gray).

[0297] Figure 8X User input 899T is shown directed toward espresso enabling representation 808EC. In various embodiments, user input 899T corresponds to a contact (e.g., a tap) detected at the location of espresso enabling representation 808EC.

[0298] Figure 8Y A CGR file composition user interface 801 is shown in response to detecting user input 899T directed to an espresso affordance 808EC. Figure 8Y808EC, in contrast to the second display styles of the other style affordances 808EA, 808EB, and 808ED. In addition, the representation of the currently selected CGR object 808DA in the view window 808D changes based on the change in the currently selected style.

[0299] As described above, in various embodiments, when a particular style is the currently selected style in style column 808E, a corresponding subset of patterns is presented for selection in pattern column 808F. Thus, as another example, in Figure 8Y In the example, based on the currently selected style being the espresso style, a subset of pattern indicators 808FA to 808FD are presented for selection. Specifically, the glass indicator 808FB is presented for selection, as indicated by the second display style, while the star indicator 808FC and the ivy indicator 808FD are not presented for selection, as indicated by the third display style.

[0300] Figure 8Y User input 899U is shown directed to teacup affordance 808EB. In various embodiments, user input 899U corresponds to a contact (e.g., a tap) detected at the location of teacup affordance 808EB.

[0301] Figure 8Z A CGR file composition user interface 801 is shown in response to detecting user input 899U pointing to a teacup affordance 808EB. Figure 8Z 808EB, in contrast to the second display styles of the other style affordances 808EA, 808EC, and 808ED. In addition, the representation of the currently selected CGR object 808DA in the view window 808D changes based on the change in the currently selected style.

[0302] As described above, in various embodiments, when a particular style is the currently selected style in style column 808E, a corresponding subset of patterns is presented for selection in pattern column 808F. Thus, as another example, in Figure 8Z In the embodiment, based on the currently selected style being the teacup style, a subset of pattern indicators 808FA to 808FD are presented for selection. Specifically, star indicators 808FC and ivy indicators 808FD are presented for selection, as indicated by the second display style, while glass indicators 808FB are not presented for selection, as indicated by the third display style.

[0303] Figure 8ZUser input 899V is shown directed toward star-shaped enable indication 808FC. In various embodiments, user input 899V corresponds to a contact (e.g., a tap) detected at the location of star-shaped enable indication 808FC.

[0304] Figure 8AA A CGR file composition user interface 801 is shown in response to detecting user input 899V pointing to a star-shaped affordance 808FC. Figure 8AA In the example, the currently selected pattern is changed to a star style, as indicated by the first display style of the star-shaped affordance 808FC, which is in contrast to the second display style or the third display style of the other pattern affordances 808FA, 808FB, and 808FD. In addition, the representation of the currently selected CGR object 808DA in the view window 808D changes based on the change of the currently selected pattern.

[0305] As described above, in various embodiments, when a particular style is the currently selected style in style column 808E, a corresponding subset of patterns is presented for selection in pattern column 808F. Thus, as another example, in Figure 8AA In the example, based on the currently selected style being the teacup style, a subset of pattern indicators 808FA to 808FD is presented for selection. Specifically, white indicator 808FA and ivy indicator 808FD are presented for selection, as indicated by the second display style, while glass indicator 808FB is not presented for selection, as indicated by the third display style.

[0306] Figure 8AA User input 899W is shown directed toward determine-enable representation 808G. In various embodiments, user input 899W corresponds to a contact (e.g., a tap) detected at the location of determine-enable representation 808G.

[0307] Figure 8AB A CGR file composition user interface 801 is shown in response to detecting user input 899W pointing to an affordance 808G. Figure 8AB In the process, the configurator user interface 808 is dismissed, the style of the currently selected CGR object is changed to a teacup style, as indicated by the appearance change of the representation of the cup CGR object 806C and the change of the style indicator 813DBB, and the pattern of the currently selected CGR object is changed to a star pattern, as indicated by the appearance change of the cup CGR object 806C and the change of the pattern indicator 813DBC.

[0308] Figure 8ABA user input 899X is shown pointing to the representation of the cup CGR object 806C. In various embodiments, the user input 899X corresponds to two contacts (e.g., a pinch or spread gesture) detected at the location of the representation of the cup CGR object 806C that move toward or away from each other.

[0309] Figure 8AC The CGR file composition user interface 801 is shown in response to detecting user input 899X pointing to a representation of a cup CGR object 806C. Figure 8AC 806C (and similarly, the change in second CGR scene affordance representation 803BB). In various embodiments, the change in size of the cup CGR object is proportional to the change in distance between the two contacts of user input 899X.

[0310] Figure 8AC User input 899Y is shown pointing to the representation of the cup CGR object 806C. In various embodiments, the user input 899Y corresponds to two contacts detected at the location of the representation of the cup CGR object 806C that move around a common center (e.g., a rotation gesture).

[0311] Figure 8AD The CGR file composition user interface 801 is shown in response to detecting user input 899Y pointing to a representation of a cup CGR object 806C. Figure 8AD , the orientation of the cup CGR object changes, as indicated by the changing orientation of the representation of the cup CGR object 806C (and similarly, the change in the second CGR scene affordance representation 803BB). In various embodiments, the cup CGR object (and its corresponding representation 806C) rotates about a z-axis that is perpendicular to the horizontal plane and passes through a spatial manipulation point (e.g., an edge, a midpoint, a center of mass, an intuitive spatial manipulation point, or a custom spatial manipulation point) of the cup CGR object.

[0312] In various embodiments, rotation about an axis passing through the midpoint or unweighted center of mass of a CGR object (or snapping the midpoint or unweighted center of mass of a CGR object to a target point, e.g., treating the distance between the midpoint or unweighted center of mass of the CGR object and the target point as zero when the distance is less than a threshold amount) results in non-intuitive spatial manipulation of the CGR object. Figure 8AE1 A side view of a cup CGR object 880 is shown. Figure 8AE2 A top view of a cup CGR object 880 is shown. The cup CGR object 880 includes a body 881A, a base 881B, and a handle 881C. Figure 8AE1 A side view of a bounding box 882 surrounding a cup CGR object 880 is shown. Figure 8AE2 A top view of a bounding box 882 surrounding a cup CGR object 880 is shown.

[0313] Figure 8AE1 and Figure 8AE2 The first spatial manipulation point 883 is shown as the midpoint of the bounding box 882. Thus, a vertical line passing through the first spatial manipulation point 883 bisects the edge of the bounding box 882. However, in various situations, performing spatial manipulation of the cup CGR object 880 using the first spatial manipulation point 883 may result in non-intuitive results, particularly because the first spatial manipulation point 883 is off-center compared to the center of the body 881A (due to the cup CGR object 880 including a handle 881C). For example, rotating the cup CGR object 880 about a z-axis passing through the first spatial manipulation point 883 may also appear to move the cup CGR object 880. As another example, snapping the cup CGR object 880 by aligning the first spatial manipulation point 883 with the center (or grid point) of the saucer CGR object will cause the cup CGR object 880 to appear offset relative to the saucer CGR object (or grid point).

[0314] Similarly, the second spatial manipulation point 884 at the center of mass of the CGR object 880 is unintuitively offset from the center of the body 881A, resulting in similar unintuitive results when the cup CGR object is rotated or moved when the second spatial manipulation point 884 is used.

[0315] Thus, in various embodiments, when performing spatial manipulation of the cup CGR object 880, an intuitive spatial manipulation point 885 is used. In various embodiments, the intuitive spatial manipulation point 885 is stored with the CGR object file that defines the cup CGR object. The intuitive spatial manipulation point 885 may be defined, for example, by a designer of the CGR object file. In various embodiments, the intuitive spatial manipulation point 885 is set by a user of the CGR file that constitutes the user interface 801.

[0316] In various embodiments, the apparent spatial manipulation point 885 is determined by the device 100. For example, in various embodiments, the device 100 determines multiple segments of the cup CGR object 880 (e.g., the body 881A, the base 881B, and the handle 881C), and determines the apparent spatial manipulation point 885 as the center of mass of the cup CGR object, where different segments are weighted differently (e.g., more weight is assigned to the body 881A than to the handle 881C). In various embodiments, the weight assigned to one or more segments is zero, for example, the segment is ignored when determining the apparent spatial manipulation point 885.

[0317] Return to Figure 8AD , Figure 8ADA user input 899Z is shown pointing to the representation of the cup CGR object 806C. In various embodiments, the user input 899Z corresponds to a detected moving contact (eg, drag or touch and drag) starting at the location of the representation of the cup CGR object 806C.

[0318] Figure 8AF The CGR file composition user interface 801 is shown in response to detecting user input 899Z pointing to a representation of a cup CGR object 806C. Figure 8AF 806C (and similarly, the change in second CGR scene affordance representation 803BB). In various embodiments, the cup CGR object (and its corresponding representation 806C) moves in an xy plane parallel to the horizontal plane. In various embodiments, the change in position of the cup CGR object is proportional to the distance of movement of the contact of user input 899Z.

[0319] Based on the changes in position of the cup CGR object, as indicated by the position of the representation of cup CGR object 806C and the position of the saucer CGR object, as indicated by the position of the representation of saucer CGR object 806B, the cup CGR object and the saucer CGR object overlap.

[0320] In various embodiments, device 100 determines whether two or more CGR objects overlap at an overlapping position in a CGR scene. In various embodiments, device 100 periodically performs this determination. In various embodiments, device 100 performs the determination each time an object is manipulated spatially (e.g., rotated, moved, or sized). In various embodiments, device 100 performs the determination in response to a user input requesting an overlap check.

[0321] In various embodiments, in response to determining that two or more CGR objects overlap at an overlapping position in the CGR scene, an overlap indication is presented. In various embodiments, the overlap indication is displayed in association with at least one of the two or more CGR objects. In various embodiments, the overlap indication is displayed in association with the overlapping position.

[0322] Figure 8AF A first overlapping indicator 809A is shown in the form of a notification bubble coupled to the representation of the cup CGR object 806C, and a second overlapping indicator 809B is in the form of a shadow, glow, hatching, or other highlight overlaid over the representation of the cup CGR object 806C and the representation of the saucer CGR object 806B.

[0323] Figure 8AFUser input 899AA is shown directed to first overlapping indicator 809A. In various embodiments, user input 899AA corresponds to a contact (e.g., a tap) detected at the location of first overlapping indicator 809A.

[0324] Figure 8AG The CGR file composition user interface 801 is shown in response to detecting user input 899AA pointing to the first overlap indicator 809A. Figure 8AG In the example, the CGR file composition user interface 801 includes an overlapping notification user interface 810 in the form of a pop-up window.

[0325] The overlap notification user interface 810 includes an overlap text description 810A describing the detected overlap of CGR objects and an overlap graphical view 810B showing the overlapping CGR objects.

[0326] The overlapping notification user interface 810 includes a cancel enable indication 810C, which is used to remove (stop displaying) the overlapping notification user interface 810, thereby returning the CGR file composition user interface 801 to the Figure 8AG Status shown.

[0327] The overlap notification user interface 810 includes an ignore affordance 810D for dismissing the overlap notification user interface 810 and the corresponding overlap indicator (without moving any of the overlapping CGR objects), thereby returning the CGR file composition user interface 801 to the Figure 8AG The state without the first overlap indicator 809A and the second overlap indicator 809B is shown.

[0328] The overlap notification user interface 810 includes a repair enable indication 810E, which is used to automatically change the position of at least one of the CGR objects so that the overlapping CGR objects no longer overlap.

[0329] In various embodiments, the CGR object that was most recently manipulated in space among two or more overlapping CGR objects is selected to be moved. In various embodiments, the selected CGR object is moved in a direction that performs the minimum movement to repair the overlap.

[0330] Figure 8AG User input 899AB is shown directed to repair affordance 810E. In various embodiments, user input 899AB corresponds to a contact (e.g., a tap) detected at the location of repair affordance 810E.

[0331] Figure 8AHA CGR file composition user interface 801 is shown in response to detecting user input 899AB pointing to a repair affordance 810E. Figure 8AH , the overlap notification user interface 810 is removed and the cup CGR object has been moved (upward) to no longer overlap with the saucer CGR object, as indicated by the movement of the representation of the cup CGR object 806C (and the corresponding change in the second CGR scene affordance representation 803BB).

[0332] Figure 8AH User input 899AC is shown directed to the representation of the cup CGR object 806C. In various embodiments, the user input 899AC corresponds to a contact (e.g., a tap) detected at the location of the representation of the cup CGR object 806C. In various embodiments, the user input 899AC corresponds to a contact (e.g., a tap) detected at the location of the first type of object selection indicator 807 surrounding the representation of the cup CGR object 806C.

[0333] Figure 8AI The CGR file composition user interface 801 is shown in response to detecting user input 899AC pointing to a representation of a cup CGR object 806C. Figure 8AI , the first type of object selection indicator 807 is replaced with a second type of object selection indicator 817 surrounding the representation of the cup CGR object 806C.

[0334] Thus, when a CGR object is selected (and the first type of object selection indicator 807 is displayed), different types of user input directed to the representation of the CGR object result in different changes to the spatial properties of the CGR object. Figure 8AB and Figure 8AC , a first type of user input 899X (e.g., pinching) directed to the representation of the cup CGR object 806C changes the size of the cup CGR object. As another example, in Figure 8AC and Figure 8AD , a second type of user input 899Y (e.g., rotation) directed to the representation of the cup CGR object 806C changes the orientation about the z-axis of the cup CGR object. As another example, in Figure 8AD and Figure 8AF , a third type of user input 899Z (e.g., dragging) directed to the representation of the cup CGR object 806C changes the position of the cup CGR object in the xy plane. As another example, in Figure 8AH and Figure 8AI, a fourth type of user input 899AB (eg, a tap) directed to the representation of the cup CGR object 806C changes the first type of object selection indicator 807 to a second type of object selection indicator 817, thereby allowing various additional spatial manipulations as described below.

[0335] Congruent (or isometric) spatial manipulations of CGR objects are spatial manipulations in which the size and shape of the CGR object do not change. Congruent spatial manipulations include translation, rotation, and reflection. Analogous (or conformal) spatial manipulations of CGR objects are spatial manipulations in which the shape of the CGR object does not change. Analogous spatial manipulations include scaling, translation, rotation, and reflection.

[0336] As described above, when a CGR object is selected (and the first type of object selection indicator 807 is displayed), various user inputs perform a first set of similar spatial manipulations. For example, in various embodiments, the first set of similar spatial manipulations includes scaling (e.g., Figure 8AB and Figure 8AC ), rotation about a first axis (as shown Figure 8AC and Figure 8AD ) and two-dimensional translation in the first plane (as Figure 8AD and Figure 8AF shown).

[0337] As described below, when a CGR object is selected (and the second type of object selection indicator 817 is displayed), various user inputs perform a second set of similar spatial manipulations that are different from the first set of similar spatial manipulations. In various embodiments, the second set of similar spatial manipulations includes one or more or all of the first set of similar spatial manipulations. For example, in various embodiments, the second set of similar spatial manipulations includes rotation around a first axis, rotation around a second axis, rotation around a third axis, and three-dimensional translation.

[0338] The second type of object selection indicator 817 includes: a first translation enable indication 817A for moving the currently selected CGR object along a first axis (e.g., the z-axis, moving the currently selected CGR object up and down), a second translation enable indication 817B for moving the currently selected CGR object along a second axis (e.g., within the xy plane), and a third translation enable indication 817C for moving the currently selected CGR object along a third axis (e.g., perpendicular to the second axis within the xy plane). The second type of object selection indicator 817 includes: a first rotation enable indication 817D for rotating the currently selected CGR object around the first axis, a second rotation enable indication 817E for rotating the currently selected CGR object around the second axis, and a third rotation enable indication 817F for rotating the currently selected CGR object around the third axis.

[0339] Figure 8AI User input 899AD is shown pointing to third rotational enable representation 817F. In various embodiments, user input 899AD corresponds to a detected moving contact (e.g., dragging or touch and drag) with a starting position at the position of third rotational enable representation 817F.

[0340] Figure 8AJ A CGR file composition user interface 801 is shown in response to detecting user input 899AD pointing to a third rotational affordance 817F. Figure 8AI , the cup CGR object rotates about the third axis, as indicated by the rotation of the representation of the cup CGR object 806C (and the corresponding change of the second CGR scene affordance representation 803BB).

[0341] Figure 8AJ User input 899AE is shown directed to first translation-enabled representation 817A. In various embodiments, user input 899AE corresponds to a detected moving contact (e.g., dragging or touch-and-drag) starting at the location of first translation-enabled representation 817A.

[0342] Figure 8AK A CGR file composition user interface 801 is shown in response to detecting user input 899AE pointing to a first translation affordance 817F. Figure 8AJ , the cup CGR object moves downward along the first axis, as indicated by the movement of the representation of the cup CGR object 806C (and the corresponding change in the second CGR scene affordance representation 803BB).

[0343] Figure 8AK User input 899AF is shown pointing to a location in the view area 803E that is away from any representation of a CGR object. In various embodiments, user input 899AF corresponds to a contact (e.g., a tap) detected at a location in the view area 803E that is away from any representation of a CGR object.

[0344] Figure 8AL The CGR file composition user interface 801 is shown in response to detecting user input 899AF pointing to a location in the view area 803E. Figure 8AL, a CGR object is not selected, as indicated by the lack of an object selection (e.g., a first type of object selection indicator 807 or a second type of object selection indicator 817). Without a CGR object being selected, scene setting affordance 803AH is selected (as indicated by the different display of scene setting affordance 803AH compared to other setting affordances 803AG, 803AI, and 803AJ), and settings area 803D includes a plurality of scene setting manipulation affordances presented via collapsible / expandable scene setting menus 803DA through 803DE.

[0345] Figure 8AL User input 899AG is shown pointing to a location in view area 803E. In various embodiments, user input 899AG corresponds to two contacts detected at a location in view area 803E moving toward or away from each other (e.g., a pinch or spread gesture). In various embodiments, the location in view area 803E is away from any representation of a CGR object (while the CGR object is selected or not). In various embodiments, the location is at the location of a representation of a CGR object that is not selected.

[0346] Figure 8AM The CGR file composition user interface 801 is shown in response to detecting user input 899AG pointing to a location in the view area 803E. Figure 8AM 803E, the perspective of the virtual camera of view area 803E is moved closer to anchor representation 803EA, and view area 803E is enlarged, resulting in an increase in the size of anchor representation 803EA, the representation of saucer CGR object 806B, and the representation of cup CGR object 806C. The sizes of the saucer CGR object and the cup CGR object (as opposed to their representations in view area 803E) are unchanged, as indicated by the lack of change in second CGR scene affordance representation 803BB.

[0347] Figure 8AM User input 899AH is shown pointing to a location in view area 803E. In various embodiments, user input 899AH corresponds to a moving contact (e.g., a drag or touch and drag gesture) detected at a location in view area 803E. In various embodiments, the location in view area 803E is away from any representation of a CGR object (while the CGR object is selected or not selected). In various embodiments, the location is at the location of a representation of a CGR object that is not selected.

[0348] Figure 8AN The CGR file composition user interface 801 is shown in response to detecting user input 899AH pointing to a location in the view area 803E. Figure 8AN803E), the perspective of the virtual camera of view area 803E is moved to one side (e.g., perpendicular to a line from the virtual camera to the center of anchor representation 803EA), and view area 803E is moved, resulting in lateral movement of anchor representation 803EA, the representation of saucer CGR object 806B, and the representation of cup CGR object 806C. The positions of the saucer CGR object and the cup CGR object (as opposed to their representations in view area 803E) are unchanged, as indicated by the lack of changes to second CGR scene affordance representation 803BB.

[0349] Figure 8AN User input 899AI is shown pointing to a location in view area 803E. In various embodiments, user input 899AI corresponds to two contacts detected at a location in view area 803E moving about a common center (e.g., a rotation gesture). In various embodiments, the location in view area 803E is far from any representation of a CGR object (while the CGR object is selected or not). In various embodiments, the location is at the location of a representation of a CGR object that is not selected.

[0350] Figure 8AO The CGR file composition user interface 801 is shown in response to detecting user input 899AI pointing to a location in the view area 803E. Figure 8AO 803E, the perspective of the virtual camera of view area 803E rotates about the center of anchor representation 803EA and causes view area 803E to rotate, resulting in rotation of anchor representation 803EA, the representation of saucer CGR object 806B, and the representation of cup CGR object 806C. The position and orientation of the saucer CGR object and cup CGR object (as opposed to their representations in view area 803E) are unchanged, as indicated by the lack of changes to second CGR scene affordance representation 803BB.

[0351] Figure 8AO User input 899AJ directed to CGR view enable representation 803AF is shown. In various embodiments, user input 899AJ corresponds to a contact (e.g., a tap) detected at the location of CGR view enable representation 803AF.

[0352] Figure 8AP A CGR file composition user interface 801 is shown in response to detecting user input 899AJ pointing to a location in a CGR view affordance 803AF. Figure 8AP , the CGR view affordance 803AF is selected, and the object view affordance 803AE is deselected (as indicated by their different display styles). Figure 8AP, view area 803A includes a representation of a second CGR scene (including an anchor representation 803EA, a representation of a saucer CGR object 806B, and a representation of a cup CGR object 806C) displayed over a camera image of the scene captured by device 100.

[0353] Figure 8AQ The CGR file showing C in response to movement of the device 100 to the right constitutes the user interface 801. Figure 8AQ 806B, and the cup CGR object 806C. Thus, the representation of the second CGR scene moves to the left within the view area 803A (in addition to other perspective effects).

[0354] Figure 8AR The CGR file composition user interface 801 is shown in response to movement of the device backwards (e.g., away from a table of a physical environment). Figure 8AR 806B, and the cup CGR object 806C. Thus, the size of the representation of the second CGR scene (among other perspective effects) is reduced within the view area 803A.

[0355] Figure 8AR User input 899AK pointing to object view affordance 803AE is shown. In various embodiments, user input 899AK corresponds to a contact (e.g., a tap) detected at the location of object view affordance 803AE.

[0356] Figure 8AS The CGR file composition user interface 801 is shown in response to detecting user input 899AK pointing to a location in the object view affordance 803AE. Figure 8AS In , the object view affordance 803AE is selected, and the CGR view affordance 803AF is deselected (as indicated by their different display styles). Figure 8AS , the scene camera image is no longer displayed behind the representation of the second CGR scene.

[0357] Figure 8AS A user input 899AL is shown pointing to a representation of the cup CGR object 806C. In various embodiments, the user input 899AL corresponds to a contact (eg, a tap) detected at the location of the cup CGR object 806C.

[0358] Figure 8AT The CGR file composition user interface 801 is shown in response to detecting user input 899AL pointing to a representation of a cup CGR object 806C. Figure 8AT , the view area 803E includes a first type of object selection indicator 807 surrounding the representation of the cup CGR object 806C. The first type of object selection indicator 807 displayed surrounding the representation of the cup CGR object 806C indicates that the cup CGR object is selected.

[0359] exist Figure 8AT , when object setting enable indication 803AI is selected (as indicated by the different display of object setting enable indication 803AI compared to other setting enable indications 803AG, 803AH and 803AJ), setting area 803D includes multiple object setting manipulation enable indications presented via collapsible / expandable object setting menus 813DA to 813DD (wherein object name manipulation enable indication 813DAA, configurator enable indication 813DBA, style enable indication 813DBB and pattern enable indication 813DBC are in Figure 8AT ). As with the first type of object selection indicator 807 displayed surrounding the representation of the cup CGR object 806C, the object name manipulation affordance 813DAA displaying the name of the cup CGR object (e.g., “Cup01”) indicates that the cup CGR object is selected.

[0360] Figure 8AT User input 899AM is shown pointing to pattern affordance 813DBC. In various embodiments, user input 899AM corresponds to a contact (e.g., a tap) detected at the location of the representation of pattern affordance 813DBC.

[0361] Figure 8AU A CGR file is shown constituting a user interface 801 in response to detecting a user input 899AM of a pointing pattern affordance 813DBC. Figure 8AU In the example, the CGR file composition user interface 801 includes a real-time view user interface 811 in the form of a pop-up window.

[0362] The real-time view user interface 811 includes a plurality of view windows 811DA to 811DC, which respectively include a plurality of representations of the currently selected CGR objects 811EA to 811EC, each CGR object having a different pattern. Specifically, the real-time view user interface 811 includes: a first view window 811DA including a representation of a cup CGR object having a white pattern 811EA; a second view window 811DB including a representation of a cup CGR object having a star pattern 811EB; and a third view window 811DC including a representation of a cup CGR object having an ivy pattern 811EC. Figure 8AU , the second view window 811DB is displayed in a different manner (eg, with a thicker border) than the other view windows 811DA and 811DC, indicating that the star pattern is the currently selected pattern.

[0363] The real-time view user interface 811 includes a text description 811C of the currently selected CGR object (eg, “Cup 01 ”) and CGR object settings (eg, pattern) manipulated via the real-time view user interface 811 .

[0364] The real-time view user interface 811 includes multiple view-enabled representations 811A to 811B, including an object view-enabled representation 811A and a CGR view-enabled representation 811B, wherein the object view-enabled representation is used to view multiple representations of the currently selected CGR object 811EA to 811EC in its corresponding view window 811DA to 811DC in a scene mode (wherein user input directed to any one of the multiple view windows 811DA to 811DC changes the perspective of the view of all multiple representations of the currently selected CGR object 811EA to 811EC), and the CGR view-enabled representation is used to view multiple representations of the currently selected CGR object 811EA to 811EC in its corresponding view window 811DA to 811DC in a CGR mode (wherein movement of the device 100 changes the perspective of the view of all multiple representations of the currently selected CGR object 811EA to 811EC).

[0365] The real-time view user interface 811 includes a confirmation enable indication 811F, which is used to change the pattern of the currently selected CGR object to the currently selected style and to remove (stop displaying) the real-time view interface 811. The real-time view user interface 811 includes a cancel enable indication 811G, which is used to remove (stop displaying) the real-time view user interface 811, thereby returning the CGR file composition user interface 801 to Figure 8AT The state shown does not change the pattern of the currently selected CGR object.

[0366] Figure 8AU User input 899AN is shown directed toward first view window 811DA. In various embodiments, user input 899AN corresponds to two contacts detected at the location of first view window 811DA that move about a common center (eg, a rotation gesture).

[0367] Figure 8AV A CGR file composition user interface 801 is shown in response to detecting user input 899AN directed to the first view window 811DA. Figure 8AV , the perspective of the virtual camera used to render each of the multiple representations of the currently selected CGR objects 811EA to 811EC in their respective view windows 811DA to 811DC has changed, rotating around the z-axis. Therefore, the differences in the respective patterns can be easily seen and discerned.

[0368] Figure 8AV User input 899AO directed to the first view window 811DA is shown. In various embodiments, the user input 899AO corresponds to two contacts (eg, a pinch or spread gesture) detected at the location of the first view window 811DA that move toward or away from each other.

[0369] Figure 8AW A CGR file composition user interface 801 is shown in response to detecting user input 899AO directed to the first view window 811DA. Figure 8AW , the perspective of the virtual camera used to render each of the multiple representations of the currently selected CGR objects 811EA to 811EC in their respective view windows 811DA to 811DC has changed, moved closer to the representation, zoomed in and increased in size.

[0370] Figure 8AW User input 899AP is shown directed to third view window 811DC. In various embodiments, user input 899AP corresponds to a contact (eg, a tap) detected at the location of third view window 811DC.

[0371] Figure 8AX A CGR file composition user interface 801 is shown in response to detecting user input 899AP directed to the third view window 811DC. Figure 8AX , the third view window 811DC is displayed in a different manner (eg, with a thicker border) than the other view windows 811DA to 811DB, indicating that the ivy pattern is the currently selected pattern.

[0372] Figure 8AXUser input 899AQ is shown directed toward determine-enable representation 81 IF. In various embodiments, user input 899AQ corresponds to a contact (e.g., a tap) detected at the location of determine-enable representation 81 IF.

[0373] Figure 8AY A CGR file composition user interface 801 is shown in response to detecting user input 899AQ pointing to an affordance 811F. Figure 8AY , the real-time view user interface 811 is dismissed, and the pattern of the currently selected CGR object is changed to an ivy pattern, such as the appearance change of the representation of the cup CGR object 806C (and, in Figure 8AY As can be more easily seen in the figure, the second CGR scene indication indicates the corresponding change of 803BB) and the pattern indication indicates the change of 813DBC.

[0374] Figure 8AY User input 899AR pointing to CGR object add enable representation 803AB is shown. In various embodiments, user input 899AR corresponds to a contact (e.g., a tap) detected at the location of CGR object add enable representation 803AB.

[0375] Figure 8AZ A CGR file composition user interface 801 is shown in response to detecting user input 899AY pointing to a CGR object to add an affordance 803AB. Figure 8AZ In the example, the CGR file composition user interface 801 includes a media library user interface 804 in the form of a pop-up window.

[0376] Figure 8AZ User input 899AS is shown directed to sphere add affordance 805C. In various embodiments, user input 899AC corresponds to a contact (e.g., a tap) detected at the location of sphere add affordance 805C.

[0377] Figure 8BA A CGR file composition user interface 801 is shown in response to detecting user input 899AS pointing to add an affordance 805B to the sphere. Figure 8BA , the media library user interface 804 stops displaying, and the view area 803E includes a representation of a sphere CGR object 806D displayed at a specific position above the representation of the anchor 803EA. In connection therewith, the second CGR scene affordance 803BB has changed to indicate that the second CGR scene includes the sphere CGR object.

[0378] exist Figure 8BA8, the view area 803E also includes a first type of object selection indicator 807 surrounding the representation of the sphere CGR object 806D. The first type of object selection indicator 807 displayed surrounding the representation of the sphere CGR object 806D indicates the sphere CGR object.

[0379] exist Figure 8BA In the figure, the setting area 803D is changed to display object setting manipulation enable indications for changing the object setting to the currently selected CGR object, the sphere CGR object (including the object name manipulation enable indication 813DAA indicating "sphere 01", the style enable indication 813DBB indicating the sphere style, and the pattern enable indication 813DBC indicating a white pattern).

[0380] Figure 8BA User input 899AT pointing to style affordance 813DBB is shown. In various embodiments, user input 899AT corresponds to a contact (e.g., a tap) detected at the location of style affordance 813DBB.

[0381] Figure 8BB A CGR file composition user interface 801 is shown in response to detecting user input 899AT of a pointing style affordance 813DBB. Figure 8BB In the example, the CGR file composition user interface 801 includes a real-time view user interface 811 in the form of a pop-up window.

[0382] exist Figure 8BB , the text description 811C of the real-time view user interface 811 indicates that the currently selected CGR object is “Sphere 01 ”, and the CGR object setting manipulated via the real-time view user interface 811 is the style.

[0383] exist Figure 8BB, the real-time view user interface 811 includes a plurality of view windows 821DA to 821DF, each of which includes a plurality of representations of currently selected CGR objects 821EA to 821EF, each CGR object having a different style. Specifically, the real-time view user interface 811 includes: a first view window 821DA, which includes a representation of a sphere CGR object having a sphere style 821EA; a second view window 821DB, which includes a representation of a sphere CGR object having a golf style 821EB; a third view window 821DC, which includes a representation of a sphere CGR object having a baseball style 821EC; a fourth view window 821DD, which includes a representation of a sphere CGR object having a basketball style 821ED; a fifth view window 821DE, which includes a representation of a sphere CGR object having a tennis style 821EE; and a sixth view window 821DF, which includes a representation of a sphere CGR object having a bowling style 821EF.

[0384] In various embodiments, appropriate user input directed to any of the multiple representations of view windows 821DA through 821DF when in object view mode, or movement of device 100 when in CGR view mode, will advantageously reveal the following differences: the representation of a sphere CGR object having sphere style 821EA versus a bowling ball style 821EF (which is included in the Figure 8BB The difference between the representation of a sphere CGR object with holes that are not visible in the viewing angle.

[0385] exist Figure 8BB , the first view window 821DA is displayed in a different manner (eg, with a thicker border) than the other view windows 821DB to 821DF, indicating that the sphere style is the currently selected style.

[0386] Figure 8BB A user input 899AU directed to the second view window 821DB is shown. In various embodiments, the user input 899AU corresponds to a contact (eg, a tap) detected at the location of the second view window 821DB.

[0387] Figure 8BC The CGR file composition user interface 801 is shown in response to detecting user input 899AU directed to the second view window 821DB. Figure 8BC , the second view window 821DB is displayed in a different manner (eg, with a thicker border) than the other view windows 821DA and 821DC to 821DF, indicating that the golf ball style is the currently selected style.

[0388] Figure 8BC User input 899AV is shown directed toward determine-enablement representation 81 IF. In various embodiments, user input 899AV corresponds to a contact (e.g., a tap) detected at the location of determine-enablement representation 81 IF.

[0389] Figure 8BD A CGR file composition user interface 801 is shown in response to detecting user input 899AV pointing to an affordance 811F. Figure 8AY , the real-time view user interface 811 is dismissed and the style of the currently selected CGR object is changed to a golf ball style, as indicated by the change in the appearance of the representation of the sphere CGR object 806D (and the corresponding change in the second CGR scene enabling representation 803BB) and the change in the style enabling representation 813DBC.

[0390] Figure 8BD User input 899AW is shown directed to physical properties menu 813DC. In various embodiments, user input 899AW corresponds to a contact (eg, a tap) detected at the location of physical properties menu 813DC.

[0391] Figure 8BE A CGR file composition user interface 801 is shown in response to detecting user input 899AW directed to a physical properties menu 813DC. Figure 8BE , the physical properties menu 813DC is expanded (within the settings area 803D) to present a physical mesh enable indication 813DCA for changing the physical mesh of the currently selected CGR object (described below), a reflectivity enable indication 813DCB for changing the reflectivity of the currently selected CGR object via a slider, and an elasticity enable indication 813DCC for changing the elasticity of the currently selected CGR object.

[0392] Figure 8BE User input 899AX directed toward physical grid affordance 813DCA is shown. In various embodiments, user input 899AX corresponds to a contact (e.g., a tap) detected at the location of physical grid affordance 813DCA.

[0393] Figure 8BF A CGR file composition user interface 801 is shown in response to detecting user input 899AX directed to a physical grid affordance 813DCA. Figure 8BF In the example, the CGR file configuration user interface 801 includes a setting value selection user interface 812 in the form of a pop-up window.

[0394] The settings selection user interface 812 includes a text description 812A of the currently selected CGR object (eg, “Sphere 01 ”) and the CGR object settings (eg, physics mesh) that are manipulated via the settings selection user interface 812 .

[0395] The setting value selection user interface 812 includes a plurality of setting value selection enable indications 812BA to 812BC for selecting a setting value. The setting value selection user interface 812 includes a sphere enable indication 812BA for selecting a sphere grid as the currently selected setting value, a golf ball enable indication 812BB for selecting a golf ball grid as the currently selected setting value, and a load enable indication 812BC for loading a custom grid as the currently selected setting value.

[0396] Golf ball enable representation 812BB includes a selection indicator 812C indicating that the golf ball grid is the currently selected setting.

[0397] The setting value selection user interface 812 includes a confirmation enable indication 812D, which is used to change the value of the CGR object setting of the currently selected CGR object to the currently selected setting value and to cancel (stop displaying) the setting value selection user interface 812. The setting value selection user interface 812 includes a cancel enable indication 812E, which is used to cancel (stop displaying) the setting value selection user interface 812, thereby returning the CGR file composition user interface 801 to Figure 8BE , without changing the value of the CGR object setting of the currently selected CGR object.

[0398] In various situations, it may be desirable that the physical mesh of a CGR object (which is used to determine the interaction of the CGR object with other CGR objects) is different from the display mesh of the CGR object (which is used to render the CGR object on a display). For example, in the case of a golf ball CGR object, the display of the CGR golf ball object may be based on a complex display mesh that includes a large number of dimples and, therefore, a large number of polygons. However, using such a mesh as a physical mesh to calculate the physical interactions of the golf ball CGR object may be computationally intensive and time consuming (potentially preventing real-time calculations). By selecting a simpler physical mesh, such as a sphere mesh with fewer polygons, the results of the physical interactions may be calculated more efficiently and quickly.

[0399] Figure 8BF User input 899AY is shown directed toward spherical enable representation 812BA. In various embodiments, user input 899AY corresponds to a contact (e.g., a tap) detected at the location of spherical enable representation 812BA.

[0400] Figure 8BG A CGR file is shown constituting user interface 801 in response to detecting user input 899AY pointing to spherical affordance 812BA. Figure 8BG , selection indicator 812C is displayed in association with ball enable indication 812BA (rather than golf ball enable indication 812BB).

[0401] Figure 8BG User input 899AZ is shown pointing to determined enable representation 812D. In various embodiments, user input 899AZ corresponds to a contact (e.g., a tap) detected at the location of determined enable representation 812D.

[0402] Figure 8BH A CGR file composition user interface 801 is shown in response to detecting user input 899AZ pointing to an affordance 812D. Figure 8BH In the example, the physical grid indication 813DCA is updated to indicate that the physical grid of the spherical CGR object is a spherical grid.

[0403] Figure 8BH User input 899BA pointing to CGR scene add affordance 803AA is shown. In various embodiments, user input 899BA corresponds to a contact (e.g., a tap) detected at the location of CGR scene add affordance 803AA.

[0404] Figure 8BI A CGR file constructing user interface 801 is shown in response to detecting user input 899BA pointing to a CGR scene adding an enable representation 803AA and further user input, wherein the further user input is used to select an anchor for a third CGR scene created by user input 899BA, rename the third CGR scene (e.g., to "Tea Time 2"), populate the third CGR scene with multiple CGR objects, manipulate CGR object settings of the multiple CGR objects, and reselect the second CGR scene as the currently selected CGR scene, etc.

[0405] Therefore, in Figure 8BI In the example, the CGR file constitutes a user interface including a third CGR scene indicator 803BC representing a third CGR scene.

[0406] Figure 8BI User input 899BB pointing to behavior-adding affordance 803AC is shown. In various embodiments, user input 899BB corresponds to a contact (e.g., a tap) detected at the location of behavior-adding affordance 803AC.

[0407] Figure 8BJA CGR file composition user interface 801 is shown adding user input 899BB of affordance 803AC in response to detecting pointing behavior. Figure 8BJ In the example, the CGR file constitutes the user interface 801 including a behavior editor user interface 831 in the form of a pop-up window.

[0408] In various embodiments, the behavior editor user interface 831 is used to create and / or edit behaviors associated with CGR scenarios. Each behavior is associated with one or more triggers and one or more actions. When a CGR application corresponding to a project is presented and any one of the one or more triggers is detected, one or more actions are performed.

[0409] The behavior editor user interface 831 includes a view window 831D that presents a representation of the currently selected CGR scene (including a representation of a cup CGR object 831DA, a representation of a saucer CGR object 831DB, and a representation of a sphere CGR object 831DC). The behavior editor user interface 831 includes a name 831C (e.g., "Behavior 01") of the behavior being edited (or created) (e.g., the first behavior) displayed above the view window 831D.

[0410] The behavior editor user interface 831 includes multiple view enable representations 831A to 831B, which include an object view enable representation 831A and a CGR view enable representation 831B, wherein the object view enable representation is used to view a representation of the currently selected CGR scene in the view window 831D in scene mode (where user input directed to the view window 831D changes the perspective of the view of the currently selected CGR scene), and the CGR view enable representation is used to view the currently selected CGR scene in the view window 831D in CGR mode (where movement of the device 100 changes the perspective of the view of the currently selected CGR object).

[0411] The behavior editor user interface 831 includes a confirmation enable indication 831E, which is used to create a behavior (and / or accept a change to a behavior) and to remove (stop displaying) the behavior editor user interface 831. The behavior editor user interface 831 includes a cancel enable indication 831F, which is used to remove (stop displaying) the behavior editor user interface 831, thereby returning the CGR file composition user interface 801 to Figure 8BI The state shown does not create (or change) the behavior.

[0412] Behavior editor user interface 831 includes an overview area 832 and a details area 837. Overview area 832 includes a trigger area 833 and an action area 835 separated by a divider 834. Trigger area 833 includes a new trigger enable indication 833A for adding a trigger to a behavior, and action area 835 includes a new action enable indication 835A for adding a new action to a behavior.

[0413] Details region 837 includes a plurality of behavior setting affordances 836A to 836C for changing the settings of the behavior. Details region 837 includes: a name affordance 836A for changing the name of the behavior, a color affordance 836B for changing the color of the behavior, and an enable / disable affordance 836C for switching between enabling and disabling the behavior.

[0414] Figure 8BJ User input 899BC is shown directed to name affordance 836A. In various embodiments, user input 899BC corresponds to a contact (e.g., a tap) detected at the location of name affordance 836A.

[0415] Figure 8BK A CGR file is shown constituting user interface 801 in response to detecting user input 899BC directed to name affordance 836A (and possibly additional user input defining a new name for the first behavior). Figure 8BK , the name indicator 836A is updated to indicate the new name of the first action (e.g., "Move the Ball").

[0416] Figure 8BK User input 899BD is shown directed to new trigger enable representation 833A. In various embodiments, user input 899BD corresponds to a contact (e.g., a tap) detected at the location of new trigger enable representation 833A.

[0417] Figure 8BL A CGR file composition user interface 801 is shown in response to detecting user input 899BD directed to a new trigger enable representation 833A. Figure 8BL In the example, the new trigger indicates that 833A can be used with Figure 8BK The detail area 837 is displayed in a second manner that is different from the first manner (e.g., black instead of white) to indicate that the detail area 837 includes an enable representation for creating a new trigger for the behavior.

[0418] For example, in Figure 8BL, detail area 837 includes a tap trigger affordance 838A for creating a tap trigger, such that when a CGR application corresponding to an item is presented and a tap on a CGR object (or any of a plurality of CGR objects) defined by the tap trigger is detected, one or more actions of the behavior are performed. Detail area 837 includes a proximity / collision trigger affordance 838B for creating a proximity / collision trigger, such that when a CGR application corresponding to an item is presented and a CGR object (or any of a plurality of CGR objects) defined by the trigger as a primary target is detected within a threshold distance (also defined by the trigger, which may be 0) of a CGR object (or any of a plurality of CGR objects) defined by the trigger as a secondary target, one or more actions of the behavior are performed. Detail area 837 includes a facial tracking trigger affordance 838C for creating a facial tracking trigger, such that when a CGR application corresponding to an item is presented and a facial function of the user (e.g., blinking, chewing, blowing, smiling, etc.) is detected, one or more actions of the behavior are performed. The details area 837 includes other trigger enable indications 838D for creating other types of triggers.

[0419] Figure 8BL User input 899BE is shown directed to tap trigger enable representation 838A. In various embodiments, user input 899BE corresponds to a contact (e.g., a tap) detected at the location of tap trigger enable representation 838A.

[0420] Figure 8BM A CGR file is shown constituting user interface 801 in response to detecting user input 899BE pointing to tap trigger affordance 838A. Figure 8BL , trigger area 833 includes a tap trigger enable indication 833B displayed in a second manner to indicate that detail area 837 includes an enable indication for creating a new tap trigger for the behavior.

[0421] For example, in Figure 8BM , the detail area 837 includes a completion enable indication 838AA for creating a tap trigger so that when a CGR application corresponding to the project is presented and a tap on the CGR object (or any one of multiple CGR objects) selected in the view window 831D is detected, one or more actions of the behavior are performed.

[0422] Figure 8BM A user input 899BF pointing to a representation of the sphere CGR object 831DC is shown. In various embodiments, the user input 899BF corresponds to a contact (eg, a tap) detected at the location of the representation of the sphere CGR object 831DC.

[0423] Figure 8BN The CGR file composition user interface 801 is shown in response to detecting user input 899BF pointing to a representation of a spherical CGR object 831DC. Figure 8BN , the representation of the sphere CGR object 831DC is highlighted, indicating that the sphere CGR object is selected for definition as a target for a tap trigger.

[0424] Figure 8BN User input 899BG pointing to completion enable representation 838AA is shown. In various embodiments, user input 899BG corresponds to a contact (e.g., a tap) detected at the location of completion enable representation 838AA. In response, a first behavior is associated with a tap trigger so that when a CGR application corresponding to the project is presented and a tap on the sphere CGR object is detected, one or more actions of the first behavior are performed.

[0425] Figure 8BO A CGR file composition user interface 801 is shown in response to detecting user input 899BG pointing to completion affordance 838AA. Figure 8BO , the new trigger enable indication 833A (and the tap trigger enable indication 833B) are displayed in a first manner to indicate that the detail area 837 includes multiple behavior setting enable indications 836A to 836C.

[0426] Figure 8BO User input 899BH is shown directed to new trigger enable representation 833A. In various embodiments, user input 899BH corresponds to a contact (e.g., a tap) detected at the location of new trigger enable representation 833A.

[0427] Figure 8BP A CGR file composition user interface 801 is shown in response to detecting user input 899BH directed to a new trigger enable representation 833A. Figure 8BP In the example, the new trigger indicates that 833A can be used with Figure 8BO 837 is displayed in a second manner that is different from the first manner (e.g., black instead of white) to indicate that detail region 837 includes an affordance for creating a new trigger for the behavior. Figure 8BL In the figure, the detail area 837 includes a facial tracking trigger enable indication 838C, etc.

[0428] Figure 8BP User input 899BI directed toward face tracking trigger enable indication 838C is shown. In various embodiments, user input 899BI corresponds to a contact (e.g., a tap) detected at the location of face tracking trigger enable indication 838C.

[0429] Figure 8BQA CGR file composition user interface 801 is shown in response to detecting user input 899BI directed toward a facial tracking trigger affordance 838C. Figure 8BQ , trigger area 833 includes a facial tracking trigger enable indication 833C displayed in a second manner to indicate that detail area 837 includes an enable indication for creating a new facial tracking trigger for the behavior.

[0430] For example, in Figure 8BQ , detail area 837 includes a blink trigger enable indication 838CA for creating a blink trigger, so that when a CGR application corresponding to an item is presented and a blink of the user is detected, one or more actions of the behavior are performed. Detail area 837 includes a chew trigger enable indication 838CB for creating a chew trigger, so that when a CGR application corresponding to an item is presented and a chewing of the user is detected, one or more actions of the behavior are performed. Detail area 837 includes a blow trigger enable indication 838CC for creating a blow trigger, so that when a CGR application corresponding to an item is presented and a blowing of the user is detected, one or more actions of the behavior are performed. Detail area 837 includes other facial tracking trigger enable indications 838CD for creating other facial tracking triggers.

[0431] Figure 8BQ User input 899BJ directed to blow trigger enable representation 838CC is shown. In various embodiments, user input 899BJ corresponds to a contact (e.g., a tap) detected at the location of blow trigger enable representation 838CC. In response, a first behavior is associated with the blow trigger, such that when a CGR application corresponding to the item is presented and a user is detected performing a blow action (e.g., like blowing out a birthday candle), one or more actions of the first behavior are performed.

[0432] Figure 8BR A CGR file is shown constituting user interface 801 in response to detecting user input 899BJ directed to the blow trigger affordance 838CC. Figure 8BR , the new trigger enable indication 833A (and the facial tracking trigger enable indication 833C) are displayed in a first manner to indicate that the detail area 837 includes multiple behavior setting enable indications 836A to 836C.

[0433] Figure 8BR User input 899BK is shown directed to new action-enabled representation 835A. In various embodiments, user input 899BK corresponds to a contact (e.g., a tap) detected at the location of new action-enabled representation 835A.

[0434] Figure 8BSA CGR file composition user interface 801 is shown in response to detecting user input 899BK directed to a new action-enabling representation 835A. Figure 8BS , trigger area 833 is collapsed to include a single grouped trigger enable indication 833X representing the new trigger enable indication 833A and any defined triggers (e.g., tap trigger enable indication 833B and face tracking trigger enable indication 833C). Figure 8BS , the new action enable indication 835A is displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating a new action for the behavior.

[0435] For example, in Figure 8BS , detail area 837 includes a move action enable representation 839A for creating a move action, so that when a CGR application corresponding to the project is presented and a trigger for a behavior including a move action is detected, the CGR target moves in the CGR environment. Detail area 837 includes an audio action enable representation 839B for creating an audio action, so that when a CGR application corresponding to the project is presented and a trigger for a behavior including an audio action is detected, an audio file is played in the CGR environment. Detail area 837 includes a scene change action enable representation 839C for creating a scene change action, so that when a CGR application corresponding to the project is presented and a trigger for a behavior including a scene change action is detected, a new CGR scene (replacing or supplementing the current CGR scene) is presented in the CGR environment. Detail area 837 includes other action enable representations 839D for creating other types of actions.

[0436] Figure 8BS User input 899BL is shown pointing to move action affordance 839A. In various embodiments, user input 899BL corresponds to a contact (e.g., a tap) detected at the location of move action affordance 839A.

[0437] Figure 8BT A CGR file is shown constituting user interface 801 in response to detecting user input 899BL of pointing movement action affordance 839A. Figure 8BT , the action area 835 includes a move action enable representation 835B displayed in a second manner to indicate that the detail area 837 includes an enable representation for creating a new move action for the behavior.

[0438] For example, in Figure 8BT, the detail area 837 includes a move-in action enable representation 839AA for creating a move-in action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a move-in action is detected, the CGR target moves to a position in the CGR environment defined by the action. The detail area 837 includes a move-away action enable representation 839AB for creating a move-away action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a move-away action is detected, the CGR target moves away from its position in the CGR environment in a manner defined by the action. The detail area 837 includes a spin action enable representation 839AC for creating a spin action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a spin action is detected, the CGR target spins at its position in the CGR environment. The detail area 837 includes other move action enable representations 839AD for creating other types of move actions.

[0439] Figure 8BT User input 899BM directed toward move-action-enable representation 839AB is shown. In various embodiments, user input 899BM corresponds to a contact (e.g., a tap) detected at the location of move-action-enable representation 839AB.

[0440] Figure 8BU A CGR file is shown constituting user interface 801 in response to detecting user input 899BM indicating 838AB pointing away. Figure 8BU , detail area 837 includes an enabling representation for creating a new removal action for the behavior.

[0441] For example, in Figure 8BU , the detail area 837 includes a forward move action enable representation 839ABA for creating a forward move action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a forward move action is detected, the CGR target moves forward in the CGR environment. The detail area 837 includes a bouncing action enable representation 839ABB for creating a bouncing action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a bouncing action is detected, the CGR target moves upward from its position in the CGR environment. The detail area 837 includes a descending action enable representation 839ABC for creating a descending action, so that when a CGR application corresponding to the item is presented and a trigger for a behavior including a descending action is detected, the CGR target descends from its position in the CGR environment. The detail area 837 includes other move away action enable representations 839ABD for creating other types of move away actions.

[0442] Figure 8BUUser input 899BN pointing to forward move enable indication 839ABA is shown. In various embodiments, user input 899BN corresponds to a contact (e.g., a tap) detected at the location of forward move enable indication 839ABA.

[0443] Figure 8BV The CGR file composition user interface 801 is shown in response to detecting user input 899BN pointing forward as an indication 838ABA. Figure 8BV , the detail area 837 includes a speed setting enable indication 839ABAA for setting the speed at which the CGR object moves forward and an interaction setting enable indication 839ABAB for enabling or disabling interaction with other CGR objects when the CGR object moves forward. The detail area 837 includes a save enable indication 839ABAC for saving the forward movement action to the first behavior (or saving the change to the forward movement action) and a cancel enable indication 839ABAD for canceling the forward movement action (or canceling the change to the forward movement action).

[0444] exist Figure 8BV , the representation of the sphere CGR object 831DC is highlighted, indicating that the sphere CGR object is the CGR object that moves during the action. By directing user input to the representation of other CGR objects, the CGR object that moves during the action can be changed.

[0445] Figure 8BV User input 899BO pointing to save enable representation 839ABAC is shown. In various embodiments, user input 899BO corresponds to a contact (e.g., a tap) detected at the location of save enable representation 839ABAC.

[0446] Figure 8BW A CGR file composition user interface 801 is shown in response to detecting user input 899BO directed to save the affordance 839ABAC. Figure 8BW In the figure, the new action enable indication 835A and the move action enable indication 835B are displayed in a first manner, and the indication detail area 837 includes a plurality of behavior setting enable indications 836A to 836C for changing the settings of the behavior.

[0447] Figure 8BW User input 899BP is shown pointing to determine enable representation 831E. In various embodiments, user input 899BP corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 831E.

[0448] Figure 8BX A CGR file composition user interface 801 is shown in response to detecting user input 899BP pointing to an indication 831E. Figure 8BX , the behavior editor user interface 831 is removed and the first behavior enable indication 803CA (having the color indicated by the first behavior) is displayed in the behavior area 803C.

[0449] exist Figure 8BX , the first behavior is selected, as indicated by the first behavior enable representation 803CA being displayed in a selected manner (e.g., having a thicker border than in an unselected manner). Because the first behavior is selected, the representation of the second CGR scene in the view area 803E indicates the CGR objects associated with the trigger of the first behavior using a first highlight (e.g., a glow) and indicates the CGR objects associated with the actions of the first behavior using a second highlight (e.g., a diagonal hash). Thus, the representation of the sphere CGR object 806D is displayed with both the first highlight (because tapping the sphere CGR object is the trigger of the first behavior) and the second highlight (because moving the sphere CGR object is the action of the first behavior).

[0450] Figure 8BX User input 899BQ directed to behavior-adding affordance 803AC is shown. In various embodiments, user input 899BQ corresponds to a contact (e.g., a tap) detected at the location of behavior-adding affordance 831AC.

[0451] Figure 8BY A CGR file composition user interface 801 is shown adding user input 899BQ of affordance 803AC in response to detecting pointing behavior. Figure 8BY In the example, the CGR file composition user interface 801 includes a behavior editor user interface 831 in the form of a pop-up window. The behavior editor user interface 831 includes a name 831C (e.g., "behavior 02") of the behavior (e.g., the second behavior) being edited (or created) displayed above the view window 831D.

[0452] Figure 8BY User input 899BR is shown pointing to name affordance 836A. In various embodiments, user input 899BR corresponds to a contact (e.g., a tap) detected at the location of name affordance 836A.

[0453] Figure 8BZ A CGR file is shown constituting user interface 801 in response to detecting user input 899BR directed to name affordance 836A (and possibly additional user input defining a new name for a second behavior). Figure 8BZ , the name indicator 836A is updated to indicate the new name of the second behavior (e.g., "Ball in the Cup").

[0454] Figure 8BZUser input 899BS is shown pointing to color-enable representation 836B. In various embodiments, user input 899BS corresponds to a contact (e.g., a tap) detected at the location of color-enable representation 836B.

[0455] Figure 8CA The CGR file composition user interface 801 is shown in response to detecting user input 899BS pointing to color affordance 836B (and possibly additional user input selecting a new color for a second behavior). Figure 8CA , the color indicator 836B is updated to indicate the new color of the second behavior (e.g., "ball in the cup").

[0456] Figure 8CA User input 899BT directed to new trigger enable representation 833A is shown. In various embodiments, user input 899BT corresponds to a contact (e.g., a tap) detected at the location of new trigger enable representation 833A.

[0457] Figure 8CB A CGR file composition user interface 801 is shown in response to detecting user input 899BD directed to a new trigger enable representation 833A. Figure 8CB , new trigger enable representation 833A is displayed in a second manner (e.g., black) to indicate that detail area 837 includes an enable representation for creating a new trigger for the behavior. Figure 8CB , detail area 837 includes: a tap trigger enable indication 838A for creating a tap trigger, a proximity / collision trigger enable indication 838B, a face tracking trigger enable indication 838C, and other trigger enable indications 838D for creating other types of triggers.

[0458] Figure 8CB User input 899BU directed toward proximity / bump trigger enable representation 838B is shown. In various embodiments, user input 899BE corresponds to a contact (e.g., a tap) detected at the location of proximity / bump trigger enable representation 838B.

[0459] Figure 8CC A CGR file is shown constituting user interface 801 in response to detecting user input 899BU directed to proximity / collision trigger affordance 838B. Figure 8CC , trigger area 833 includes a proximity / collision trigger enable indication 833D displayed in a second manner to indicate that detail area 837 includes an enable indication for creating a new proximity / collision trigger for the behavior.

[0460] For example, in Figure 8CC, the detail area 837 includes a distance-to-grid trigger enable representation 838BA for creating a distance-to-grid trigger such that when a CGR application corresponding to the project is presented and any point of the grid of the CGR object (or any of the multiple CGR objects) defined as a primary target by the trigger is detected within a threshold distance (also defined by the trigger, which may be 0) of any point of the grid of the CGR object (or any of the multiple CGR objects) defined as a secondary target by the trigger, one or more actions of a behavior are performed.

[0461] The detail area 837 includes a distance-to-center trigger enable representation 838BB for creating a distance-to-center trigger such that when a CGR application corresponding to the project is presented and a CGR object (or any one of multiple CGR objects) defined as a primary target by the trigger is detected within a threshold distance (also defined by the trigger, which may be 0) of the center (or other spatial manipulation point) of the CGR object (or any one of multiple CGR objects) defined as a secondary target by the trigger, one or more actions of a behavior are performed.

[0462] The detail area 837 includes a distance-to-coordinate trigger enable representation 838BC for creating a distance-to-coordinate trigger such that when a CGR application corresponding to the project is presented and a CGR object (or any one of multiple CGR objects) defined by the trigger as a primary target is detected within a threshold distance (also defined by the trigger, which may be 0) of the coordinates defined by the trigger as a secondary target, one or more actions of a behavior are performed.

[0463] Details area 837 includes other proximity / collision trigger enable representations 838BD for creating other types of proximity / collision triggers.

[0464] Figure 8CC User input 899BV pointing to distance-from-center enable indication 838BB is shown. In various embodiments, user input 899BV corresponds to a contact (e.g., a tap) detected at a location away from center distance trigger enable indication 838BB.

[0465] Figure 8CD A CGR file composition user interface 801 is shown in response to detecting user input 899BV pointing to a distance-from-center trigger indication 838BB. Figure 8CD, the detail area 837 includes a completion enable indication 838BBA for completing the selection of the primary goal, so that when a CGR application corresponding to the project is presented and the CGR object (or any one of multiple CGR objects) selected in the view window 831D is detected within a threshold distance (also defined by the trigger, which may be 0) of the CGR object (or any one of multiple CGR objects) defined as a secondary goal by the trigger, one or more actions of the behavior are performed.

[0466] Figure 8CD A user input 899BW pointing to a representation of the sphere CGR object 831DC is shown. In various embodiments, the user input 899BW corresponds to a contact (eg, a tap) detected at the location of the representation of the sphere CGR object 831DC.

[0467] Figure 8CE The CGR file composition user interface 801 is shown in response to detecting user input 899BW pointing to a representation of a spherical CGR object 831DC. Figure 8CE , the representation of the sphere CGR object 831DC is highlighted, indicating that the sphere CGR object is selected for definition as the primary target for the proximity / collision trigger.

[0468] Figure 8CE User input 899BX is shown pointing to completion enable representation 838BBA. In various embodiments, user input 899BX corresponds to a contact (e.g., a tap) detected at the location of completion enable representation 838BBA.

[0469] Figure 8CF A CGR file composition user interface 801 is shown in response to detecting user input 899BX pointing to completion affordance 838BBA. Figure 8CF , the detail area 837 includes a completion enable indication 838BBB for completing the selection of the secondary goal, so that when a CGR application corresponding to the project is presented and a CGR object (or any one of multiple CGR objects) defined as the primary goal by the trigger is detected within a threshold distance (also defined by the trigger, which may be 0) of the CGR object (or any one of multiple CGR objects) selected in the view window 831D, one or more actions of the behavior are performed.

[0470] Figure 8CF A user input 899BY pointing to a representation of a cup CGR object 831DA is shown. In various embodiments, the user input 899BY corresponds to a contact (eg, a tap) detected at the location of the representation of the cup CGR object 831DA.

[0471] Figure 8CGThe CGR file composition user interface 801 is shown in response to detecting user input 899BY pointing to a representation of a cup CGR object 831DA. Figure 8CG , the representation of the cup CGR object 831DA is highlighted, indicating that the cup CGR object is selected for definition as a secondary target for the proximity / collision trigger.

[0472] Figure 8CG User input 899BZ pointing to completion enable indication 838BBB is shown. In various embodiments, user input 899BZ corresponds to a contact (e.g., a tap) detected at the location of completion enable indication 838BBB. Figure 8CG , detail area 837 includes a distance setting enable indication 838BBC for setting a threshold distance for a proximity / collision trigger. A proximity trigger is created by setting the threshold distance to a number greater than zero. A collision trigger is created by setting the threshold distance to zero. Detail area 837 includes a volumetric trigger setting enable indication 838BBD for enabling or disabling the volumetric trigger setting for the proximity / collision trigger. When the volumetric trigger setting is enabled, when a CGR application corresponding to the project is presented and any point of the grid of the CGR object (or any of the multiple CGR objects) defined as a primary target by the trigger is detected within a threshold distance (also defined by the trigger) of the center (or other spatial manipulation point) of the CGR object (or any of the multiple CGR objects) defined as a secondary target, one or more actions of the behavior are performed. When the volumetric trigger setting is disabled, when a CGR application corresponding to the project is presented and the center (or other spatial manipulation point) of the CGR object (or any of multiple CGR objects) defined as the primary target by the trigger is detected within a threshold distance (also defined by the trigger) of the center (or other spatial manipulation point) of the CGR object (or any of multiple CGR objects) defined as the secondary target, one or more actions of the behavior are performed.

[0473] The detail area 837 includes a save enable indication 838BBE for saving the proximity / collision trigger to the second behavior (or saving changes to the proximity / collision trigger) and a remove enable indication 838BBF for removing the proximity / collision trigger (or removing changes to the proximity / collision trigger).

[0474] Figure 8CH User input 899CA pointing to distance setting enable representation 838BBC is shown. In various embodiments, user input 899CA corresponds to a contact (e.g., a tap) detected at the location of the representation of distance setting enable representation 838BBC.

[0475] Figure 8CIA CGR file is shown constituting user interface 801 in response to detecting user input 899CA pointing to distance setting affordance 838BBC (and possibly additional user input for setting the distance setting to zero). Figure 8CI In the display, the distance setting indicator shows that the 838BBC indicates that the distance setting has been set to zero.

[0476] Figure 8CI User input 899CB is shown pointing to save enable representation 838BBE. In various embodiments, user input 899CB corresponds to a contact (e.g., a tap) detected at the location of save enable representation 838BBE. In response, a second behavior is associated with the collision trigger so that when a CGR application corresponding to the item is presented and any point of the grid of the sphere CGR object is detected at the location of the center of the cup CGR object, one or more actions of the second behavior are performed.

[0477] Figure 8CJ A CGR file composition user interface 801 is shown in response to detecting user input 899CB directed to save affordance 838BBE. Figure 8CJ , the new trigger enable indication 833A (and the proximity / collision trigger enable indication 833D) are displayed in a first manner, indicating that the detail area 837 includes multiple behavior setting enable indications 836A to 836C for changing the settings of the behavior.

[0478] Figure 8CJ User input 899CC is shown directed to new action-enabled representation 835A. In various embodiments, user input 899CC corresponds to a contact (e.g., a tap) detected at the location of the representation of new action-enabled representation 835A.

[0479] Figure 8CK A CGR file composition user interface 801 is shown in response to detecting user input 899CC directed to a new action affordance 835A. Figure 8CK , the new action enable indication 835A is displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating a new action for the behavior.

[0480] For example, in Figure 8CK , detail area 837 includes a move action enable indication 839A, an audio action enable indication 839B, a scene change action enable indication 839C, and other action enable indications 839D for creating other types of actions.

[0481] Figure 8CKUser input 899CD directed to audio action affordance 839B is shown. In various embodiments, user input 899CD corresponds to a contact (e.g., a tap) detected at the location of audio action affordance 839B.

[0482] Figure 8CL A CGR file composition user interface 801 is shown in response to detecting user input 899CD directed to audio action affordance 839B. Figure 8CK , the new action enable indication 835A is displayed in a first manner, and the action area 835 includes an audio action enable indication 835C displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating an audio action.

[0483] For example, detail area 837 includes ambient audio action affordances 839BA for creating ambient audio actions, such that when a CGR application corresponding to an item is presented and a trigger for an action is detected, an audio file is played omnidirectionally. Detail area 837 includes directional audio action affordances 839BB for creating directional audio actions, such that when a CGR application corresponding to an item is presented and a trigger for an action is detected, an audio file is played from a specific point in the CGR environment. Detail area 837 includes other audio action affordances 839BC for creating other types of audio actions.

[0484] Figure 8CL User input 899CE directed toward ambient audio action affordance 839BB is shown. In various embodiments, user input 899CK corresponds to a contact (e.g., a tap) detected at the location of ambient audio action affordance 839BB.

[0485] Figure 8CM A CGR file composition user interface 801 is shown in response to detecting user input 899CE directed toward ambient audio action affordance 839BA. Figure 8CM Detail area 837 includes an audio file setting enable indication 839BAA for setting the audio file to be played and a volume setting enable indication 839BAB for setting the volume of the audio file to be played. Detail area 837 includes a save enable indication 839BAC for saving the ambient audio action to the second behavior (or saving the change to the ambient audio action) and a cancel enable indication 839BAD for canceling the ambient audio action (or canceling the change to the ambient audio action).

[0486] Figure 8CMUser input 899CF directed toward save enable representation 839BAC is shown. In various embodiments, user input 899CF corresponds to a contact (e.g., a tap) detected at the location of save enable representation 839BAC.

[0487] Figure 8CN A CGR file composition user interface 801 is shown in response to detecting user input 899CF directed to save affordance 839BAC. Figure 8CN In the figure, the new action enable indication 835A and the audio action enable indication 835C are displayed in a first manner, indicating that the detail area 837 includes multiple behavior setting enable indications 836A to 836C for changing the settings of the behavior.

[0488] Figure 8CN User input 899CG is shown directed to new action-enabled representation 835A. In various embodiments, user input 899CG corresponds to a contact (e.g., a tap) detected at the location of new action-enabled representation 835A.

[0489] Figure 8CO A CGR file composition user interface 801 is shown in response to detecting user input 899CG directed to a new action-enabling representation 835A. Figure 8CO , the new action enable indication 835A is displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating a new action for the behavior.

[0490] For example, in Figure 8CO , detail area 837 includes a move action enable indication 839A, an audio action enable indication 839B, a scene change action enable indication 839C, and other action enable indications 839D for creating other types of actions.

[0491] Figure 8CO User input 899CH is shown pointing to move action-enabled representation 839A. In various embodiments, user input 899CH corresponds to a contact (e.g., a tap) detected at the location of move action-enabled representation 839A.

[0492] Figure 8CP A CGR file composition user interface 801 is shown in response to detecting user input 899CH of a pointing movement action affordance 839A. Figure 8CP , the new action enable indication 835A is displayed in a first manner, and the action area 835 includes a move action enable indication 835D displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating a move action.

[0493] For example, in Figure 8CPIn the figure, the detail area 837 includes a move-in action enable indication 839AA, a move-away action enable indication 839AB, a spin action enable indication 839AC, and other move action enable indications 839AD for creating other types of move actions.

[0494] Figure 8CP User input 899CI directed toward spin action enable indication 839AC is shown. In various embodiments, user input 899CI corresponds to a contact (e.g., a tap) detected at the location of spin action enable indication 839CI.

[0495] Figure 8CQ A CGR file is shown constituting user interface 801 in response to detecting user input 899CI of pointing movement affordance 839AC. Figure 8CQ , the detail area 837 includes a spin speed setting enable indication 839ACA for setting the speed of the CGR object spinning, a spin time setting enable indication 839ACB for setting the time of the CGR object spinning, and a spin axis setting enable indication 839ACC for setting the axis around which the CGR object spins. The detail area 837 includes a save enable indication 839ACD for saving the spinning action to a second behavior (or saving a change to the spinning action) and a cancel enable indication 839ACE for canceling the spinning action (or canceling a change to the spinning action).

[0496] exist Figure 8CQ , the representation of the sphere CGR object 831DC is highlighted, indicating that the sphere CGR object is the CGR object that spins during the action. By directing user input to the representation of other CGR objects, the CGR object that spins during the action can be changed.

[0497] Figure 8CQ User input 899CJ directed to save enable representation 839ACD is shown. In various embodiments, user input 899CJ corresponds to a contact (e.g., a tap) detected at the location of save enable representation 839ACD.

[0498] Figure 8CR A CGR file composition user interface 801 is shown in response to detecting user input 899CJ directed to save the affordance 839ACD. Figure 8CR In the figure, the new action enable indication 835A and the move action enable indication 835D are displayed in a first manner, indicating that the detail area 837 includes multiple behavior setting enable indications 836A to 836C for changing the settings of the behavior.

[0499] Figure 8CRUser input 899CK is shown directed to new action-enabled representation 835A. In various embodiments, user input 899CK corresponds to a contact (e.g., a tap) detected at the location of new action-enabled representation 835A.

[0500] Figure 8CS A CGR file composition user interface 801 is shown in response to detecting user input 899CK directed to a new action-enabling representation 835A. Figure 8CS , the new action enable indication 835A is displayed in a second manner to indicate that the detail area 837 includes an enable indication for creating a new action for the behavior.

[0501] For example, in Figure 8CS , detail area 837 includes a move action enable indication 839A, an audio action enable indication 839B, a scene change action enable indication 839C, and other action enable indications 839D for creating other types of actions.

[0502] Figure 8CS User input 899CL is shown directed toward scene change action affordance 839C. In various embodiments, user input 899CL corresponds to a contact (e.g., a tap) detected at the location of scene change action affordance 839C.

[0503] Figure 8CT A CGR file is shown constituting user interface 801 in response to detecting user input 899CL directed to scene change action affordance 839C. Figure 8CT , new action enable indication 835A is displayed in a first manner, and action area 835 includes scene change action enable indication 835E displayed in a second manner to indicate that detail area 837 includes an enable indication for creating a scene change action.

[0504] For example, in Figure 8CT , detail area 837 includes a setting enable indication 839CA for setting CGR as the target scene to be changed. Detail area 837 includes a save enable indication 839CB for saving the scene change action to the second behavior (or saving the change to the scene change action) and a cancel enable indication 839CC for canceling the scene change action (or canceling the change to the scene change action).

[0505] Figure 8CT User input 899CM pointing to save enable representation 839CB is shown. In various embodiments, user input 899CM corresponds to a contact (e.g., a tap) detected at the location of save enable representation 839CB.

[0506] Figure 8CUA CGR file composition user interface 801 is shown in response to detecting user input 899CM directed to save affordance 839CB. Figure 8CU In the figure, the new action enable indication 835A and the scene change action enable indication 835E are displayed in a first manner, indicating that the detail area 837 includes multiple behavior setting enable indications 836A to 836C for changing the settings of the behavior.

[0507] When a CGR application corresponding to an item is presented and a trigger for an action is detected, each action of the action is performed in the order displayed in the action area 835. Figure 8CU In the second behavior shown, when a proximity / collision trigger is detected, the device 100 plays an audio file, then spins the sphere CGR object once the audio file has completed, and then changes the CGR scene once the spin time defined by the spin time setting has elapsed. However, as described below, the actions of the behavior can be reordered and grouped to change the timing of the actions.

[0508] Figure 8CU User input 899CN pointing to move action enable representation 835D is shown. In various embodiments, user input 899CN corresponds to a detected moving contact (e.g., dragging or touch and drag) with a starting position at the position of move action enable representation 835D and an ending position to the left of audio action enable representation 835C.

[0509] Figure 8CV A CGR file composition user interface 801 is shown in response to detecting user input 899CN directed to move action affordance 835D. Within action region 835, the action affordances are reordered so that move action affordance 835D is to the left of audio action affordance 835C. Thus, for example, Figure 8CV The second behavior shown is that when a proximity / collision trigger is detected, the device 100 spins the spherical CGR object, then plays the audio file once the spin time defined by the spin time setting has elapsed, and then changes the CGR scene once the audio file has completed.

[0510] Figure 8CV User input 899CO is shown directed to audio action enable representation 835C. In various embodiments, user input 899CO corresponds to a detected moving contact (e.g., dragging or touch and drag) with a starting position at the location of audio action enable representation 835C and an ending position at the location of moving action enable representation 835D.

[0511] Figure 8CWA CGR file constructing user interface 801 is shown in response to detecting user input 899CO directed to audio action enable representation 835C.

[0512] In action area 835, audio action enable representation 835C and mobile action enable representation 835D are replaced with grouped action enable representation 835X representing audio action enable representation 835C and mobile action enable representation 835D. Figure 8CW The second behavior shown is that when a proximity / collision trigger is detected, the device 100 simultaneously spins the spherical CGR object and plays the audio file, and then changes the CGR scene once the spin time defined by the spin time setting has elapsed and the audio file has completed.

[0513] Figure 8CW User input 899CP is shown pointing to determine enable representation 831E. In various embodiments, user input 899CP corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 831E.

[0514] Figure 8CX A CGR file composition user interface 801 is shown in response to detecting user input 899CP pointing to an affordance 831E. Figure 8CX , the behavior editor user interface 831 is removed and the second behavior enable indication 803CB (having the color indicated by the second behavior) is displayed in the behavior area 803C.

[0515] exist Figure 8CX , the second behavior is selected, as indicated by the second behavior enable indication 803CB displayed in a selected manner (e.g., having a thicker border than in an unselected manner such as the first behavior enable indication 803CA). Because the second behavior is selected, the representation of the second CGR scene in the view area 803E indicates the CGR objects associated with the trigger of the first behavior using a first highlight (e.g., a glow) and indicates the CGR objects associated with the actions of the first behavior using a second highlight (e.g., a diagonal hash). Thus, the representation of the sphere CGR object 806D and the representation of the cup CGR object 806B are both displayed with a first highlight (because the sphere CGR object colliding with the center of the cup CGR object is the trigger of the second behavior). In addition, the representation of the sphere CGR object 806D is also displayed with a second highlight (because spinning the sphere CGR object is the action of the second behavior). The view area 803E also includes an off-screen action indicator 803EC, which indicates that the behavior includes one or more actions with non-displayed targets (e.g., off-screen CGR objects or non-displayed elements, such as audio file playing and scene changes).

[0516] Figure 8CX User input 899CQ directed to preview enable representation 803AD is shown. In various embodiments, user input 899CQ corresponds to a contact (e.g., a tap) detected at the location of preview enable representation 803AD.

[0517] Figure 8CY A CGR file composition user interface 801 is shown in response to detecting user input 899CQ pointing to a preview affordance 803AD. Figure 8CY In the example, the CGR file constitutes the user interface 801 including the preview user interface 802D. The preview user interface 802D includes a representation of the table 802DBA within the preview display area 802DB.

[0518] Figure 8CZ The user interface 801 is shown to be composed of a CGR file in response to the device 100 detecting that a portion of the scene camera image (e.g., a representation of the table 802DBA) includes a horizontal plane, which is an anchor for the second CGR scene. Figure 8CZ , the preview display area 802DB includes a second CGR scene (including a cup CGR object 840A, a plate CGR object 840B, and a sphere CGR object 840C) displayed as a representation anchored to the table 802DBA.

[0519] Figure 8DA The CGR file composition user interface 801 is shown in response to moving the device 100 within the physical environment closer to the table. Figure 8DA , the preview display area 802DB includes a representation of the second CGR scene (including a cup CGR object 840A, a plate CGR object 840B, and a sphere CGR object 840C) and a representation of the table 802DBA that is displayed larger.

[0520] Figure 8DB The CGR file composition user interface 801 is shown in response to moving the device 100 within the physical environment around the table to one side of the table. Figure 8L , the perspective of the device 100 has changed (so that the representation of the table 802DBA is seen from a different perspective), and the perspective of the virtual camera used to render the second CGR scene has similarly changed. Therefore, the cup CGR object 840A, the plate CGR object 840B, and the sphere CGR object 840C are displayed at different locations in the preview display area 802DB.

[0521] Figure 8DC The CGR file composition user interface 801 is shown in response to detecting a user's puff after a first amount of time. Because the puff is a trigger for the first behavior, the device 100 performs the action of the first behavior, thereby moving the sphere CGR object 840C forward. Figure 8DC , in the preview display area 802DB, the sphere CGR object 840C is moved and displayed to be closer to the cup CGR object 840A and the plate CGR object 840B.

[0522] Figure 8DD The CGR file composition user interface 801 is shown in response to detecting a user's puff after the second amount of time. Figure 8DC , the sphere CGR object 840C moves even further forward (and is displayed in the preview display area 802DB) to a point where the mesh of the sphere CGR object 840C intersects the center of the cup CGR object 840A.

[0523] Figure 8DE The CGR file composition user interface 801 is shown in response to detecting that the sphere CGR object 840C intersects the center of the cup CGR object 840A. Because the intersection of the sphere CGR object 840C and the center of the cup CGR object 840A is a trigger for the second behavior, the device 100 performs the actions of the second behavior in its defined order. Figure 8DE In FIG. 8 , while the audio file is playing, the sphere CGR object 840C spins within the cup CGR object 840A for a spin time.

[0524] Figure 8DF The CGR file composition user interface 801 is shown in response to the spin time elapsed and the audio file playing completed. Figure 8DF , because the second behavior includes a scene change action, the second CGR scene is replaced by a third CGR scene 840D displayed in the preview display area 802DB anchored to the representation of the table 802DBA.

[0525] In various embodiments, the CGR file composition user interface 801 includes an enabling representation for saving a project as a project file, which can be read by a CGR file composition application that is executed by the device 100 and presented via the CGR file composition user interface 801 as described above.

[0526] In various embodiments, the CGR file composition user interface 801 includes an enabling representation for compiling a project into a CGR file that can be read by a CGR rendering application that is executed by device 100 or another device (such as electronic device 520A).

[0527] In various embodiments, a CGR file comprises an XML (Extensible Markup Language) or JSON (JavaScript Object Notation) file that describes the CGR contents and references various data structures (e.g., display meshes, physics meshes, textures, images, audio, etc.) that are also included in the CGR file.

[0528] Fig. 9 900 is a flowchart representation of a method 900 for presenting a CGR scene using a backup anchor according to some embodiments. In various embodiments, the method 900 is performed by a computer having one or more processors, non-volatile memory, a scene camera, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 900 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0529] The method 900 begins at block 910, where a device receives user input via one or more input devices for presenting a computer generated reality (CGR) scene including one or more CGR objects, where the CGR scene is associated with a first anchor and a second anchor. Figure 8H , device 100 detects user input 899H directed to preview enable indication 803AD to present a first CGR scene including a diamond CGR object, wherein the first CGR scene is associated with a primary anchor of an anchor image (as indicated by primary anchor enable indication 803DBA) and a secondary anchor of a horizontal plane (as indicated by secondary anchor enable indication 803DBC).

[0530] In various embodiments, the first anchor is an anchor image and the second anchor is a horizontal plane or a vertical plane. In various embodiments, the CGR scene is associated with the anchor image by selecting an image file. In various embodiments, the CGR scene is associated with the anchor image by capturing the anchor image using a camera.

[0531] In various embodiments, the first anchor is a tabletop or floor and the second anchor is a horizontal plane.

[0532] Method 900 continues at block 920, where the device captures an image of the physical environment using a camera. Figure 8I , the device 100 displays a preview display area 802DB that includes a scene camera image captured by the device 100 of the physical environment.

[0533] Method 900 continues at block 930, where the device determines that the image of the physical environment lacks a portion corresponding to the first anchor. Figure 8L , the preview display area 802DB includes a scene camera image that lacks a portion of a primary anchor corresponding to the first CGR scene, for example, an image of block A.

[0534] Method 900 continues at block 940, where the device detects a portion of the image of the physical environment corresponding to the second anchor. Figure 8L , the preview display area 802DB includes a scene camera image that includes a portion of a secondary anchor corresponding to the first CGR scene, for example, a horizontal plane, such as a representation of a table 802DBA.

[0535] Method 900 continues at block 950, where the device displays on the display the CGR scene at a location of the display corresponding to the second anchor in response to determining that the image of the physical environment lacks a portion corresponding to the first anchor and detecting a portion of the image of the physical environment corresponding to the second anchor. Figure 8M , the preview display area 802DB includes a first CGR scene (including a diamond CGR object 802DBD) displayed at the location of the representation of the table 802DBA.

[0536] In various embodiments, displaying the CGR scene at a location of the display corresponding to the second anchor includes displaying an image of the physical environment and displaying the CGR scene at a portion of the image of the physical environment corresponding to the second anchor.

[0537] In various embodiments, displaying the CGR scene at a location of the display corresponding to the second anchor includes displaying the CGR scene at a location of the optically transparent display (eg, a transparent display) where the second anchor is visible to a user.

[0538] In various embodiments, displaying the CGR scene at a position of the display corresponding to the second anchor includes: displaying the CGR scene anchored to the second anchor. In various embodiments, in response to a change in the viewing angle of the camera, the display of the CGR scene changes accordingly on the display. For example, in Figure 8DA and Figure 8DB , the second CGR scene is displayed anchored to the top of the table, and in response to movement of the device 100, the display of the second CGR scene changes accordingly.

[0539] In various embodiments, a portion of the image of the physical environment corresponds to a physical object, and in response to movement of the physical object, the display of the CGR scene moves accordingly on the display. Figure 8J and Figure 8K, the first CGR scene is displayed as anchored to the representation of the Ace of Diamonds card 802DBB, and in response to the movement of the Ace of Diamonds card, the display of the first CGR scene (including the Diamond CGR object) moves accordingly on the display.

[0540] In various embodiments, displaying the CGR scene at a location of the display corresponding to the second anchor is performed in response to determining that a plurality of images of the physical environment captured within a threshold amount of time each lack a portion corresponding to the first anchor. Figure 8M In the embodiment of the present invention, a first CGR scene is displayed in response to detecting a representation of a horizontal plane in a portion of a scene camera image, and in various embodiments, in response to failing to detect a portion of the scene camera image that matches the anchor image for at least a threshold amount of time (e.g., 3 seconds).

[0541] In various embodiments, the CGR scene is associated with a third anchor, and determining that the image of the physical environment lacks a portion corresponding to the first anchor is performed in response to determining that the image of the physical environment lacks a portion corresponding to the third anchor. Fig.8F A three-level anchor affordance 803DBD for defining a three-level anchor is shown. In various embodiments where a three-level anchor is defined for a CGR scenario, the primary anchor corresponds to the third anchor of method 900, the secondary anchor corresponds to the first anchor of method 900, and the tertiary anchor corresponds to the second anchor of method 900. In various embodiments where a three-level anchor is not defined for a CGR scenario, the primary anchor corresponds to the first anchor of method 900, and the secondary anchor corresponds to the second anchor of method 900.

[0542] Fig.10 1 is a flowchart representation of a method 1000 for configuring a CGR object according to some embodiments. In various embodiments, the method 1000 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1000 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1000 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0543] Method 1000 begins at block 1010, where a device displays, on a display, a representation of a computer generated reality (CGR) object associated with a first parameter and a second parameter, where the first parameter has a first parameter value of a plurality of first parameter values ​​and the second parameter has a first parameter value of a plurality of second parameter values. Figure 8X, the device 100 displays a representation of a currently selected CGR object 808DA, such as a cup CGR object, wherein the cup CGR object is associated with a first parameter (e.g., style) and a second parameter (e.g., pattern). The first parameter has a first parameter value of a plurality of first parameter values ​​(e.g., a “mug” value of those listed in the style column 808E) and a first parameter value of a plurality of second parameter values ​​(e.g., a “white” value of those listed in the pattern column 808F).

[0544] Method 1000 continues at block 1020, where the device displays on the display a first user interface element for selecting a second parameter value from the plurality of first parameter values. Figure 8X , device 100 displays a style bar 808E, which includes a plurality of style indicators 808EA to 808ED for changing a currently selected style of style bar 808E.

[0545] Method 1000 continues at block 1030, where the device displays a second user interface element on the display for selecting a second parameter value in the plurality of second parameter values, wherein based on a first parameter value in the plurality of first parameter values ​​and one or more selection rules, a subset of the plurality of second parameter values ​​is presented for selection via the second user interface element. Figure 8X In FIG. 8 , device 100 displays pattern bar 808F, which includes a plurality of pattern affordances 808FA to 808FD for changing a currently selected pattern of pattern bar 808F. Figure 8X 808F. In the second display style, the mug style is the currently selected style of style column 808E, and the white pattern is the currently selected pattern of pattern column 808F. Based on the currently selected style being the mug style, a subset of pattern affordances 808FA through 808FD are presented for selection. Specifically, star affordance 808FC and ivy affordance 808FD are presented for selection, as indicated by the second display style, while glass affordance 808FB is not presented for selection because it is grayed out.

[0546] In various embodiments, the second user interface element includes a set of selectable affordances corresponding to a subset of the plurality of second parameters and a set of non-selectable affordances corresponding to other parameters in the plurality of second parameters. Figure 8X In the figure, pattern bar 808F includes a set of selectable enable indications (eg, star enable indication 808FC and ivy enable indication 808FD) and a set of non-selectable enable indications (eg, glass enable indication 808FB).

[0547] In various embodiments, the set of selectable indicators can be displayed in a first manner, and the set of non-selectable indicators can be displayed in a second manner different from the first manner. Figure 8X In the figure, the star indicator 808FC and the ivy indicator 808FD are displayed in white, while the glass indicator 808FB is displayed in gray.

[0548] In various embodiments, method 1000 further includes receiving user input, via one or more input devices, selecting a second parameter value from among the plurality of first parameter values, and updating display of a second user interface element, wherein based on the second parameter value from among the plurality of first parameter values ​​and the one or more selection rules, a second subset of the plurality of second parameters is presented for selection via the second user interface element. Figure 8X , device 100 detects user input 899T pointing to espresso indication 808EC. Figure 8Y , device 100 displays pattern bar 808F, in which glass indicator 808FB is selectable, and star indicator 808FC and ivy indicator 808FD are not selectable.

[0549] In various embodiments, method 1000 also includes updating the display of the representation of the CGR object. Figure 8Y , the device 100 displays a representation of the currently selected CGR object 808A having a different shape (eg, as an espresso cup instead of a mug), such as a cup CGR object.

[0550] In various embodiments, method 1000 includes receiving user input to update a first parameter of a CGR object to a second parameter value of a plurality of first parameter values. Figure 8AA , device 100 detects user input 899W pointing to determine affordance 808G, thereby updating the style parameter of the cup CGR object from a value of "mug" to a value of "tea cup", such as Figure 8AB shown.

[0551] In various embodiments, method 1000 further includes displaying a plurality of view-enabled representations on a display, the plurality of view-enabled representations including an object view-enabled representation and a CGR view-enabled representation, the object view-enabled representation being used to enter an object view mode in which user input directed to a CGR object changes the viewing angle of the display of the representation of the CGR object, and the CGR view-enabled representation being used to enter a CGR view mode in which movement of the display changes the viewing angle of the display of the representation of the CGR object. For example, in Figure 8X , device 100 displays an object view enable representation 808A and a CGR view enable representation 808B.

[0552] In various embodiments, the CGR object is also associated with a third parameter, wherein the third parameter has a first parameter value among multiple third parameter values, and also includes displaying a third user interface element on the display for selecting a second parameter value among the multiple third parameter values, wherein based on the first parameter value among the multiple first parameter values, the first parameter value among the multiple second parameter values, and one or more selection rules, a subset of the multiple third parameter values ​​is presented for selection via the third user interface element.

[0553] Fig.11 1 is a flowchart representation of a method 1100 for resolving overlap of two CGR objects according to some embodiments. In various embodiments, the method 1100 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1100 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0554] The method 1100 begins at block 1110, where a device displays a representation of a CGR scene on a display, including displaying respective representations of a plurality of CGR objects of the CGR scene. Figure 8AF , the device 100 displays a representation of the second CGR scene, including displaying a representation of a plate CGR object 806B and a representation of a cup CGR object 806C.

[0555] The method 1100 continues at block 1120, where the device determines that two of the plurality of CGR objects overlap in the CGR scene. In various embodiments, determining that two of the plurality of CGR objects overlap in the CGR scene is performed periodically. In various embodiments, determining that two of the plurality of CGR objects overlap in the CGR scene is performed in response to a user input spatially manipulating one of the plurality of CGR objects. For example, Figure 8AF In, with Figure 8AD In contrast, in response to user input 899Z pointing to the representation of cup CGR object 806C, the position of the cup CGR object is changed (e.g., the cup CGR object is moved). In various embodiments, determining that two CGR objects of the plurality of CGR objects overlap in the CGR scene is performed in response to user input requesting an overlap check.

[0556] The method 1100 continues at block 1130 where the device displays an indicator on the display in association with at least one respective representation of two CGR objects in the plurality of CGR objects that two CGR objects in the plurality of CGR objects overlap in the CGR scene. Figure 8AF , the device 100 displays a first overlapping indicator 809A in the form of a notification bubble coupled to the representation of the cup CGR object 806C and a second overlapping indicator 809B in the form of a shadow, glow, hatching, or other highlight overlaying over the representation of the cup CGR object 806C and the representation of the saucer CGR object 806B.

[0557] In various embodiments, the overlap indication is displayed in association with respective representations of two CGR objects in the plurality of CGR objects. Figure 8AF , a second overlap indicator 809B is displayed over both the representation of the cup CGR object 806C and the representation of the saucer CGR object 806B.

[0558] In various embodiments, the overlap indication is displayed in association with a location where two CGR objects in the plurality of CGR objects overlap in the CGR scene. Figure 8AF , a second overlap indicator 809B is displayed above a location where the representation of the cup CGR object 806C and the representation of the saucer CGR object 806B overlap.

[0559] In various embodiments, the overlapping indication includes a highlight displayed above at least one corresponding representation of two CGR objects in the plurality of CGR objects. Figure 8AF , a second overlap indicator 809B is displayed over both the representation of the cup CGR object 806C and the representation of the saucer CGR object 806B.

[0560] In various embodiments, the overlapping indication includes a notification user element displayed adjacent to at least one corresponding representation of two CGR objects in the plurality of CGR objects. Figure 8AF , the device 100 displays a first overlapping indicator 809A in the form of a notification bubble that is adjacent to and coupled to the representation of the cup CGR object 806C.

[0561] In various embodiments, method 1100 also includes receiving user input pointing to the overlap indication via one or more input devices. Figure 8AF In the embodiment of the present invention, device 100 detects user input 899AA pointing to first overlap indicator 809A. Method 1100 also includes displaying an ignore affordance in response to receiving user input pointing to the overlap indication. For example, in Figure 8AG, device 100 displays ignore affordance 810D. Method 1100 also includes receiving user input directed to the ignore affordance via one or more input devices, and ceasing display of the overlay indication in response to receiving the user input directed to the ignore affordance.

[0562] In various embodiments, method 1100 also includes receiving user input pointing to the overlap indication via one or more input devices. Figure 8AF In the embodiment of the present invention, device 100 detects user input 899AA pointing to first overlap indicator 809A. Method 1100 also includes displaying a repair affordance in response to receiving user input pointing to the overlap indicator. For example, in Figure 8AB In the embodiment of the present invention, device 100 displays repair enable representation 810E. Method 1100 also includes receiving user input pointing to the repair enable representation via one or more input devices. For example, in Figure 8AG In the method 1100, device 100 detects user input 899AB pointing to the repair affordance 810E. Method 1100 also includes, in response to receiving the user input pointing to the repair affordance, changing the position of at least one of two CGR objects in the plurality of CGR objects so that the two CGR objects in the plurality of CGR objects do not overlap in the CGR scene. For example, in Figure 8AH , the position of the cup CGR object changes (as indicated by the different position of the representation of the cup CGR object 806C) such that the cup CGR object and the saucer CGR object no longer overlap.

[0563] Fig.12 1 is a flowchart representation of a method 1200 for spatially manipulating a CGR object in different spatial manipulation modes according to some embodiments. In various embodiments, the method 1200 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1200 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1200 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0564] The method 1200 begins at block 1210, where a device displays a representation of a CGR scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. Figure 8U , the device 100 displays a representation of the second CGR scene, including displaying a representation of a plate CGR object 806B and a representation of a cup CGR object 806C.

[0565] The method 1200 continues at block 1220, where the device receives user input via one or more input devices directed to a particular representation of one or more CGR objects. Figure 8U , device 100 detects user input 899Q directed to the representation of cup CGR object 806C.

[0566] The method 1200 continues at block 1230, where the device provides a first manipulation mode associated with a first set of conformal space manipulations of the corresponding CGR object in response to receiving a user input directed to a particular representation of one or more CGR objects. In various embodiments, the method 1200 includes displaying a first type of object selection indicator associated with the corresponding CGR object in response to receiving a user input directed to a particular representation of one or more CGR objects. For example, in Figure 8V , device 100 displays a first type of object selection indicator 807 surrounding a representation of cup CGR object 806C in response to user input 899Q.

[0567] For example, when a CGR object is selected (and a first type of object selection indicator 807 is displayed), different types of user input directed to a representation of the CGR object result in different changes to the spatial properties of the CGR object. Figure 8AB and Figure 8AC , a first type of user input 899X (e.g., pinching) directed to the representation of the cup CGR object 806C changes the size of the cup CGR object. As another example, in Figure 8AC and Figure 8AD , a second type of user input 899Y (e.g., rotation) directed to the representation of the cup CGR object 806C changes the orientation about the z-axis of the cup CGR object. As another example, in Figure 8AD and Figure 8AF , a third type of user input 899Z (eg, dragging) directed to the representation of the cup CGR object 806C changes the position of the cup CGR object in the xy plane.

[0568] The method 1200 continues at block 1240, where the device receives user input via one or more input devices to switch from the first manipulation mode to a second manipulation mode associated with a second set of conformal space manipulations of the corresponding CGR object. Figure 8AH , device 100 detects user input 899AB pointing to the representation of cup CGR object 806C, thereby changing the first type of object selection indicator 807 to a second type of object selection indicator 817 (eg, Figure 8AI shown).

[0569] In various embodiments, the user input pointing to the specific representation of the one or more CGR objects and the user input switching from the first manipulation mode to the second manipulation mode are the same type of user input. In various embodiments, the same type of user input is a tap at the location of the specific representation of the one or more CGR objects. For example, in Figure 8U and Figure 8AH , user inputs 899Q and 899AB are both taps at the location of the representation of the cup CGR object 806C.

[0570] Method 1200 continues at block 1250, where the device provides the second manipulation mode in response to receiving user input to switch from the first manipulation mode to the second manipulation mode. In various embodiments, method 1200 includes displaying a second type of object selection indicator associated with the corresponding CGR object in response to receiving user input to switch from the first manipulation mode to the second manipulation mode. For example, when the CGR object is selected (and the second type of object selection indicator 807 is displayed), user input directed to different portions of the second type of object selector results in different changes to the spatial properties of the CGR object. For example, in Figure 8AI and Figure 8AJ , user input 899AD pointing to the third rotation affordance 817F causes the cup CGR object to rotate about a third axis. As another example, in Figure 8AJ and Figure 8AK , user input 899AE pointing to the first translation indicator 817A causes the cup CGR object to move along the z-axis of the cup CGR object.

[0571] In various embodiments, wherein the second set of conformal spatial manipulations of the corresponding CGR object includes at least one conformal spatial manipulation of the first set of conformal spatial manipulations of the corresponding CGR object. In various embodiments, the second set of conformal spatial manipulations of the corresponding CGR object includes all conformal spatial manipulations of the first set of conformal spatial manipulations of the corresponding CGR object. In various embodiments, the first set of conformal spatial manipulations of the corresponding CGR object includes a translation of the corresponding CGR object in a plane, but does not include a translation of the corresponding CGR object perpendicular to the plane, wherein the second set of conformal spatial manipulations of the corresponding CGR object includes a translation of the corresponding CGR object perpendicular to the plane. In various embodiments, the first set of conformal spatial manipulations of the corresponding CGR object includes a rotation of the corresponding CGR object around an axis, but does not include a translation of the corresponding CGR object around other axes, wherein the second set of conformal spatial manipulations of the corresponding CGR object includes a translation of the corresponding CGR object around other axes. In various embodiments, the first set of conformal spatial manipulations of the corresponding CGR object includes changing the size of the corresponding CGR object.

[0572] Fig.131 is a flowchart representation of a method 1300 for spatially manipulating a CGR object using an intuitive spatial manipulation point according to some embodiments. In various embodiments, the method 1300 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1300 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0573] The method 1300 begins in block 1310, where a device displays a representation of a computer generated reality (CGR) scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. Figure 8AC , the device 100 displays a representation of the second CGR scene, including displaying a representation of the dish CGR object 806B and a representation of the cup CGR object 806B.

[0574] The method 1300 continues at block 1310, where the device receives user input via one or more input devices to spatially manipulate a particular CGR object among the one or more CGR objects, where the particular CGR object is associated with an intuitive spatial manipulation point. Figure 8AC , the device 100 receives a user input 899Y pointing to the representation of the cup CGR object 806C to rotate the cup CGR object. As another example, in Figure 8AD , the device 100 receives a user input 899Z pointing to the representation of the cup CGR object 806C to move the cup CGR object. Figure 8AE1 and Figure 8AE2 As shown, in various embodiments, the cup CGR object is associated with an apparent spatial manipulation point 885.

[0575] In various embodiments, the intuitive spatial manipulation point is neither the edge of the particular CGR object, the midpoint of the bounding box surrounding the particular CGR object, nor the unweighted center of mass of the particular CGR object. Figure 8AE1 and Figure 8AE2 , the intuitive spatial manipulation point 885 is different from the first spatial manipulation point 883 , the midpoint of the bounding box 882 surrounding the cup CGR object 880 , and is different from the second spatial manipulation point 884 , the unweighted center of mass of the cup CGR object 880 .

[0576] In various embodiments, the apparent spatial manipulation point is stored with a CGR object file that defines a particular CGR object.In various embodiments, method 1300 includes receiving user input defining the apparent spatial manipulation point via one or more input devices.

[0577] In various embodiments, method 1300 includes determining an intuitive spatial manipulation point prior to receiving user input to spatially manipulate a particular CGR object. For example, in various embodiments, determining the intuitive spatial manipulation point includes determining a plurality of segments of the particular CGR object, and determining the intuitive spatial manipulation point based on weights of the plurality of segments of the particular CGR object. For example, Figure 8AE1 and Figure 8AE2 , the intuitive spatial manipulation point 885 is determined to be the center of mass of the cup CGR object 880 ignoring the base 881B and handle 881C.

[0578] The method 1300 continues at block 1330, where the device changes a spatial property of the specific CGR object based on the user input and the intuitive spatial manipulation point. In various embodiments, changing the spatial property of the specific CGR object based on the user input and the intuitive spatial manipulation point includes rotating the specific CGR object around an axis passing through the intuitive spatial manipulation point. For example, Figure 8AD , the representation of the cup CGR object 806C rotates about a z-axis that passes through the intuitive spatial manipulation point. In various embodiments, changing the spatial properties of a particular CGR object based on the user input and the intuitive spatial manipulation point includes: moving the particular CGR object, wherein the intuitive spatial manipulation is aligned with the target point. In various embodiments, changing the spatial properties of a particular CGR object based on the user input and the intuitive spatial manipulation point includes: moving the particular CGR object, wherein the intuitive spatial manipulation point snaps to the target point. For example, in Figure 8AF , the representation of the cup CGR object 806C is moved, snapped to, and aligned with the center of the representation of the dish CGR object 806B (and the grid points aligned with the center of the representation of the dish CGR object 806B).

[0579] Fig.14 1400 is a flowchart representation of a method 1400 for configuring a CGR object according to some embodiments. In various embodiments, the method 1400 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1400 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0580] The method 1400 begins at block 1410, where a device displays a representation of a CGR scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. Figure 8AS , the device 100 displays a representation of the second CGR scene, including displaying a representation of a plate CGR object 806B and a representation of a cup CGR object 806C.

[0581] The method 1400 continues at block 1420, where the device receives, via one or more input devices, a user input directed to a representation of a particular CGR object among the one or more CGR objects, the particular CGR object being associated with the parameter. Figure 8AS , device 100 detects user input 899AL directed to a representation of a cup CGR object 806C, which is associated with a style setting and a pattern setting.

[0582] The method 1400 continues at block 1430, where the device displays, on the display, a plurality of representations of the particular CGR object from the first perspective, wherein each of the plurality of representations of the particular CGR object is associated with a different respective value of the parameter. Figure 8AU , the device 100 displays a real-time view user interface 811, which includes a plurality of view windows 811DA to 811DC, which respectively include a plurality of representations of cup CGR objects 811EA to 811EC, each cup CGR object having a different pattern.

[0583] In various embodiments, method 1400 also includes displaying textual indications and parameters of specific CGR objects. Figure 8AU , the real-time view user interface 811 includes a text description 811C of the cup CGR object and CGR object settings (eg, pattern) that are manipulated via the real-time view user interface 811 .

[0584] In various embodiments, the particular CGR object is associated with a second parameter, and method 1400 further includes displaying, when displaying multiple representations of the particular CGR object from a first perspective, a second plurality of representations of the particular CGR object from the first perspective, wherein each of the second plurality of representations of the particular CGR object is associated with a different corresponding value of the second parameter.

[0585] Method 1400 continues at block 1440, where the device receives, via one or more input devices, a user input to change the first perspective to a second perspective. In various embodiments, the user input to change the first perspective to the second perspective includes a user input pointing to any one of a plurality of representations of a particular CGR object. For example, in Figure 8AU, device 100 detects user input 899AN directed to first view window 811DA, thereby changing the viewing angle of view windows 811DA to 811DC. In various embodiments, the user input for changing the first viewing angle to the second viewing angle includes moving the display. For example, in Figure 8AU , when the CGR view mode is active (e.g., by selecting CGR view enable indication 811B), moving device 100 changes the viewing angle of view windows 811DA through 811DC.

[0586] In various embodiments, method 1400 includes displaying multiple view-enabled representations on a display, the multiple view-enabled representations including an object view-enabled representation and a CGR view-enabled representation, the object view-enabled representation being used to enter an object view mode, in which a user input for changing a first perspective to a second perspective includes a user input pointing to any one of multiple representations of a specific CGR object, the CGR view-enabled representation being used to enter a CGR view mode, in which a user input for changing a first perspective to a second perspective includes moving the display. For example, in Figure 8AU , device 100 displays an object view enable representation 811A and a CGR view enable representation 811B.

[0587] Method 1400 continues at block 1450, where the device displays multiple representations of the specific CGR object from the second perspective on the display in response to receiving user input to change the first perspective to the second perspective. Figure 8AV In the device, Figure 8AU Multiple representations of the cup CGR objects 811EA to 811EC are shown at different viewing angles.

[0588] In various embodiments, a difference between a first representation of the plurality of representations of a particular CGR object and a second representation of the plurality of representations of the particular CGR object is not visible from the display of the first viewing angle, and the difference is displayed from the second viewing angle. Figure 8AU In the example, the difference between the star and ivy patterns of the CGR cup object is not visible in the first perspective, but is visible when viewed from the second perspective. Figure 8AV The difference is shown in .

[0589] In various embodiments, method 1400 includes receiving user input selecting a particular representation of a plurality of representations of a particular CGR object, and setting a parameter of the particular CGR object to a corresponding value of the parameter of the particular representation. Figure 8AW , the device 100 detects a user input 899AP directed to the third view window 811DC, and in response to detecting the user input 899AP directed to the third view window 811DC Figure 8AXThe OK indication in 811F indicates user input 899AQ, and sets the pattern setting of the cup CGR object to the "ivy" value, such as Figure 8AY The pattern in can represent what is indicated by 813DBC.

[0590] Fig.15 1 is a flowchart representation of a method 1500 for presenting a CGR scene according to some embodiments. In various embodiments, the method 1500 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1500 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0591] Method 1500 begins at block 1510, where a device displays a representation of a CGR scene on a display, including displaying a representation of a first CGR object of the CGR scene, wherein displaying the representation of the first CGR object is based on a display grid associated with the first CGR object. Figure 8DB , the device 100 displays a representation of a second CGR scene, including a representation of a sphere CGR object 840C, a representation of a dish CGR object 840B, and a representation of a cup CGR object 840A. Displaying the representation of the sphere CGR object 840C is based on a display grid defined by a golf ball pattern.

[0592] In various embodiments, displaying a representation of the first CGR object is also based on a texture associated with the first CGR object. Figure 8DB , the display of the representation of the sphere CGR object 840C is based on a texture defined by a white pattern.

[0593] The method 1500 continues at block 1520, where the device determines an interaction of the first CGR object with a second CGR object of the CGR scene based on a physical grid associated with the first CGR object, where the physical grid associated with the first CGR object is different from a display grid associated with the first CGR object. In various embodiments, the second CGR object corresponds to a physical object. For example, in Figure 8DC In the embodiment, the device 100 determines the interaction of the sphere CGR object with the CGR object corresponding to the table. In various embodiments, the second CGR object is a virtual object. For example, in Figure 8DDIn the embodiment of the present invention, the device 100 determines the interaction of the ball CGR object with the dish CGR object and / or the cup CGR object. In both cases, the interaction is determined based on a physical grid (e.g., a ball grid) that is different from the display grid defined by the golf ball pattern.

[0594] In various embodiments, the physical mesh includes fewer polygons than the display mesh. Figure 8DB to Figure 8DD In , the sphere mesh includes fewer polygons than the golf ball mesh. In various embodiments, the physical mesh is smoother than the display mesh. For example, in Figure 8DB to Figure 8DD In the example above, the sphere mesh is smoother (has no dimples) than the golf ball mesh.

[0595] In various embodiments, method 1500 includes: before determining the interaction, receiving user input to change the physical grid from a first grid to a second grid, the first grid being the same as a display grid associated with the first CGR object, the second grid being different from the display grid associated with the first CGR object; and in response to receiving the user input to change the physical grid, associating the second grid with the first CGR object as the physical grid. Figure 8BE In FIG. 8 , the physical grid affordance 813DCA indicates that the physical grid of the sphere CGR object is the same as the display grid (as indicated by the style affordance 813DBB). Figure 8BE The user enters 899AX, Figure 8BF The user enters 899AY and Figure 8BG The user inputs 899AZ to change the physical mesh of the sphere CGR object to a sphere mesh, such as Figure 8BH The physical grid in can represent what is indicated by 813DCA.

[0596] In various embodiments, the first CGR object is described by a CGR object file including a plurality of meshes, the plurality of meshes including a first mesh and a second mesh. Figure 8BF , device 100 displays a setting value selection user interface 812, which includes a plurality of setting value selection affordances 812BA to 812BC corresponding to a grid of the sphere CGR object file.

[0597] In various embodiments, method 1500 includes receiving user input for importing a mesh into a CGR object file, and in response to receiving the user input for importing a mesh into the CGR object file, importing a third mesh into the CGR object file. Figure 8BH , device 100 displays a loading indicator 812BC for loading a custom mesh into a sphere CGR object file.

[0598] Fig.161 is a flowchart representation of a method 1600 for associating a behavior with a CGR scene according to some embodiments. In various embodiments, the method 1600 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 In some embodiments, the method 1600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1600 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0599] The method 1600 begins at block 1610, where a device displays a representation of a computer generated reality (CGR) scene on a display, including displaying corresponding representations of one or more CGR objects of the CGR scene. Figure 8BI , the device 100 displays a representation of a second CGR scene, including a representation of a saucer CGR object 806B, a representation of a cup CGR object 806C, and a representation of a sphere CGR object 806D.

[0600] Method 1600 continues at block 1620, where the device receives user input via one or more input devices to add a behavior to the CGR scene. Figure 8BI , device 100 detects the pointing behavior adding an indication of user input 899BB 803AC.

[0601] Method 1600 continues at block 1630, where the device receives user input via one or more input devices that defines a first trigger for an action. Figure 8BK The new trigger in the 833A can indicate the user input 899BD, pointing to Figure 8BL The tap trigger indicator in the 838A indicates user input 899BE, pointing to Figure 8BM The user input 899BF and the pointing Figure 8BN The completion indicator in 838AA represents the tap trigger that defines the first action by user input 899BG.

[0602] In various embodiments, the first trigger is a user action. For example, in various embodiments, the user action is at least one of a tap by the user on a specific CGR object among the one or more CGR objects, a gaze by the user on the specific CGR object, or a facial action of the user.

[0603] In various embodiments, the first trigger is a CGR scene condition. For example, in various embodiments, the CGR scene condition is at least one of proximity of a particular CGR object of the one or more CGR objects to a location in the CGR scene (e.g., coordinates or location of another CGR object) or CGR scene time lapse (e.g., after the CGR scene has been presented for a threshold amount of time).

[0604] Method 1600 continues at block 1630, where the device receives user input of a first action defining a behavior via one or more input devices, wherein the first action is performed in response to detecting a first trigger while the CGR scene is being presented. Figure 8BR The new action indicator in 835A can indicate user input 899BK, point to Figure 8BS The moving affordances in 839A represent user input 899BL, pointing to Figure 8BT The removal action in the 839AB can indicate the user input 899BM, pointing to Figure 8BU The forward movement indicator in the 839ABA user input 899BN and the pointing Figure 8BV The save indication in 839ABAC represents the user input 899BO that limits the moving action of the first behavior.

[0605] In various embodiments, the first action is at least one of spatial manipulation of a particular CGR object among the one or more CGR objects, playing an audio file, or presenting a new CGR scene.

[0606] In various embodiments, method 1600 further includes receiving user input, via one or more input devices, of a second trigger defining an action, wherein when the CGR scene is presented, the first action is performed in response to detecting the second trigger. Figure 8BO The new trigger in the 833A can indicate the user input 899BH, pointing to Figure 8BP The face tracking trigger in the can indicate user input 899BI of 838C and pointing Figure 8BQ The blow trigger indicator in 838CC indicates that the user input 899BJ defines the facial tracking trigger of the first behavior.

[0607] In various embodiments, method 1600 further includes receiving user input of a second ac...

Claims

1. A method, comprising: receiving user input generating a computer-generated scene via one or more input devices; receiving, via the one or more input devices, user input associating an anchor with the computer-generated scene, wherein the anchor is associated with a visual characteristic; receiving, via the one or more input devices, user input associating one or more objects with the computer-generated scene; displaying on a display a graphical user interface for associating a behavior with the computer-generated scenario, wherein a first state of the graphical user interface displays a set of triggers and a second state of the graphical user interface displays a set of actions; receiving user input via the one or more input devices, the user input defining corresponding relationships between one or more triggers and one or more actions by selecting one or more triggers from the set of triggers displayed in the first state of the graphical user interface and selecting one or more actions from the set of actions displayed in the second state of the graphical user interface; as well as When the computer-generated scene including the one or more objects is displayed on the display in association with the visual characteristic associated with the anchor: receiving, via the one or more input devices, user input that satisfies the one or more triggers; and In response to receiving the user input satisfying the one or more triggers, The one or more actions are performed, including modifying the computer-generated scene on the display. The method of claim 1 , wherein the anchor is an anchor image.

3. The method of claim 1 , wherein the anchors include a first anchor and a second anchor, and wherein displaying the one or more objects includes: In response to determining that the image of the physical environment lacks a portion corresponding to the first anchor, the one or more objects are displayed in association with the second anchor.

4. The method of claim 1 , wherein receiving the user input via the one or more input devices to associate the one or more objects with the computer-generated scene comprises: displaying, on the display, a representation of an object associated with a first parameter and a second parameter, wherein the first parameter has a first parameter value of a plurality of first parameter values ​​and the second parameter has a first parameter value of a plurality of second parameter values; displaying on the display a first user interface element for selecting a second parameter value of the plurality of first parameter values; as well as A second user interface element is displayed on the display for selecting a second parameter value from among the plurality of second parameter values, wherein a subset of the plurality of second parameter values ​​is presented for selection via the second user interface element based on the first parameter value from among the plurality of first parameter values ​​and one or more selection rules.

5. A method according to claim 4, wherein the second user interface element includes a set of selectable enable representations corresponding to the subset of the multiple second parameters and a set of non-selectable enable representations corresponding to other parameters in the multiple second parameters.

6. The method of claim 1 , wherein receiving the user input via the one or more input devices to associate the one or more objects with the computer-generated scene comprises: displaying on the display a particular representation of an object associated with the computer-generated scene; receiving, via the one or more input devices, user input directed to the particular representation of the object; in response to receiving the user input directed to the particular representation of the object, providing a first manipulation mode associated with a first set of conformal spatial manipulations of the corresponding object; receiving, via the one or more input devices, a user input switching from the first manipulation mode to a second manipulation mode associated with a second set of conformal spatial manipulations of the corresponding object; as well as In response to receiving the user input to switch from the first manipulation mode to the second manipulation mode, the second manipulation mode is provided.

7. A method according to claim 6, wherein the first set of conformal spatial manipulations of the corresponding object includes a translation of the corresponding object in a plane, but does not include a translation of the corresponding object perpendicular to the plane, and wherein the second set of conformal spatial manipulations of the corresponding object includes a translation of the corresponding object perpendicular to the plane.

8. A method according to claim 6, wherein the first set of conformal spatial manipulations of the corresponding object includes rotation of the corresponding object around an axis, but does not include translation of the corresponding object around other axes, and wherein the second set of conformal spatial manipulations of the corresponding object includes translation of the corresponding object around other axes.

9. The method of claim 1 , wherein receiving the user input via the one or more input devices to associate the one or more objects with the computer-generated scene comprises: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input to spatially manipulate the particular object, wherein the particular object is associated with a spatial manipulation point; as well as A spatial property of the specific object is changed based on the user input and the spatial manipulation point. 10 . The method of claim 9 , wherein the spatial manipulation point is neither an edge of the specific object, a midpoint of a bounding box surrounding the specific object, nor an unweighted centroid of the specific object.

11. The method of claim 1 , wherein receiving the user input via the one or more input devices to associate the one or more objects with the computer-generated scene comprises: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input directed to the representation of the particular object, the particular object being associated with a parameter; displaying, on the display, a plurality of representations of the particular object from a first perspective, wherein each representation of the plurality of representations of the particular object is associated with a different respective value of the parameter; receiving, via the one or more input devices, a user input to change the first perspective to a second perspective; as well as In response to receiving the user input to change the first perspective to the second perspective, the plurality of representations of the particular object are displayed on the display from the second perspective.

12. The method of claim 11, wherein receiving the user input via the one or more input devices to associate the one or more objects with the computer-generated scene further comprises: receiving user input selecting a particular representation of the plurality of representations of the particular object; as well as The parameter of the particular object is set to the corresponding value of the particular representation parameter.

13. The method of claim 1 , wherein a first object of the one or more objects is associated with a display grid and a physical grid different from the display grid, and wherein displaying the first object is based on the display grid, and wherein interaction of the first object with a second object of the one or more objects is determined based on the physical grid. The method of claim 13 , wherein the physical mesh comprises fewer polygons than the display mesh.

15. A method according to claim 1, wherein the behavior includes a trigger associated with a first object among the one or more objects, wherein the behavior includes an action associated with a second object among the one or more objects, and the method further includes displaying a representation of the first object on the display with a first highlight and displaying the representation of the second object on the display with a second highlight different from the first highlight.

16. A method according to claim 1, wherein the user input that defines the corresponding relationship defines a first relationship between a first trigger and a first action by selecting a first trigger from the trigger set displayed in the first state of the graphical user interface and a first action from the action set displayed in the second state of the graphical user interface, and defines a second relationship between a second trigger and a second action by selecting a second trigger from the trigger set displayed in the first state of the graphical user interface and a second action from the action set displayed in the second state of the graphical user interface, wherein the first trigger is different from the second trigger.

17. The method of claim 1, wherein displaying the graphical user interface comprises: displaying the first state of the graphical user interface including the set of triggers; detecting, via the one or more input devices, a selection of a trigger from the set of triggers while displaying the first state of the graphical user interface; upon detecting selection of the trigger in the set of triggers, displaying the second state of the graphical user interface including the set of actions; as well as While the second state of the graphical user interface is displayed, a selection of an action from the set of actions is detected via the one or more input devices.

18. The method according to claim 17, further comprising: After detecting selection of the action in the set of actions, a third state of the graphical user interface is displayed on the display including an affordance that, when selected, further defines the selected action in the set of actions.

19. The method of claim 1, further comprising: prior to displaying the first state of the graphical user interface, detecting, via the one or more input devices, a user input corresponding to a request to display the first state of the graphical user interface; as well as Prior to displaying the second state of the graphical user interface, detecting, via the one or more input devices, a user input corresponding to a request to display the second state of the graphical user interface.

20. An electronic device, comprising: one or more input devices; non-transitory memory; monitor; and One or more processors configured to: receiving, via the one or more input devices, user input for generating a computer-generated scene; receiving, via the one or more input devices, user input associating an anchor with the computer-generated scene, wherein the anchor is associated with a visual characteristic; receiving, via the one or more input devices, user input associating one or more objects with the computer-generated scene; displaying on a display a graphical user interface for associating a behavior with the computer-generated scenario, wherein a first state of the graphical user interface displays a set of triggers and a second state of the graphical user interface displays a set of actions; receiving user input via the one or more input devices, the user input defining corresponding relationships between one or more triggers and one or more actions by selecting one or more triggers from the set of triggers displayed in the first state of the graphical user interface and selecting one or more actions from the set of actions displayed in the second state of the graphical user interface; as well as When the computer-generated scene including the one or more objects is displayed on the display in association with the visual characteristic associated with the anchor: receiving, via the one or more input devices, user input that satisfies the one or more triggers; as well as In response to receiving the user input satisfying the one or more triggers, performing the one or more actions includes modifying the computer-generated scene on the display.

21. The electronic device of claim 20, wherein the one or more processors are configured to receive the user input associating the one or more objects with the computer-generated scene via the one or more input devices by: displaying on the display a particular representation of an object associated with the computer-generated scene; receiving, via the one or more input devices, user input directed to the particular representation of the object; in response to receiving the user input directed to the particular representation of the object, providing a first manipulation mode associated with a first set of conformal spatial manipulations of the corresponding object; receiving, via the one or more input devices, a user input switching from the first manipulation mode to a second manipulation mode associated with a second set of conformal spatial manipulations of the corresponding object; as well as In response to receiving the user input to switch from the first manipulation mode to the second manipulation mode, the second manipulation mode is provided.

22. The electronic device of claim 20, wherein the one or more processors are configured to receive the user input associating the one or more objects with the computer-generated scene via the one or more input devices by: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input to spatially manipulate the particular object, wherein the particular object is associated with a spatial manipulation point; as well as A spatial property of the specific object is changed based on the user input and the spatial manipulation point.

23. The electronic device of claim 20, wherein the one or more processors are configured to receive the user input associating the one or more objects with the computer-generated scene via the one or more input devices by: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input directed to the representation of a particular object, the particular object being associated with a parameter; displaying, on the display, a plurality of representations of the particular object from a first perspective, wherein each representation of the plurality of representations of the particular object is associated with a different respective value of the parameter; receiving, via the one or more input devices, a user input to change the first perspective to a second perspective; as well as In response to receiving the user input to change the first perspective to the second perspective, the plurality of representations of the particular object are displayed on the display from the second perspective.

24. An electronic device according to claim 20, wherein the behavior includes a trigger associated with a first object among the one or more objects, wherein the behavior includes an action associated with a second object among the one or more objects, and the one or more processors are further used to display a representation of the first object on the display with a first highlight and to display the representation of the second object on the display with a second highlight different from the first highlight.

25. A non-transitory computer-readable medium having encoded thereon instructions that, when executed by an electronic device comprising one or more input devices, one or more processors, and a display, cause the electronic device to: receiving, via the one or more input devices, user input for generating a computer-generated scene; receiving, via the one or more input devices, user input associating an anchor with the computer-generated scene, wherein the anchor is associated with a visual characteristic; receiving, via the one or more input devices, user input associating one or more objects with the computer-generated scene; displaying on a display a graphical user interface for associating a behavior with the computer-generated scene, wherein a first state of the graphical user interface displays a set of triggers and a second state of the graphical user interface displays a set of actions; receiving user input via the one or more input devices, the user input defining a corresponding relationship between one or more triggers and one or more actions by selecting one or more triggers from the set of triggers displayed in the first state of the graphical user interface and selecting one or more actions from the set of actions displayed in the second state of the graphical user interface; as well as When the computer-generated scene including the one or more objects is displayed on the display in association with the visual characteristic associated with the anchor: receiving, via the one or more input devices, user input that satisfies the one or more triggers; and In response to receiving the user input satisfying the one or more triggers, The one or more actions are performed, including modifying the computer-generated scene on the display.

26. The non-transitory computer readable medium of claim 25, wherein the instructions, when executed, cause the electronic device to receive user input via the one or more input devices to associate the one or more objects with the computer-generated scene by: displaying on the display a particular representation of an object associated with the computer-generated scene; receiving, via the one or more input devices, user input directed to the particular representation of the object; in response to receiving the user input directed to the particular representation of the object, providing a first manipulation mode associated with a first set of conformal spatial manipulations of the corresponding object; receiving, via the one or more input devices, a user input switching from the first manipulation mode to a second manipulation mode associated with a second set of conformal spatial manipulations of the corresponding object; as well as In response to receiving the user input to switch from the first manipulation mode to the second manipulation mode, the second manipulation mode is provided.

27. The non-transitory computer readable medium of claim 25, wherein the instructions, when executed, cause the electronic device to receive user input via the one or more input devices to associate the one or more objects with the computer-generated scene by: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input to spatially manipulate the particular object, wherein the particular object is associated with a spatial manipulation point; as well as A spatial property of the specific object is changed based on the user input and the spatial manipulation point.

28. The non-transitory computer readable medium of claim 25, wherein the instructions, when executed, cause the electronic device to receive user input via the one or more input devices to associate the one or more objects with the computer-generated scene by: displaying on the display a representation of a particular object associated with the computer-generated scene; receiving, via the one or more input devices, a user input directed to the representation of the particular object, the particular object being associated with a parameter; displaying, on the display, a plurality of representations of the particular object from a first perspective, wherein each representation of the plurality of representations of the particular object is associated with a different respective value of the parameter; receiving, via the one or more input devices, a user input to change the first perspective to a second perspective; as well as In response to receiving the user input to change the first perspective to the second perspective, the plurality of representations of the particular object are displayed on the display from the second perspective.

29. A non-transitory computer-readable medium according to claim 25, wherein the behavior includes a trigger associated with a first object among the one or more objects, wherein the behavior includes an action associated with a second object among the one or more objects, and wherein the instructions, when executed, further cause the electronic device to display a representation of the first object on the display with a first highlight and to display the representation of the second object on the display with a second highlight different from the first highlight.

Images