Device, method and graphical user interface for manipulating 3d objects on 2d screen

By implementing graphical user interface and spatial manipulation of user interface elements on electronic devices, the problem of complexity of CGR application design is solved, and the manipulation efficiency and user experience of three-dimensional objects on two-dimensional screens are improved.

CN118981271APending Publication Date: 2024-11-19APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410809940.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-09-27
Filing Date
2020-05-29
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the prior art, computer-generated reality (CGR) applications are complex and time-consuming to design and code, especially the difficulty in positioning and orientation operations of 3D objects in the scene, resulting in high barriers and cognitive burdens.

Method used

Provides an electronic device for a two-dimensional screen, allowing users to quickly and efficiently manipulate CGR objects in three-dimensional space, including displaying space manipulating user interface elements and detecting users Input is in response to changes in the virtual camera viewing angle, thereby simplifying spatial operation of CGR objects.

Benefits of technology

Reduces cognitive burden related to the user interface, improves operating efficiency, is suitable for battery-driven devices, extends battery life, and provides a more intuitive user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981271A_ABST
    Figure CN118981271A_ABST
Patent Text Reader

Abstract

One or more embodiments of the present disclosure relate to devices, methods, and graphical user interfaces for manipulating 3D objects on a 2D screen. A method of spatially manipulating a three-dimensional object includes displaying a spatial manipulation user interface element in association with the three-dimensional object. The spatial manipulation user interface element includes a first set of spatial manipulation affordance associated with a first set of spatial manipulations when the three-dimensional object is displayed from a first virtual camera perspective. When the three-dimensional object is displayed from a second virtual camera perspective, the spatial manipulation user interface element includes a second set of spatial manipulation affordance associated with a second set of spatial manipulations. The first set of spatial maneuvers includes at least one spatial maneuver excluded from the second set of spatial maneuvers and the second set of spatial maneuvers includes at least one spatial maneuver excluded from the first set of spatial maneuvers.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 035151, international application date May 29, 2020, application number 202080029991.X, and invention name “Device, method and graphical user interface for manipulating 3D objects on a 2D screen”, which entered the Chinese national phase on October 20, 2021.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application claims priority to U.S. Provisional Patent No. 62 / 856,056 filed on June 1, 2019 and U.S. Provisional Patent No. 62 / 906,936 filed on September 27, 2019, both of which are hereby incorporated by reference in their entirety. Technical Field

[0004]

[0013] This document generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces for spatially manipulating three-dimensional objects on a two-dimensional screen. Background Art

[0005] Programming computer generated reality (CGR) applications can be a difficult and time-consuming process, requiring expertise in, for example, 3D object design and application coding. This creates a high barrier to the generation of high-quality CGR applications.

[0006] In particular, placing one or more CGR objects at a desired location and / or orientation within a CGR scene can be a cumbersome and / or confusing process. Summary of the invention

[0007] Therefore, there is a need for electronic devices with faster, more efficient methods and interfaces for spatially manipulating three-dimensional objects (such as CGR objects) using a two-dimensional screen. Such methods and interfaces optionally supplement or replace conventional methods for spatially manipulating three-dimensional objects using a two-dimensional screen. Such methods and interfaces reduce the cognitive burden on the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0008] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some specific implementations, the device is a desktop computer. In some specific implementations, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some specific implementations, the device has a touchpad. In some specific implementations, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some specific implementations, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some specific implementations, the user interacts with the GUI primarily through stylus and / or finger contact and gestures on the touch-sensitive surface. In some specific implementations, these functions optionally include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] According to some specific implementations, a method is performed at a device having one or more processors, a non-volatile memory, and a display. The method includes displaying a three-dimensional object in a three-dimensional space from a first virtual camera perspective. The method includes displaying a spatial manipulation user interface element including a first set of spatial manipulation affordances associated with a first set of spatial manipulations, respectively. The method includes detecting a user input that changes the first virtual camera perspective to a second virtual camera perspective. In response to detecting a user input that changes the first virtual camera perspective to a second virtual camera perspective, the method includes: displaying a three-dimensional object in a three-dimensional space from a second virtual camera perspective; and displaying a spatial manipulation user interface element including a second set of spatial manipulation affordances associated with a second set of spatial manipulations, respectively, wherein the first set of spatial manipulations includes at least one spatial manipulation excluded from the second set of spatial manipulations and the second set of spatial manipulations includes at least one spatial manipulation excluded from the first set of spatial manipulations.

[0010] According to some specific implementations, an electronic device includes a display, one or more input devices, one or more processors, a non-transitory memory, and one or more programs; one or more programs are stored in the non-transitory memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some specific implementations, a non-transitory computer-readable storage medium has instructions stored therein, which, when executed by one or more processors of an electronic device having a display and one or more input devices, cause the device to perform or cause the execution of the operations of any of the methods described herein. According to some specific implementations, a graphical user interface on an electronic device having a display, one or more input devices, a non-transitory memory, and one or more processors configured to execute one or more programs stored in the non-transitory memory includes one or more elements of the elements displayed in any of the above methods, which are updated in response to input, as described in any of the methods described herein. According to some specific implementations, an electronic device includes: a display, one or more input devices; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some specific implementations, an information processing device used in an electronic device having a display and one or more input devices includes a device for performing or causing the execution of the operations of any of the methods described herein.

[0011] Thus, electronic devices having displays and input devices (such as touch-sensitive surfaces) are provided with faster, more efficient methods and interfaces for spatially manipulating three-dimensional objects, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for spatially manipulating three-dimensional objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various described implementations, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the drawings.

[0013] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some implementations.

[0014] Figure 1B is a block diagram illustrating example components for event processing according to some implementations.

[0015] Figure 2 A portable multifunction device with a touch screen is shown according to some implementations.

[0016] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some implementations.

[0017] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown in accordance with some implementations.

[0018] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface that is separate from the display is shown in accordance with some implementations.

[0019] Figure 5 is a block diagram of an exemplary operational architecture according to some specific implementations.

[0020] Figure 6 is a block diagram of an example controller according to some implementations.

[0021] Figure 7 is a block diagram of an exemplary electronic device according to some implementations.

[0022] FIG. 8A to FIG. 8Y An exemplary user interface for generating a CGR file according to some implementations is shown.

[0023] Fig. 9 is a flowchart representation of a method for spatially manipulating a three-dimensional object in three-dimensional space according to some specific implementations. DETAILED DESCRIPTION

[0024] According to various implementations, a graphical user interface (GUI) is provided to simplify spatial manipulation of three-dimensional objects in three-dimensional space, such as CGR objects in a CGR scene.

[0025] under, Figure 1A to Figure 1B , Figures 2 to 3 as well as FIG. 4A to FIG. 4B A description of an exemplary CGR scene generation apparatus is provided. Figure 5 , Figure 6 and Figure 7 A description of an exemplary CGR scene rendering apparatus is provided. FIG. 8A to FIG. 8Y An exemplary user interface for spatially manipulating CGR objects in a CGR scene is shown. FIG. 8A to FIG. 8Y The user interface in Fig. 9 process.

[0026] Exemplary CGR file composition device

[0027] Reference will now be made in detail to implementations, embodiments of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure various aspects of the implementation.

[0028] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described specific implementations. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0029] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various embodiments described and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their groupings.

[0030] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0031] Implementations of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player. Exemplary implementations of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices are optionally used, such as a laptop or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a trackpad). It should also be understood that in some implementations, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a trackpad).

[0032] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0033] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0034] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0035] Attention is now turned to a specific implementation of a portable device with a touch-sensitive display. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some specific implementations. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0036] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0037] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0038] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 102 by other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) is optionally controlled by a memory controller 122.

[0039] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0040] In some implementations, peripherals interface 118, CPU 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other implementations, they are, optionally, implemented on separate chips.

[0041] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0042] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some implementations, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0043] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative implementations, the input controller 160 is optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0044] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some specific implementations, some or all of the visual output corresponds to user interface objects.

[0045] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary implementation, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0046] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other implementations. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In one exemplary implementation, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0047] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some implementations, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). The user optionally uses any suitable object or appendage such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some implementations, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0048] In some implementations, in addition to the touch screen, the device 100 optionally includes a touch pad for activating or deactivating specific functions. In some implementations, the touch pad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touch pad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0049] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0050] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some specific implementations, the optical sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some specific implementations, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user views other video conference participants on the touch screen, etc.).

[0051] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to force sensor controller 159 in I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some implementations, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some implementations, at least one contact force sensor is located on the back of device 100 opposite touch-sensitive display system 112 located on the front of device 100.

[0052] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some implementations, the proximity sensor turns off and disables the touch-sensitive display system 112 when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call).

[0053] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 163 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or an electromechanical device that converts energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). One or more tactile output generators 163 receive tactile feedback generation instructions from the tactile feedback module 133 and generate tactile outputs on the device 100 that can be felt by a user of the device 100. In some specific implementations, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some implementations, at least one tactile output generator sensor is located on a back portion of device 100 opposite touch-sensitive display system 112 located on a front portion of device 100 .

[0054] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some implementations, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver for obtaining information about the location of device 100.

[0055] In some implementations, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some implementations, the memory 102 stores a device / global internal state 157, as shown in FIG. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0056] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0057] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some specific implementations, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and The external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in the device. In some implementations, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0058] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on a touch pad.

[0059] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0060] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0061] In some implementations, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data if necessary, and then generates screen image data to be output to the display controller 156.

[0062] Haptic feedback module 133 includes various software components for generating instructions used by one or more tactile output generators 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0063] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0064] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0065] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0066] Contacts module 137 (sometimes called address book or contact list);

[0067] Telephone module 138;

[0068] Video conferencing module 139;

[0069] Email client module 140;

[0070] Instant messaging (IM) module 141;

[0071] Fitness support module 142;

[0072] A camera module 143 for still images and / or video images;

[0073] Image management module 144;

[0074] Browser module 147;

[0075] Calendar module 148;

[0076] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0077] A widget creator module 150 for forming a user-created widget 149-6;

[0078] Search module 151;

[0079] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0080] Memo module 153;

[0081] Map module 154; and / or

[0082] Online video module 155.

[0083] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0084] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0085] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0086] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0087] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0088] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: entering a character sequence corresponding to an instant message, modifying previously entered characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some specific implementations, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both telephony-based messaging (eg, messages sent using SMS or MMS) and Internet-based messaging (eg, messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0089] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0090] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0091] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0092] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0093] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0094] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some specific implementations, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some specific implementations, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0095] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0096] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0097] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some implementations, device 100 optionally includes an MP3 player such as an iPod MP3 player. (trademark of Apple Inc. (Cupertino, California)).

[0098] In conjunction with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc. according to user instructions.

[0099] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0100] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, the text input module 134, the email client module 140, and the browser module 147, the online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen 112 or on an external display connected wirelessly or via the external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some implementations, the instant messaging module 141 is used instead of the email client module 140 to send a link to a particular online video.

[0101] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various specific implementations. In some specific implementations, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0102] In some implementations, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a trackpad. By using the touch screen and / or the trackpad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0103] A predefined set of functions that are uniquely performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some implementations, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a main menu, or a root menu. In such implementations, a "menu button" is implemented using a trackpad. In some other implementations, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0104] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some implementations. In some implementations, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0105] Event classifier 170 receives event information and determines an application 136-1 and an application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some implementations, application 136-1 includes application internal state 192 that indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some implementations, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine application view 191 to which the event information is to be delivered.

[0106] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0107] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0108] In some implementations, the event monitor 171 sends requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 transmits event information. In other implementations, the peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0109] In some implementations, event classifier 170 also includes a hit view determination module 172 and / or an activity event identifier determination module 173 .

[0110] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0111] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0112] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0113] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some implementations, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other implementations, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other implementations, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain actively participating views.

[0114] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizer 180). In some implementations that include active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some implementations, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0115] In some implementations, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another implementation, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0116] In some implementations, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some implementations, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some implementations, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0117] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event from the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some specific implementations, the event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0118] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some specific implementations, the event includes a device rotating from one orientation to another orientation (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0119] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some specific implementations, event comparator 184 includes event definition 186. Event definition 186 includes definitions of events (e.g., predefined sub-event sequences), such as event 1 (187-1), event 2 (187-2), and others. In some specific implementations, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lift-off of the touch (touch end). In some implementations, the event also includes information for one or more associated event handlers 190.

[0120] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0121] In some implementations, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0122] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0123] In some implementations, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some implementations, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some implementations, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0124] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some implementations, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0125] In some implementations, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0126] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0127] In some implementations, the event handler 190 includes or has access to the data updater 176, the object updater 177, and the GUI updater 178. In some implementations, the data updater 176, the object updater 177, and the GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0128] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0129] Figure 2 A touch screen (e.g., Figure 1AThe portable multifunction device 100 of the embodiment of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics will occur when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have made contact with the device 100 (from right to left, from left to right, up and / or down). In some embodiments or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0130] The device 100 optionally also includes one or more physical buttons, such as a "home button" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some implementations, the menu button is implemented as a soft key in a GUI displayed on a touch screen display.

[0131] In some implementations, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and locking the device, a volume adjustment button 208, a user identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button pressed for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some implementations, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0132] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some specific implementations. The device 300 need not be portable. In some specific implementations, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169 described above. The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some specific implementations, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0133] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to an instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various specific implementations. In some specific implementations, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0134] Attention is now turned to the specific implementation of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0135] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 is shown according to some implementations. A similar user interface is optionally implemented on device 300. In some implementations, user interface 400 includes the following elements, or a subset or superset thereof:

[0136] Signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0137] Time 404;

[0138] Bluetooth indicator 405;

[0139] Battery status indicator 406;

[0140] A tray 408 with icons for commonly used applications, such as:

[0141] o an icon 416 of the phone module 138 labeled “Phone”, the icon 416 optionally including an indicator 414 of the number of missed calls or voicemails;

[0142] o an icon 418 of the email client module 140 labeled “Mail”, the icon 418 optionally including an indicator 410 of the number of unread emails;

[0143] o an icon 420 labeled "Browser" of the browser module 147; and

[0144] o Video and music player module 152 (also known as iPod (Apple Inc.

[0145] (a trademark of Cupertino, California)) module 152) is marked as

[0146] An icon 422 for “iPod”; and

[0147] Icons for other apps, such as:

[0148] o Icon 424 labeled "Text" of IM module 141;

[0149] o Icon 426 labeled “Calendar” of calendar module 148;

[0150] o Icon 428 labeled “Photos” of the image management module 144;

[0151] o Icon 430 labeled “Camera” of camera module 143;

[0152] o Icon 432 labeled “Online Video” of the online video module 155;

[0153] o Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0154] o an icon 436 labeled “MAP” of the map module 154;

[0155] o Icon 438 labeled “Weather” of the Weather widget 149-1;

[0156] o Icon 440 labeled as “clock” of the alarm clock widget 169-6;

[0157] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0158] o Icon 444 labeled "Notepad" of the Notepad module 153; and

[0159] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0160] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, in some implementations, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some implementations, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some implementations, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0161] Figure 4B A touch-sensitive surface 451 (eg, Figure 3A device (e.g., a tablet or touch pad 355) Figure 3 Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 451 and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0162] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some implementations, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some implementations, a touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these implementations, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0163] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some implementations, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contacts are optionally used simultaneously.

[0164] Exemplary CGR file rendering device

[0165] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0166] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0167] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0168] Examples of CGR include virtual reality and mixed reality.

[0169] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0170] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0171] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0172] Examples of mixed reality include augmented reality and augmented virtuality.

[0173] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0174] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0175] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0176] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one specific implementation, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0177] Figure 5 5 is a block diagram of an exemplary operational architecture 500 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operational architecture 500 includes a controller 110 and an electronic device 520.

[0178] In some implementations, the controller 510 is configured to manage and coordinate the user experience. In some implementations, the controller 510 includes a suitable combination of software, firmware, and / or hardware. Figure 2Controller 510 is described in more detail. In some implementations, controller 510 is a computing device that is located locally or remotely relative to scene 503. For example, controller 510 is a local server located within scene 503. As another example, controller 510 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 503. In some implementations, controller 510 is communicatively coupled to electronic device 520 via one or more wired or wireless communication channels 544 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). As another example, controller 510 is included in a housing of electronic device 520. In some implementations, the functionality of controller 510 is provided by electronic device 520 and / or combined with the electronic device.

[0179] In some implementations, the electronic device 520 is configured to present CGR content to the user. In some implementations, the electronic device 520 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 520 presents CGR content to the user via a display 522 while the user is physically present within a scene 503, the scene including a table 507 within a field of view 511 of the electronic device 520. In some implementations, the user holds the electronic device 520 in one or both of his / her hands. In some implementations, when providing CGR content, the electronic device 520 is configured to display virtual objects (e.g., a virtual cylinder 509) and to enable video pass-through of the scene 503 (e.g., including a representation 517 of the table 507) on the display 522. Reference is made below to Figure 3 The electronic device 520 is described in more detail.

[0180] In some implementations, the user wears the electronic device 520 on his / her head. For example, in some implementations, the electronic device 520 includes a head mounted system (HMS), a head mounted device (HMD), or a head mounted housing (HME). Therefore, the electronic device 520 includes one or more CGR displays configured to display CGR content. For example, in various implementations, the electronic device 520 surrounds the user's field of view. In some implementations, the electronic device 520 is a handheld device (such as a smart phone or a tablet) configured to present CGR content, and the user no longer wears the electronic device 520 but holds the device, with the display facing the user's field of view, and the camera facing the scene 503. In some implementations, the handheld device can be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 520 is replaced with a CGR cabin, housing, or chamber configured to present CGR content, in which the user no longer wears or holds the electronic device 520.

[0181] Figure 6 is a block diagram of an example of a controller 510 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some specific implementations, the controller 510 includes one or more processing units 602 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 606, one or more communication interfaces 608 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), BLUETOOTH, ZIGBEE, etc. type interface), one or more programming (e.g., I / O) interfaces 610, a memory 620, and one or more communication buses 604 for interconnecting these components and various other components.

[0182] In some implementations, the one or more communication buses 604 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 606 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0183] The memory 620 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 620 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 620 or the non-transitory computer-readable storage medium of the memory 620 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 630 and a CGR content module 640.

[0184] The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the CGR content module 640 is configured to manage and coordinate the presentation of CGR content for one or more users (e.g., a single set of CGR content for one or more users, or multiple sets of CGR content for corresponding groups of one or more users). To this end, in various implementations, the CGR content module 640 includes a data acquisition unit 642, a tracking unit 644, a coordination unit 646, and a data transmission unit 648.

[0185] In some implementations, the data acquisition unit 642 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the electronic device 520. To this end, in various implementations, the data acquisition unit 642 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0186] In some implementations, the tracking unit 644 is configured to map the scene 503 and to track at least the electronic device 520 relative to the scene 503. Figure 5 The location / position of the scene 503. To this end, in various specific implementations, the tracking unit 644 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0187] In some implementations, the coordination unit 646 is configured to manage and coordinate CGR content presented to the user by the electronic device 520. To this end, in various implementations, the coordination unit 646 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0188] In some implementations, the data transfer unit 648 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the electronic device 520. To this end, in various implementations, the data transfer unit 648 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0189] Although the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 are shown as residing on a single device (e.g., controller 510), it should be understood that in other specific implementations, any combination of the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 may be located in separate computing devices.

[0190] also, Figure 6It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 6 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0191] Figure 7 5 is a block diagram of an example of an electronic device 520 according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some specific implementations, the electronic device 520 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more CGR displays 712, one or more optional internal-facing and / or external-facing image sensors 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0192] In some implementations, the one or more communication buses 704 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices and sensors 706 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones 707A, one or more speakers 707B, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), and the like.

[0193] In some implementations, one or more CGR displays 712 are configured to display CGR content to the user. In some implementations, one or more CGR displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more CGR displays 712 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, the electronic device 520 includes a single CGR display. As another example, the electronic device 520 includes a CGR display for each eye of the user.

[0194] In some implementations, the one or more image sensors 714 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (either sensor may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 714 are configured to face forward so as to acquire image data corresponding to a scene that the user would see if the electronic device 520 were not present (and thus may be referred to as a scene camera). The one or more optional image sensors 714 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0195] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules and data structures or their subsets, including an optional operating system 730 and a CGR rendering module 740.

[0196] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the CGR presentation module 740 is configured to present CGR content to a user via one or more CGR displays 712 and / or I / O devices and sensors 706 (such as one or more speakers 707B). For this purpose, in various implementations, the CGR presentation module 740 includes a data acquisition unit 742, a CGR content presentation unit 744, and a data transmission unit 746.

[0197] In some specific implementations, the data acquisition unit 742 is configured to at least Figure 5 The controller 510 of the present invention acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). In various specific implementations, the data acquisition unit acquires the CGR file. To this end, in various specific implementations, the data acquisition unit 742 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0198] In some implementations, the CGR content presentation unit 744 is configured to present CGR content to a user. In various implementations, the CGR content presentation unit 744 presents the CGR content of the CGR file according to the rules set forth in the CGR file. To this end, in various implementations, the CGR content presentation unit 744 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0199] In some implementations, the data transfer unit 746 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the controller 510. To this end, in various implementations, the data transfer unit 746 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0200] Although the data acquisition unit 742, the CGR content presentation unit 744, and the data transmission unit 746 are shown as residing on a single device (e.g., Figure 5 However, it should be understood that in other specific implementations, any combination of the data acquisition unit 742, the CGR content presentation unit 744 and the data transmission unit 746 may be located in a separate computing device.

[0201] also, Figure 7 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0202] In various implementations, CGR content is presented to a user from a CGR file that includes data about the CGR content and also includes data describing how to present the CGR content. In various implementations, the CGR file includes data about one or more CGR scenes. The CGR file also includes data about triggers (e.g., detectable events) for presenting various CGR scenes. As an example, in various implementations, the CGR file includes data about a CGR scene representing a game board and also includes data indicating that a CGR scene representing a game board is to be presented when a horizontal flat surface is detected in the field of view of a scene camera.

[0203] The CGR file also includes data about one or more CGR objects associated with the corresponding CGR scene. The CGR file also includes data about triggers related to the actions of the CGR objects. As an example, in various specific implementations, the CGR file includes data about multiple CGR objects associated with a CGR scene representing a game board, each CGR object in the multiple CGR objects represents a game piece. The CGR file also includes data indicating the actions of the multiple CGR objects in response to the detected triggers, for example, when a collision of two game pieces is detected, stop displaying the game piece.

[0204] Creating such CGR files can be a difficult and time-consuming process, requiring, for example, expertise in 3D object design and application coding. This creates a high barrier to the generation of high-quality CGR applications. Therefore, in various embodiments, a graphical user interface (GUI) is provided to simplify the generation of CGR files, which can be read (or executed) by a user device to present a CGR application including the CGR content described by the CGR file. Specifically, placing one or more CGR objects in a desired position and / or orientation in a CGR scene can be a cumbersome and / or confusing process. Therefore, in various embodiments, a GUI is provided that includes spatial manipulation user interface elements for spatially manipulating CGR objects within a CGR scene.

[0205] User interface and associated processes

[0206] Attention now turns to a specific implementation of a user interface ("UI") and related processes that may be implemented on an electronic device, such as a portable multifunction device (PMD) 100 having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, or a device 300 having one or more processors, non-volatile memory, a display, and an input device.

[0207] FIG. 8A to FIG. 8Y 1 shows an exemplary user interface for spatially manipulating CGR objects in a CGR scene according to some specific implementations. The user interface in these figures is used to illustrate the process described below, including Fig. 9 Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and a display are combined), in some implementations, the device detects input on a touch-sensitive surface 451 that is separate from the display 450, such as Figure 4B as shown in .

[0208] Fig. 8A A CGR scene manipulation user interface 801 is shown displayed by a portable multifunction device 100 (hereinafter referred to as "device 100"). In various implementations, the CGR scene manipulation user interface 801 is displayed by a CGR file composition application that is executed by the device 100.

[0209] The CGR scene manipulation user interface 801 includes a CGR scene (e.g., a three-dimensional space). The CGR scene manipulation user interface 801 includes a ground plane indicator 802 with a grid 803 overlaid thereon. The CGR scene manipulation user interface 801 includes a CGR mug 810 (e.g., a three-dimensional object in a three-dimensional space). Fig. 8A , the CGR scene manipulation user interface 801 displays the CGR scene (and the CGR mug 810) from a first virtual camera perspective.

[0210] In various implementations, the CGR scene is associated with a global three-dimensional coordinate system, which includes an x-direction ( Fig. 8A from left to right), y direction ( Fig. 8A in the front to back) and z direction ( Fig. 8A from top to bottom). Fig. 8A , the CGR mug 810 is displayed at a first position within the CGR scene that is associated with an x-coordinate, a y-coordinate, and a z-coordinate of the global three-dimensional coordinate system.

[0211] Fig. 8AA user input 899A is shown directed toward the CGR mug 810. In various implementations, the user input 899A corresponds to a contact (eg, a tap) detected at the location of the CGR mug 810.

[0212] Figure 8B A CGR scene manipulation user interface 801 is shown in response to detecting a user input 899A directed toward a CGR mug 810. Figure 8B In FIG. 8 , the CGR spatial manipulation user interface 801 includes a spatial manipulation user interface element 820 surrounding a CGR mug 810. The spatial manipulation user interface element 820 indicates that the CGR mug 810 is the selected CGR object. Figure 8B , the spatial manipulation user interface element 820 includes a first set of spatial manipulation enable representations that are respectively associated with a first set of spatial manipulations.

[0213] Specifically, in Figure 8B , spatial manipulation user interface element 820 includes y-axis ring 821Y, x-axis arrow 822X, y-axis arrow 822Y, and z-axis arrow 822Z. Y-axis ring 821Y is a spatial manipulation affordance for scaling (e.g., resizing) a selected CGR object, rotating the selected CGR object about the y-axis, and translating the selected CGR object in two dimensions within the xz plane. In various implementations, y-axis ring 821Y includes edges (e.g., Figure 8B 821Y is shown as a bold line) and inside (within the edge). In various embodiments, the edge is a spatial manipulation enable representation for scaling the selected CGR object and rotating the selected CGR object, and the inside is a spatial manipulation enable representation for translating the selected CGR object in two dimensions. In various embodiments, when the y-axis ring 821Y is displayed, the spatial manipulation enable representation for translating the selected CGR object associated with the y-axis ring 821Y in two dimensions extends beyond the edge to cover the selected CGR object.

[0214] The x-axis arrow 822X is a spatial manipulation affordance for causing the selected CGR object to perform a one-dimensional translation along the x-axis. The x-axis arrow 822X is substantially aligned with the x-axis of the global three-dimensional coordinate system (including at least one point such that the center of the selected CGR object and the point define a line segment parallel to the x-axis), and points to a positive or negative value of the x-coordinate. The y-axis arrow 822Y is a spatial manipulation affordance for causing the selected CGR object to perform a one-dimensional translation along the y-axis. The y-axis arrow 822Y is substantially aligned with the y-axis of the global three-dimensional coordinate system (including at least one point such that the center of the selected CGR object and the point define a line segment parallel to the y-axis), and points to a positive or negative value of the y-coordinate. The z-axis arrow 822Z is ​​a spatial manipulation affordance for causing the selected CGR object to perform a one-dimensional translation along the z-axis. The z-axis arrow 822Z is ​​substantially aligned with the z-axis of the global three-dimensional coordinate system (including at least one point such that the center of the selected CGR object and the point define a line segment parallel to the z-axis), pointing to positive or negative values ​​of the z-coordinate.

[0215] exist Figure 8B , spatial manipulation user interface element 820 does not include a z-axis ring 821Z (described below), a spatial manipulation affordance for rotating a selected CGR object about the z-axis, etc. Spatial manipulation user interface element 820 includes a y-axis ring 821Y (rather than a z-axis ring 821Z) based on the first virtual camera perspective.

[0216] Device 100 determines whether the current virtual camera viewing angle is most perpendicular to the xz plane, xy plane, or yz plane of the global three-dimensional coordinate system. Fig. 8A If the current virtual camera angle is most perpendicular to the xy plane (as described below), the spatial manipulation user interface element 820 includes a y-axis ring 821Y. Figure 8M If the current virtual camera perspective is most perpendicular to the yz plane, the spatial manipulation user interface element 820 includes a z-axis ring 821Z. If the current virtual camera perspective is most perpendicular to the yz plane, the spatial manipulation user interface element 820 includes an x-axis ring, a spatial manipulation affordance for rotating the selected CGR object around the x-axis, and so on.

[0217] Figure 8B User input 899B is shown directed toward the interior of y-axis ring 821Y. In various implementations, user input 899B corresponds to a moving contact (e.g., a drag or touch and drag) detected with a starting position at a location within y-axis ring 821Y, an arcuate path to intermediate positions both to the left and above the starting position, and an ending position further to the left of the starting position.

[0218] Figure 8CThe CGR scene manipulation user interface 801 is shown in response to detecting a first portion of user input 899B directed toward the interior of the y-axis ring 821Y. In response to the first portion of user input 899B, the contact moving leftward and upward on the screen, the CGR mug 810 (and the surrounding spatial manipulation user interface elements 820) moves leftward and upward, which corresponds to a change in position of the global three-dimensional coordinate system of the CGR mug 810 within the xz plane (e.g., further leftward in the x-dimension and further upward in the z-dimension, as shown by the shadow 811 displayed below the CGR mug 810).

[0219] Fig.8D The CGR scene manipulation user interface 801 is shown relative to detecting a second portion of user input 899B directed toward the interior of the y-axis ring 821Y. In response to the second portion of user input 899B, further leftward and downward movement of the contact on the screen, the CGR mug 810 (and the surrounding spatial manipulation user interface elements 820) moves leftward and downward, corresponding to a further change in position of the global three-dimensional coordinate system of the CGR mug 810 within the xz plane (e.g., further leftward in the x-dimension and back downward in the z-dimension).

[0220] Fig. 8E Shows Fig.8D 822X, wherein user input 899C points to x-axis arrow 822X. In various implementations, user input 899C corresponds to a moving contact (e.g., drag or touch and drag) detected with a starting position at the position of x-axis arrow 822X, a straight path to a mid-position both to the right and above the starting position, and an ending position further to the right and above the mid-position.

[0221] Fig.8F The CGR scene manipulation user interface 801 is shown in response to detecting a first portion of user input 899C directed toward the x-axis arrow 822X. In response to the first portion of user input 899C, the contact moving rightward and upward on the screen, the CGR mug 810 (and the surrounding spatial manipulation user interface element 820) moves only to the right (e.g., not upward), which corresponds to a change in position along the x-axis in the global three-dimensional coordinate system of the CGR mug 810 (e.g., further to the right in the x-dimension).

[0222] Figure 8GThe CGR scene manipulation user interface 801 is shown in response to detecting a second portion of user input 899C directed toward the x-axis arrow 822X. In response to the second portion of user input 899C, further rightward and upward movement of the contact on the screen, the CGR mug 810 (and the surrounding spatial manipulation user interface element 820) moves only to the right (e.g., not upward), which corresponds to a further position change along the x-axis in the global three-dimensional coordinate system of the CGR mug 810 (e.g., further to the right in the x-dimension).

[0223] Thus, unlike the upward and downward components of user input 899B directed toward the interior of y-axis ring 821Y, the upward component of user input 899C directed toward x-axis arrow 822X does not change the vertical position of CGR mug 810 within CGR scene manipulation user interface 801 or the z-coordinate of CGR mug 810 within the global three-dimensional coordinate system. Thus, rather than translating two-dimensionally within the xz plane in response to user input 899B, CGR mug 810 translates one-dimensionally along the x-axis in response to user input 899C.

[0224] Figure 8H Shows Figure 8G 801, wherein user input 899D is directed to an edge of y-axis ring 821Y. In various implementations, user input 899D corresponds to a moving contact (e.g., drag or touch and drag) detected at a starting position at a position of the edge of y-axis ring 821Y at a first distance from the center of y-axis ring 821Y and an ending position at a second distance from the center of y-axis ring 821Y.

[0225] Figure 8I The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899D directed to the edge of the y-axis ring 821Y. Figure 8I 821Y) is scaled (e.g., increased in size). In various implementations, a user input (e.g., user input 899D) that moves away from the center of y-axis ring 821Y increases the size of CGR mug 810, whereas a user input that moves toward the center of y-axis ring 821Y decreases the size of CGR mug 810.

[0226] Figure 8J Shows Figure 8I801, wherein user input 899E points to an edge of y-axis ring 821Y. In various implementations, user input 899E corresponds to a moving contact (e.g., drag or touch and drag) detected at a starting position at the edge of y-axis ring 821Y and moving along y-axis ring 821Y (e.g., starting from a line perpendicular to the starting position and the center of y-axis ring 821Y).

[0227] Figure 8K1 The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899E directed to the edge of the y-axis ring 821Y. Figure 8K1 , the CGR mug 810 (rather than the spatial manipulation user interface element 820) is rotated about the y-axis. Since the rotation of the CGR mug 810 does not change the global three-dimensional coordinate system, the spatial manipulation user interface element 820 is not changed. However, in various implementations, the spatial manipulation user interface 820 is not aligned with the global three-dimensional coordinate system, but is aligned with the local three-dimensional coordinate system of the selected CGR object.

[0228] Figure 8K2 A CGR scene manipulation user interface 801 is shown in response to detecting a user input 899E having an alternative spatial manipulation user interface element 829 that is aligned with the local three-dimensional coordinate system of the CGR mug 810. The local three-dimensional coordinate system includes an x-axis aligned with the center of the CGR mug 810 and its handle, a z-axis aligned with the center of the CGR mug 810 and its top and bottom, and a y-axis perpendicular to both the x-axis and the z-axis. Thus, in Figure 8K2 , in response to detecting user input 899E, CGR mug 810 (and spatial manipulation user interface element 820) is rotated about the y-axis.

[0229] Figure 8K2 The x-axis arrow 822X in Figure 8J 822Y) is substantially aligned with the x-axis of the local three-dimensional coordinate system (including at least one point such that the center of the CGR mug 810 and the point define a line segment parallel to the x-axis). The y-axis arrow 822Y and the z-axis arrow 822Z are similarly aligned.

[0230] Figure 8L Shows Figure 8J 801, wherein a user input 899F is directed to a location away from the spatial manipulation user interface element 820. In various implementations, the user input 899F corresponds to a moving contact (e.g., a drag or a touch and drag) detected with a starting position at a location away from the spatial manipulation user interface element 820 and an ending position below the starting position.

[0231] Figure 8M The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899F directed away from a location of the spatial manipulation user interface element 820. Figure 8M In the CGR scene manipulation user interface 801, the CGR scene manipulation user interface 801 includes a second virtual camera perspective (different from FIG. 8A to FIG. 8L The CGR scene displayed by the first virtual camera perspective).

[0232] As described above, the device 100 determines whether the current virtual camera viewing angle is most perpendicular to the xz plane, xy plane, or yz plane of the global three-dimensional coordinate system. Fig. 8A If the current virtual camera angle is most perpendicular to the xy plane (e.g., the first virtual camera angle of view), the spatial manipulation user interface element 820 includes a y-axis ring 821Y. Figure 8M If the current virtual camera perspective is most perpendicular to the yz plane, the spatial manipulation user interface element 820 includes a z-axis ring 821Z. If the current virtual camera perspective is most perpendicular to the yz plane, the spatial manipulation user interface element 820 includes an x-axis ring, a spatial manipulation affordance for rotating the selected CGR object around the x-axis, and so on.

[0233] In response to the change in the virtual camera perspective, the spatial manipulation user interface element 820 is also changed. The spatial manipulation user interface element 820 no longer includes the y-axis ring 821Y, but the spatial manipulation user interface element 820 does include the z-axis ring 821Z. The z-axis ring 821Z is a spatial manipulation affordance for scaling (e.g., resizing) the selected CGR object, rotating the selected CGR object about the y-axis, and translating the selected CGR object in two dimensions within the xy plane. In various implementations, the z-axis ring 821Z includes an edge (e.g., Figure 8M 821Z is shown as a bold line) and inside (within the edge). In various embodiments, the edge is a spatial manipulation enable representation for scaling the selected CGR object and rotating the selected CGR object, and the inside is a spatial manipulation enable representation for translating the selected CGR object in two dimensions. In various embodiments, when the z-axis ring 821Z is displayed, the spatial manipulation enable representation for translating the selected CGR object associated with the z-axis ring 821Z in two dimensions extends beyond the edge to encompass the selected CGR object.

[0234] Therefore, in Figure 8M, spatial manipulation user interface element 820 includes a second set of spatial manipulation affordances respectively associated with a second set of spatial manipulations. The first set of spatial manipulations includes at least one spatial manipulation that is excluded from the second set of spatial manipulations (e.g., rotation about the y-axis or two-dimensional translation in the xz plane using y-axis ring 821Y) and the second set of spatial manipulations includes at least one spatial manipulation that is excluded from the first set of spatial manipulations (e.g., rotation about the z-axis or two-dimensional translation in the xy plane using z-axis ring 821Z).

[0235] Figure 8N Shows Figure 8M 801, wherein user input 899G is directed to the interior of z-axis ring 821Z. In various implementations, user input 899G corresponds to a moving contact (e.g., drag or touch and drag) detected with a starting position at a location within z-axis ring 821Z and an ending position to the left and above the starting position.

[0236] Fig.8O The CGR scene manipulation user interface 801 is shown in response to detecting user input 899G directed toward the interior of the z-axis ring 821Z. In response to user input 899B, the CGR mug 810 (and surrounding spatial manipulation user interface elements 820) moves left and up, corresponding to a change in position of the CGR mug 810 in the global three-dimensional coordinate system within the xy plane (e.g., left in the x dimension and back in the y dimension). The z dimension of the CGR mug 810 is unchanged.

[0237] Figure 8P Shows Fig.8O 801, wherein user input 899H is directed to an edge of z-axis ring 821Z. In various implementations, user input 899H corresponds to a moving contact (e.g., drag or touch and drag) detected at a starting position at a location of the edge of z-axis ring 821Z and moving along z-axis ring 821Z (e.g., starting from a line perpendicular to the starting position and the center of z-axis ring 821Z).

[0238] Figure 8Q1 A CGR scene manipulation user interface 801 is shown in response to detecting a user input 899H directed toward an edge of the z-axis ring 821Z. Figure 8Q1 In the embodiment of the present invention, the CGR mug 810 (rather than the spatial manipulation user interface element 820) is rotated about the z-axis. Since the rotation of the CGR mug 810 does not change the global three-dimensional coordinate system, the spatial manipulation user interface element 820 is not changed. However, in various implementations, the spatial manipulation user interface 820 is not aligned with the global three-dimensional coordinate system, but is aligned with the local three-dimensional coordinate system of the selected CGR object.

[0239] Figure 8Q2 The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899H with an alternative spatial manipulation user interface element 829 that is aligned with the local three-dimensional coordinate system of the CGR mug 810. Figure 8Q2 , in response to detecting user input 899H, CGR mug 810 (and spatial manipulation user interface element 820) is rotated about the z-axis.

[0240] Figure 8R Shows Figure 8Q1 801, wherein user input 899I points to z-axis arrow 822Z. In various implementations, user input 899I corresponds to a contact (e.g., a tap) detected at the location of z-axis arrow 822Z.

[0241] Figure 8S The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899I pointing to the z-axis arrow 822Z. Figure 8S In the figure, the z-axis arrow 822Z is Figure 8R The spatial manipulation user interface element 820 is displayed in a different manner (e.g., highlighted, in a different color, in a different size, in a different brightness, etc.) to indicate that it is locked to the z-axis. Therefore, a change in the virtual camera perspective does not change the set of spatial manipulation affordances of the spatial manipulation user interface element 820.

[0242] Figure 8S User input 899J is shown directed to a location away from spatial manipulation user interface element 820. In various implementations, user input 899J corresponds to a moving contact (e.g., a drag or touch and drag) detected with a starting position at a location away from spatial manipulation user interface element 820 and an ending position above the starting position.

[0243] Figure 8T The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899J directed away from a location of the spatial manipulation user interface element 820. Figure 8T , the CGR scene manipulation user interface 801 includes a first virtual camera perspective ( FIG. 8A to FIG. 8L The first virtual camera perspective is different from Figure 8M to Figure 8S The CGR scene displayed by the second virtual camera perspective).

[0244] As described above, in various implementations, device 100 determines whether the current virtual camera viewing angle is most perpendicular to the xz plane, xy plane, or yz plane of the global three-dimensional coordinate system, and displays the corresponding ring. However, when the spatial manipulation user interface element is locked to an axis, the corresponding ring is displayed regardless of the viewing angle of the virtual camera.

[0245] Therefore, in Figure 8T , spatial manipulation user interface element 820 also includes z-axis ring 821Z even though the first virtual camera perspective is most perpendicular to the xz plane (and if spatial manipulation user interface element 820 was not locked, y-axis ring 821Y would be displayed).

[0246] Figure 8U The CGR scene manipulation user interface 801 is shown with a user input 899K directed to an edge of the z-axis ring 821Z. In various implementations, the user input 899K corresponds to a moving contact (e.g., a drag or a touch and drag) detected at a starting position at a location of the edge of the z-axis ring 821Z and moving along the z-axis ring 821Z (e.g., starting from a line perpendicular to the starting position and the center of the z-axis ring 821Z).

[0247] Figure 8V A CGR scene manipulation user interface 801 is shown in response to detecting a user input 899K directed toward an edge of the z-axis ring 821Z. Figure 8V , the CGR mug 810 (rather than the spatial manipulation user interface element 820) is rotated about the z-axis.

[0248] Figure 8W Shows Figure 8V The CGR scene of FIG. 801 manipulates the user interface 801, wherein the user input 899L points to the x-axis arrow 822X. In various implementations, the user input 899L corresponds to a contact (eg, a tap) detected at the location of the x-axis arrow 822X.

[0249] Figure 8X The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899L pointing to the x-axis arrow 822X. Figure 8X , z-axis arrow 822Z is ​​displayed in its original manner, and x-axis arrow 822X is displayed in a different manner to indicate that spatial manipulation user interface element 820 is locked to the x-axis. Thus, a change in the virtual camera perspective does not change the set of spatial manipulation affordances for spatial manipulation user interface element 820. However, changing the axis to which spatial manipulation user interface element 820 is locked changes the set of spatial manipulation affordances. Specifically, Figure 8X , spatial manipulation user interface element 820 includes an x-axis ring 821X instead of a z-axis ring 821Z.

[0250] The x-axis ring 821X is a spatial manipulation affordance for scaling (e.g., resizing) the selected CGR object, rotating the selected CGR object about the x-axis, and translating the selected CGR object in two dimensions within the yz plane. In various implementations, the x-axis ring 821X includes edges (e.g., Figure 8X In various embodiments, the edges are spatial manipulation affordances for scaling the selected CGR object and rotating the selected CGR object, and the interior is spatial manipulation affordances for translating the selected CGR object in two dimensions. In various embodiments, when x-axis ring 821X is displayed, the spatial manipulation affordances for translating the selected CGR object associated with x-axis ring 821X in two dimensions extend beyond the edges to encompass the selected CGR object. When the interior of x-axis ring 821X is smaller, as shown Figure 8X Thus, a user can perform a two-dimensional translation of the CGR mug 810 in the yz plane by dragging and dropping the CGR mug 810 by its handles (the handles are outside the interior of the x-axis ring 821X).

[0251] Figure 8X User input 899M is shown directed to an edge of the x-axis ring 821X. In various implementations, the user input 899M corresponds to a moving contact (e.g., a drag or a touch and drag) detected at a starting position at a location of the edge of the x-axis ring 821X and moving along the x-axis ring 821X (e.g., starting from a line perpendicular to the starting position and the center of the x-axis ring 821X).

[0252] Figure 8Y The CGR scene manipulation user interface 801 is shown in response to detecting a user input 899M directed to the edge of the x-axis ring 821X. Figure 8Y , the CGR mug 810 (rather than the spatial manipulation user interface element 820) is rotated about the x-axis.

[0253] Fig. 9 9 is a flowchart representation of a method 900 for spatially manipulating a three-dimensional object in a three-dimensional space according to some implementations. In various implementations, the method 900 is performed by a computer having one or more processors, non-volatile memory, a display, and one or more input devices (e.g., Figure 1A The portable multifunction device 100 or Figure 5 The method 900 is performed by a device of the electronic device 520). In some specific implementations, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 900 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0254] Method 900 begins at block 910, where a device displays a three-dimensional object in a three-dimensional space from a first virtual camera perspective, where the three-dimensional space is defined by a three-dimensional coordinate system including three perpendicular axes. Figure 8J , the device 100 displays the CGR mug 810 in the CGR scene from a first virtual camera perspective.

[0255] In various implementations, the three-dimensional coordinate system is a global three-dimensional coordinate system that does not change in response to spatial manipulation of the three-dimensional object. Figure 8K1 , device 100 displays a spatial manipulation user interface element 820 that includes an x-axis arrow 822X, a y-axis arrow 822Y, and a z-axis arrow 822Z aligned with three perpendicular axes of a global three-dimensional coordinate system of the CGR scene. In various implementations, the three-dimensional coordinate system is a local three-dimensional coordinate system that is fixedly aligned with the three-dimensional object when the three-dimensional object moves in a global three-dimensional coordinate system that does not change in response to spatial manipulation of the three-dimensional object. For example, in Figure 8K2 , device 100 displays a spatial user interface element 820 that includes an x-axis arrow 822X, a y-axis arrow 822Y, and a z-axis arrow 822Z aligned with three vertical axes of the local three-dimensional coordinate system of the CGR mug 810 after the CGR mug 810 has been moved (e.g., rotated) within the global three-dimensional coordinate system of the CGR scene.

[0256] Method 900 continues at block 920, where the device displays a spatial manipulation user interface element including a first set of spatial manipulation affordances respectively associated with a first set of spatial manipulations of the three-dimensional object, wherein the first set of spatial manipulations is based on a first virtual camera perspective. Figure 8J , device 100 displays a spatial manipulation user interface element 820, which includes an x-axis arrow 822X for one-dimensional translation along the x-axis, a y-axis arrow 822Y for one-dimensional translation along the y-axis, a z-axis arrow 822Z for one-dimensional translation along the z-axis, and a y-axis ring 821Y for rotation around the y-axis (based on the y-axis being the axis closest to being parallel to the first virtual camera perspective, or in other words, based on the xz plane being the plane closest to being perpendicular to the first virtual camera perspective), and so on.

[0257] Method 900 continues at block 930, where the device detects user input to change the first virtual camera perspective to a second virtual camera perspective. Figure 8L , device 100 detects user input 899F to change the virtual camera viewing angle.

[0258] Method 900 continues at block 940, where the device displays the three-dimensional object in the three-dimensional space from the second virtual camera perspective in response to detecting a user input to change the first virtual camera perspective to the second virtual camera perspective. Figure 8M In response to detecting user input 899F, device 100 displays CRG mug 810 from a second virtual camera perspective.

[0259] Method 900 continues in box 950, where the device also displays a spatial manipulation user interface element including a second set of spatial manipulation enable representations associated with a second set of spatial manipulations of three-dimensional objects, respectively, in response to detecting user input to change the first virtual camera perspective to a second virtual camera perspective, wherein the second set of spatial manipulations is based on the second virtual camera perspective, wherein the first set of spatial manipulations includes at least one spatial manipulation that is excluded from the second set of spatial manipulations and the second set of spatial manipulations includes at least one spatial manipulation that is excluded from the first set of spatial manipulations.

[0260] For example, in Figure 8M In response to detecting user input 899F, device 100 displays a spatial manipulation user interface element 820, which includes an x-axis arrow 822X for one-dimensional translation along the x-axis, a y-axis arrow 822Y for one-dimensional translation along the y-axis, a z-axis arrow 822Z for one-dimensional translation along the z-axis, and a z-axis ring 821Z for rotation around the z-axis (based on the z-axis being the axis closest to being parallel to the second virtual camera's perspective, or in other words, based on the xy plane being the plane closest to being perpendicular to the second virtual camera's perspective), etc. Although Figure 8J The spatial manipulation user interface element 820 includes an affordance for rotating about the y-axis (e.g., y-axis ring 821Y), but Figure 8J The spatial manipulation user interface element 820 does not include an affordance for rotating about the z-axis. Figure 8M The spatial manipulation user interface element 820 includes an affordance for rotating about the z-axis (e.g., z-axis ring 821Z), but Figure 8M The spatial manipulation user interface element 820 does not include an affordance for rotating about the y-axis.

[0261] In various implementations, at least one spatial manipulation excluded from the second set of spatial manipulations includes a two-dimensional translation in a first plane perpendicular to the three perpendicular axes that is closest to a first axis parallel to the first virtual camera's field of view, and at least one spatial manipulation excluded from the first set of spatial manipulations includes a two-dimensional translation in a second plane perpendicular to the three perpendicular axes that is closest to a second axis parallel to the second virtual camera's field of view. For example, while Figure 8JThe spatial manipulation user interface element 820 includes an affordance for two-dimensional translation in the xz plane (e.g., y-axis ring 821Y), but Figure 8J The spatial manipulation user interface element 820 does not include an affordance for two-dimensional translation in the xy plane. Figure 8M The spatial manipulation user interface element 820 includes an affordance for two-dimensional translation in the xy plane (e.g., a z-axis ring 821Z), but Figure 8M The spatial manipulation user interface element 820 does not include an affordance for two-dimensional translation in the xz plane.

[0262] In various implementations, at least one spatial manipulation excluded from the second set of spatial manipulations includes a rotation about a first axis of three perpendicular axes that is closest to parallel to the first virtual camera's perspective and at least one spatial manipulation excluded from the first set of spatial manipulations includes a rotation about a second axis of three perpendicular axes that is closest to parallel to the second virtual camera's perspective. Figure 8J The spatial manipulation user interface element 820 includes an affordance for rotating about the y-axis (e.g., y-axis ring 821Y), but Figure 8J The spatial manipulation user interface element 820 does not include an affordance for rotating about the z-axis. Figure 8M The spatial manipulation user interface element 820 includes an affordance for rotating about the z-axis (e.g., z-axis ring 821Z), but Figure 8M The spatial manipulation user interface element 820 does not include an affordance for rotating about the y-axis.

[0263] In various implementations, the first set of spatial manipulations and the second set of spatial manipulations include one-dimensional translations along each of three perpendicular axes. For example, Figure 8J and Figure 8M The spatial manipulation user interface elements 820 for both include an x-axis arrow 822X for one-dimensional translation along the x-axis, a y-axis arrow 822Y for one-dimensional translation along the y-axis, and a z-axis arrow 822Z for one-dimensional translation along the z-axis.

[0264] In various implementations, the first set of spatial manipulations and the second set of spatial manipulations include scaling. Figure 8J The spatial manipulation user interface element 820 includes an affordance for zooming (e.g., y-axis ring 821Y), and Figure 8M The spatial manipulation user interface element 820 also includes an affordance for zooming (e.g., z-axis ring 821Z).

[0265] In various implementations, the spatial manipulation user interface element is displayed in association with a three-dimensional object. In various implementations, the spatial manipulation user interface element is displayed as an overlay on the three-dimensional object. In various implementations, the spatial manipulation user interface element is displayed around the three-dimensional object. For example, in Figure 8J , spatial manipulation user interface elements 820 are displayed as overlaid on and around CGR mug 810.

[0266] In various implementations, method 900 includes detecting user input that locks a spatial manipulation user interface element to a second set of spatial manipulations while displaying the three-dimensional object in the three-dimensional space from a second virtual camera perspective. Figure 8R , device 100 detects user input 899I to lock spatial manipulation user interface element 820 to the z-axis.

[0267] In various implementations, method 900 includes detecting a user input to change the second virtual camera perspective to a third virtual camera perspective. Figure 8S , device 100 detects user input 899J to change the virtual camera perspective back to the first virtual camera perspective.

[0268] In various implementations, method 900 includes displaying a three-dimensional object in three-dimensional space from a third virtual camera perspective in response to detecting a user input to change the second virtual camera perspective to a third virtual camera perspective and based on determining that the spatial manipulation user interface element is locked to a second set of spatial manipulations. Figure 8T , the device 100 displays the CGR mug 810 from a first virtual camera perspective.

[0269] In various implementations, method 900 further includes, in response to detecting user input to change the second virtual camera perspective to a third virtual camera perspective and based on determining that the spatial manipulation user interface element is locked to the second set of spatial manipulations, displaying a spatial manipulation user interface element including a second set of spatial manipulation affordances. Figure 8T , device 100 displays spatial manipulation user interface element 820 including z-axis ring 821Z instead of y-axis ring 821Y, even if the y-axis is most nearly perpendicular to the current virtual camera perspective (eg, the first virtual camera perspective).

[0270] In various implementations, method 900 includes detecting a third set of spatial manipulation user inputs that lock spatial manipulation user interface elements to the three-dimensional object while the three-dimensional object is displayed in the three-dimensional space from a second virtual camera perspective. Figure 8W , device 100 detects user input 899L to lock spatial manipulation user interface element 820 to the x-axis.

[0271] In various implementations, method 900 includes displaying the three-dimensional object in three-dimensional space from a second virtual camera perspective in response to detecting a third set of spatial manipulation user inputs that locks the spatial manipulation user interface elements to the three-dimensional object. Figure 8X In response to detecting user input 899L, device 100 receives a signal from Figure 8W The same virtual camera perspective in FIG. 8 shows the CGR mug 810 .

[0272] In various embodiments, method 900 also includes, in response to detecting user input that locks the spatial manipulation user interface element to a third group of spatial manipulations of the three-dimensional object, displaying a spatial manipulation user interface element including a third group of spatial manipulation enable representations associated with the third group of spatial manipulations of the three-dimensional object, wherein the third group of spatial manipulations includes at least one spatial manipulation that is excluded from the second group of spatial manipulations and the second group of spatial manipulations includes at least one spatial manipulation that is excluded from the third group of spatial manipulations.

[0273] For example, in Figure 8X , device 100 displays a spatial manipulation user interface element 820 that includes an x-axis arrow 822X for one-dimensional translation along the x-axis, a y-axis arrow 822Y for one-dimensional translation along the y-axis, a z-axis arrow 822Z for one-dimensional translation along the z-axis, and (based on spatial manipulation user interface element 820 being locked to the x-axis) an x-axis ring 821X for rotation about the x-axis, and so on. Although Figure 8M The spatial manipulation user interface element 820 includes an affordance for rotating about the z-axis (e.g., z-axis ring 821Z), but Figure 8M The spatial manipulation user interface element 820 does not include an affordance for rotating about the x-axis. Figure 8X The spatial manipulation user interface element 820 includes an affordance for rotating about the x-axis (e.g., x-axis ring 821X), but Figure 8X The spatial manipulation user interface element 820 does not include an affordance for rotating about the z-axis.

[0274] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A , Figure 3 and Figure 5 described) or application-specific chips. In addition, the above reference Fig. 9 The described operations are optionally performed by Figure 1A to Figure 1B. For example, user input and user interface elements are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on touch-sensitive surface 604, and event distributor module 174 delivers event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally utilizes or calls data updater 176 or object updater 177 to update the application internal state 192. In some specific implementations, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A-1B How the components depicted in the figure can implement other processes.

[0275] For the purpose of explanation, the preceding description is described by reference to a specific implementation. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. Many modifications and variations are possible according to the above teachings. The implementation is selected and described in order to best illustrate the principles of the present invention and its practical application, so that other persons skilled in the art can best use the present invention and the various described implementations with various modifications suitable for the specific purpose contemplated.

Claims

1. A method comprising: At a device comprising one or more processors, non-transitory memory, one or more input devices, and a display: Displaying a three-dimensional object in a three-dimensional space, wherein the three-dimensional space is defined by a three-dimensional coordinate system; displaying a spatial manipulation user interface element, the spatial manipulation user interface element comprising a set of spatial manipulation affordances respectively associated with a set of spatial manipulations of the three-dimensional object, wherein each spatial manipulation affordance in the set of spatial manipulation affordances corresponds to a translational movement of the three-dimensional object along a corresponding axis of the three-dimensional space; detecting, via one or more input devices, a first user input directed to a first spatial manipulation affordance of the spatial manipulation affordances, wherein the first spatial manipulation affordance is associated with a first axis of the three-dimensional space; as well as In response to detecting the first user input directed toward the first spatial manipulation affordance, the three-dimensional object is translationally moved along the first axis of the three-dimensional space. 2 . The method of claim 1 , wherein each spatial manipulation in the set of spatial manipulations corresponds to a translational movement along a different axis of the three-dimensional space.

3. The method of claim 2, wherein the set of spatial manipulation affordances comprises: an x-axis arrow, the x-axis arrow enabling the three-dimensional object to translate along the x-axis of the three-dimensional space, a y-axis arrow, the y-axis arrow enabling the three-dimensional object to be translated along the y-axis of the three-dimensional space, and A z-axis arrow enables the three-dimensional object to be translated along the z-axis of the three-dimensional space. 4 . The method of claim 1 , further responsive to detecting the first user input, moving the spatial manipulation user interface element translationally along the first axis of the three-dimensional space.

5. A method according to claim 1, wherein the first user input includes a selection of the first spatial manipulation enable representation.

6. A method according to claim 1, wherein the first user input includes a moving contact starting at the first spatial manipulation enable representation. The method of claim 6 , wherein the moving contact corresponds to dragging or touch and dragging.

8. The method according to claim 1, further comprising: detecting, via one or more input devices, a second user input directed to a second spatial manipulation affordance of the spatial manipulation affordances, wherein the second spatial manipulation affordances is associated with a second axis of the three-dimensional space different from the first axis; as well as In response to detecting the second user input directed toward the second spatial manipulation affordance, the three-dimensional object is translationally moved along the second axis of the three-dimensional space.

9. A method according to claim 1, wherein the three-dimensional object is displayed from a specific virtual camera perspective, and wherein the display of the set of spatial manipulation enable representations is based on the specific virtual camera perspective.

10. The method of claim 1, wherein the spatial manipulation user interface element is displayed adjacent to the three-dimensional object.

11. The method of claim 1, wherein the spatial manipulation user interface element is displayed superimposed on the three-dimensional object.

12. The method of claim 1, wherein the three-dimensional coordinate system is a global three-dimensional coordinate system that does not change in response to spatial manipulation of the three-dimensional object.

13. A method according to claim 1, wherein the three-dimensional coordinate system is a local three-dimensional coordinate system, and the local three-dimensional coordinate system is fixedly aligned with the three-dimensional object when the three-dimensional object moves in a global three-dimensional coordinate system that does not change in response to spatial manipulation of the three-dimensional object.

14. A device comprising: one or more operating input devices; monitor; non-transitory memory; as well as One or more processors to: Displaying a three-dimensional object in a three-dimensional space, wherein the three-dimensional space is defined by a three-dimensional coordinate system; displaying a spatial manipulation user interface element, the spatial manipulation user interface element comprising a set of spatial manipulation affordances respectively associated with a set of spatial manipulations of the three-dimensional object, wherein each spatial manipulation in the set of spatial manipulations corresponds to a translational movement of the three-dimensional object along a corresponding axis of the three-dimensional space; detecting, via one or more input devices, a first user input directed to a first spatial manipulation affordance of the spatial manipulation affordances, wherein the first spatial manipulation affordance is associated with a first axis of the three-dimensional space; as well as In response to detecting a first user input directed toward the first spatial manipulation affordance, the three-dimensional object is translationally moved along the first axis of the three-dimensional space.

15. A device according to claim 14, wherein the first user input comprises a moving contact starting at the first spatial manipulation enable representation.

16. The apparatus of claim 14, wherein the one or more processors: detecting, via one or more input devices, a second user input directed to a second spatial manipulation affordance of the spatial manipulation affordances, wherein the second spatial manipulation affordances is associated with a second axis of the three-dimensional space different from the first axis; and In response to detecting the second user input directed toward the second spatial manipulation affordance, the three-dimensional object is translationally moved along the second axis of the three-dimensional space.

17. A method comprising: At a device comprising one or more processors, non-transitory memory, one or more input devices, and a display: while executing a computer generated reality (CGR) file composition application, displaying a three-dimensional object in a three-dimensional space via the display, wherein the three-dimensional space is defined by a three-dimensional coordinate system; detecting, via one or more input devices, a first user input directed toward the three-dimensional object in the three-dimensional space; as well as In response to detecting the first user input, a spatial manipulation user interface element is displayed via the display, the spatial manipulation user interface element comprising a set of spatial manipulation affordances respectively associated with a set of spatial manipulations of the three-dimensional object within the three-dimensional space.

18. The method according to claim 17, further comprising: detecting, via one or more input devices, a second user input directed to a first spatial manipulation affordance of the spatial manipulation affordances, wherein the first spatial manipulation affordance is associated with a first axis of the three-dimensional space; as well as In response to detecting the second user input directed toward the first spatial manipulation affordance, the three-dimensional object is translationally moved along the first axis of the three-dimensional space.

19. A method according to claim 18, wherein the second user input comprises a moving contact starting at the first spatial manipulation enable representation.

20. The method of claim 17, wherein each spatial manipulation in the set of spatial manipulations corresponds to one of: a translational movement along a different axis of the three-dimensional space; Rotational motion around different axes of the three-dimensional space; Or a scaling operation associated with the three-dimensional object.

21. A method according to claim 17, wherein the three-dimensional object is displayed from a specific virtual camera perspective, and wherein the display of the set of spatial manipulation enable representations is based on the specific virtual camera perspective.

22. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having a display, cause the device to: When executing the computer generated reality (CGR) file composition application, displaying the three-dimensional object in a three-dimensional space via the display, wherein the three-dimensional space is defined by a three-dimensional coordinate system; detecting, via one or more input devices, a first user input directed toward the three-dimensional object within the three-dimensional space; and In response to detecting the first user input, a spatial manipulation user interface element is displayed via the display, the spatial manipulation user interface element comprising a set of spatial manipulation affordances respectively associated with a set of spatial manipulations of the three-dimensional object within the three-dimensional space.

23. The non-transitory memory of claim 22, wherein the program, when executed by the one or more processors of the device, causes the device to further: detecting, via one or more input devices, a second user input directed to a first spatial manipulation affordance of the spatial manipulation affordances, wherein the first spatial manipulation affordance is associated with a first axis of the three-dimensional space; and In response to detecting a second user input directed toward the first spatial manipulation affordance, the three-dimensional object is translationally moved along the first axis of the three-dimensional space.

24. A non-transitory memory according to claim 23, wherein the second user input comprises a moving contact starting at the first spatial manipulation enable representation.

25. The non-transitory memory of claim 22, wherein each spatial manipulation in the set of spatial manipulations corresponds to one of: a translational movement along a different axis of the three-dimensional space; Rotational motion around different axes of the three-dimensional space; Or a scaling operation associated with the three-dimensional object.

26. A non-volatile memory according to claim 22, wherein the program, when executed by the one or more processors of the device, causes the device to display the three-dimensional object from a specific virtual camera perspective and to display the set of spatial manipulation enable representations based on the specific virtual camera perspective.