Device, method and graphical user interface for generating CGR objects

Abstract

In one embodiment, a method for generating a 3D object is performed by a device including a processor, a non-transitory memory, and one or more input devices. The method includes receiving user input via one or more input devices to select a file representing two-dimensional (2D) content and having a file type. The method includes receiving user input requesting to generate a three-dimensional (3D) object based on a file via one or more input devices. The method includes generating a 3D object representing 2D content based on the file type.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 16 / 403,835, filed May 6, 2019, which is incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that generate computer-generated reality (CGR) objects. Background Art

[0004] Various computer-aided design (CAD) tools exist to help generate three-dimensional objects, each of which is associated with a wireframe and one or more texture images. However, generating three-dimensional objects can be a difficult and time-consuming process. Summary of the invention

[0005] Therefore, there is a need for electronic devices with faster and more efficient methods and interfaces for writing computer generated reality (CGR) files. Such methods and interfaces optionally supplement or replace conventional methods for writing CGR files. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at a device having one or more processors, a non-volatile memory, and one or more input devices. The method includes receiving user input via one or more input devices to select a file representing two-dimensional (2D) content and having a file type. The method includes receiving user input requesting to generate a three-dimensional (3D) object based on a file via one or more input devices. The method includes selecting a first 3D object template in response to the file type of the 2D content being a first file type. The method includes selecting a second 3D object template different from the first 3D object template in response to the file type of the 2D content being a second file type different from the first file type. The method includes generating a 3D object representing 2D content by displaying the 2D content on at least a portion of a selected one of the first 3D object template and the second 3D object template based on the file type.

[0008] According to some embodiments, an electronic device includes a display, one or more input devices, one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein, which, when executed by one or more processors of an electronic device having a display and one or more input devices, cause the device to perform or cause the execution of the operations of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display, one or more input devices, a non-transitory memory, and one or more processors configured to execute one or more programs stored in the non-transitory memory includes one or more elements of the elements displayed in any of the above methods, which are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display, one or more input devices; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display and one or more input devices includes a device for performing or causing the execution of the operations of any of the methods described herein.

[0009] Thus, electronic devices having displays and input devices (such as touch-sensitive surfaces) are provided with faster, more efficient methods and interfaces for authoring CGR files, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for authoring CGR files. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0011] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0012] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0013] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0014] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0015] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0016] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0017] Figure 5 is a block diagram of an exemplary operating architecture according to some embodiments.

[0018] Figure 6 is a block diagram of an exemplary controller according to some embodiments.

[0019] Figure 7 is a block diagram of an exemplary apparatus according to some embodiments.

[0020] FIG. 8A to FIG. 8W An exemplary user interface for generating a 3D object according to some embodiments is shown.

[0021] Fig. 9 A flowchart of a method of generating a 3D object according to some embodiments is shown. DETAILED DESCRIPTION

[0022] According to various embodiments, a graphical user interface (GUI) is provided to simplify generating a 3D object based on a file representing 2D content (such as an image, video, or text). In various embodiments, a wireframe is selected based on the file type of the file (e.g., an image file or a video file), and one or more texture images representing the 2D content are generated.

[0023] under, Figure 1A to Figure 1B , Figures 2 to 3 as well as FIG. 4A to FIG. 4B A description of an exemplary CGR object generation apparatus is provided. Figure 5 , Figure 6 and Figure 7 A description of an exemplary CGR object rendering device is provided. FIG. 8A to FIG. 8W An exemplary user interface for generating a 3D object is shown. FIG. 8A to FIG. 8W The user interface in Fig. 9 process.

[0024] Exemplary CGR Object Generation Device

[0025] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0026] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0027] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0028] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0029] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices are optionally used, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0030] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0031] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0032] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0033] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0034] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0035] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0036] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0037] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0038] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0039] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0040] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0041] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0042] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

[0043] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0044] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0045] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0046] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0047] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0048] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0049] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0050] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0051] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 163 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or an electromechanical device that converts energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). One or more tactile output generators 163 receive tactile feedback generation instructions from the tactile feedback module 133 and generate tactile outputs on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front portion of the device 100 .

[0052] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver for obtaining information about the location of device 100.

[0053] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in FIG. 1A and FIG. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0054] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0055] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in Apple Inc. (Cupertino, California). iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0056] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touch pad.

[0057] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0058] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0059] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0060] Haptic feedback module 133 includes various software components for generating instructions used by one or more tactile output generators 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0061] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0062] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0063] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0064] Contacts module 137 (sometimes called address book or contact list);

[0065] Telephone module 138;

[0066] Video conferencing module 139;

[0067] Email client module 140;

[0068] Instant messaging (IM) module 141;

[0069] Fitness support module 142;

[0070] A camera module 143 for still images and / or video images;

[0071] Image management module 144;

[0072] Browser module 147;

[0073] Calendar module 148;

[0074] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0075] A widget creator module 150 for forming a user-created widget 149-6;

[0076] Search module 151;

[0077] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0078] Memo module 153;

[0079] Map module 154; and / or

[0080] Online video module 155.

[0081] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0082] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0083] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0084] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0085] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0086] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0087] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0088] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0089] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0090] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0091] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0092] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0093] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0094] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0095] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0096] In conjunction with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc. according to user instructions.

[0097] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0098] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0099] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0100] In some embodiments, the device 100 is a device on which operation of a predefined set of functions is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0101] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0102] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0103] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0104] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0105] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0106] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0107] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0108] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0109] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0110] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0111] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0112] Event distributor module 174 distributes the event information to event recognizers (e.g., event recognizer 180). In some embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0113] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0114] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0115] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0116] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (for example, from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0117] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0118] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0119] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0120] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0121] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0122] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0123] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0124] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0125] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0126] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0127] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user breaks contact with the one or more graphics, selection of the one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some embodiments or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0128] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0129] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0130] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169). The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0131] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0132] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0133] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0134] Signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0135] Time 404;

[0136] Bluetooth indicator 405;

[0137] Battery status indicator 406;

[0138] A tray 408 with icons for commonly used applications, such as:

[0139] o an icon 416 of the phone module 138 labeled “Phone”, which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0140] o an icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0141] o An icon 420 labeled “Browser” of the browser module 147; and

[0142] o an icon 422 labeled “iPod” of a video and music player module 152 (also referred to as an iPod (trademark of Apple Inc.) module 152); and

[0143] Icons for other apps, such as:

[0144] o Icon 424 of the IM module 141 labeled “Text”;

[0145] o Icon 426 labeled “Calendar” of calendar module 148;

[0146] o Icon 428 labeled “Photos” of the image management module 144;

[0147] o An icon 430 labeled “Camera” of the camera module 143;

[0148] o Icon 432 labeled “Online Video” of the online video module 155;

[0149] o Icon 434 labeled "Stock Market" of the Stock Market widget 149-2;

[0150] o an icon 436 labeled “MAP” of the map module 154;

[0151] o Icon 438 labeled "Weather" of the Weather widget 149-1;

[0152] o Icon 440 labeled “Clock” of the alarm clock widget 169-6;

[0153] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0154] o An icon 444 labeled "Notepad" of the notepad module 153; and

[0155] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0156] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, in some embodiments, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0157] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 451 and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0158] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0159] In addition, although the following examples are given primarily with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0160] Exemplary CGR Object Rendering Device

[0161] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0162] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0163] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0164] Examples of CGR include virtual reality and mixed reality.

[0165] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0166] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0167] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0168] Examples of mixed reality include augmented reality and augmented virtuality.

[0169] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0170] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0171] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0172] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0173] Figure 5 5 is a block diagram of an exemplary operational architecture 500A according to some embodiments. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, as a non-limiting example, the operational architecture 500A includes an electronic device 520A.

[0174] In some embodiments, the electronic device 520A is configured to present CGR content to the user. In some embodiments, the electronic device 520A includes a suitable combination of software, firmware and / or hardware. According to some embodiments, the electronic device 520A presents CGR content to the user via a display 522 when the user is physically present in a physical environment 503, which includes a table 507 within a field of view 511 of the electronic device 520A. Therefore, in some embodiments, the user holds the electronic device 520A in his / her hand. In some specific implementations, when providing augmented reality (AR) content, the electronic device 520A is configured to display an AR object (e.g., an AR column 509) and enable video pass-through of the physical environment 503 (e.g., including a representation 517 of the table 507) on the display 522.

[0175] Figure 6 5 is a block diagram of a controller 510 according to some embodiments. In some implementations, the operating environment includes the controller 510 and a head mounted device (HMD) 520B (eg, Figure 7 ). In some embodiments, the controller 510 is configured to manage and coordinate the presentation of CGR content for the user. In some embodiments, the controller 510 includes a suitable combination of software, firmware and / or hardware. In some embodiments, the controller 510 is a computing device that is locally or remotely located relative to the scene. For example, the controller 510 is a local server located within the scene. In another example, the controller 510 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene. In some embodiments, the controller 510 is communicatively coupled to the HMD 520B via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). As another example, the controller 510 is included in the housing of the HMD 520B.

[0176] In some embodiments, HMD 520B is configured to present CGR content to the user. In some embodiments, HMD 520B includes a suitable combination of software, firmware and / or hardware. In some embodiments, the functionality of controller 510 is provided by HMD 520B and / or combined with the HMD.

[0177] According to some embodiments, HMD 520B presents CGR content to the user while the user is virtually and / or physically present within the scene.

[0178] In some embodiments, the user wears the HMD 520B on his head. Thus, the HMD 520B includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the HMD 520B surrounds the user's field of view. In some embodiments, such as in Figure 5 In some embodiments, the HMD 520B is replaced with a handheld device (such as a smartphone or tablet) configured to present CGR content, and the user does not wear the HMD 520B but holds the device with the display facing the user's field of view and the camera facing the scene. In some embodiments, the handheld device can be placed in a housing that can be worn on the user's head. In some embodiments, the HMD 520B is replaced with a CGR cabin, housing, or chamber configured to present CGR content, in which the user does not wear or hold the HMD 520B.

[0179] While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and in order to not obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the controller 510 includes one or more processing units 602 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 606, one or more communication interfaces 608 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 610, a memory 620, and one or more communication buses 604 for interconnecting these components and various other components.

[0180] In some embodiments, one or more communication buses 604 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices 606 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0181] The memory 620 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 620 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 620 or the non-transitory computer-readable storage medium of the memory 620 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 630 and a CGR content module 640.

[0182] The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR content module 640 is configured to manage and coordinate the presentation of CGR content for one or more users (e.g., a single set of CGR content for one or more users, or multiple sets of CGR content for corresponding groups of one or more users). To this end, in various embodiments, the CGR content module 640 includes a data acquisition unit 642, a tracking unit 644, a coordination unit 646, and a data transmission unit 648.

[0183] In some embodiments, the data acquisition unit 642 is configured to acquire data (e.g., presentation data, interaction data, sensor data, position data, etc.) from at least the HMD 520B. For this purpose, in various embodiments, the data acquisition unit 642 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0184] In some embodiments, the tracking unit 644 is configured to map the scene and track the position / location of at least the HMD 520B relative to the scene. For this purpose, in various embodiments, the tracking unit 644 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0185] In some embodiments, the coordination unit 646 is configured to manage and coordinate the presentation of CGR content to the user by the HMD 520B. For this purpose, in various embodiments, the coordination unit 646 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0186] In some embodiments, the data transmission unit 648 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the HMD 520B. For this purpose, in various embodiments, the data transmission unit 648 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0187] Although the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 are shown as residing on a single device (e.g., controller 510), it should be understood that in other embodiments, any combination of the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 may be located in separate computing devices.

[0188] also, Figure 6 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 6 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0189] Figure 75 is a block diagram of an example of an HMD 520B according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the HMD 520B includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more CGR displays 712, one or more optional internal-facing and / or external-facing image sensors 714, memory 720, and one or more communication buses 704 for interconnecting these and various other components.

[0190] In some embodiments, one or more communication buses 704 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 706 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones 707A, one or more speakers 707B, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0191] In some embodiments, one or more CGR displays 712 are configured to display CGR content to the user. In some embodiments, one or more CGR displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more CGR displays 712 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, HMD 520B includes a single CGR display. In another example, HMD 520B includes a CGR display for each eye of the user.

[0192] In some embodiments, the one or more image sensors 714 are configured to acquire image data corresponding to the user's face including at least a portion of the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 714 are configured to face forward so as to acquire image data corresponding to the scene that the user would see in the absence of the HMD 520B (and may be referred to as a scene camera). The one or more optional image sensors 714 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0193] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some embodiments, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules and data structures or a subset thereof, including an optional operating system 730 and a CGR rendering module 740.

[0194] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR presentation module 740 is configured to present CGR content to the user via one or more CGR displays 712 and / or I / O devices and sensors 706 (such as one or more speakers 707B). For this purpose, in various embodiments, the CGR presentation module 740 includes a data acquisition unit 742, a CGR content presentation unit 744, and a data transmission unit 746.

[0195] In some embodiments, the data acquisition unit 742 is configured to at least Figure 6 The controller 510 of the controller 510 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). In various embodiments, the data acquisition unit acquires CGR files. For this purpose, in various embodiments, the data acquisition unit 742 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0196] In some embodiments, the CGR content presentation unit 744 is configured to present CGR content to a user. In various embodiments, the CGR content presentation unit 744 presents the CGR content of the CGR file according to the rules set forth in the CGR file. To this end, in various embodiments, the CGR content presentation unit 744 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0197] In some embodiments, the data transfer unit 746 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the controller 510. For this purpose, in various embodiments, the data transfer unit 746 includes instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0198] Although the data acquisition unit 742, the CGR content presentation unit 744, and the data transfer unit 746 are shown as residing on a single device (e.g., HMD 520B), it should be understood that in other embodiments, any combination of the data acquisition unit 742, the CGR content presentation unit 744, and the data transfer unit 746 may be located in separate computing devices.

[0199] also, Figure 7 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0200] Generating CGR objects can be a difficult and time-consuming process, requiring, for example, specialized knowledge in 3D object design. This creates a high barrier to the generation of high-quality CGR applications (including such CGR objects). Therefore, in various embodiments, a graphical user interface (GUI) is provided to simplify the generation of three-dimensional CGR objects based on files representing two-dimensional content (such as images, videos, or text).

[0201] User interface and associated processes

[0202] Attention now turns to embodiments of a user interface ("UI") and related processes that may be implemented on an electronic device, such as a portable multifunction device (PMD) 100 having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, or a device 300 having one or more processors, non-volatile memory, a display, and an input device.

[0203] FIG. 8A to FIG. 8W An exemplary user interface for generating a 3D object according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including Fig. 9 Although some of the examples that follow will be given with reference to input on a touch screen display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface 451 that is separate from the display 450, such as Figure 4B as shown in .

[0204] Fig. 8A A CGR environment user interface 801 is shown displayed by a portable multifunction device 100 (hereinafter referred to as "device 100").

[0205] exist Fig. 8A In the embodiment, the CGR environment user interface 801 includes a toolbar area 801A and a display area 801B displaying the CGR environment. The toolbar area 801A includes an object adding affordance 801AA for adding a CGR object to the CGR environment. The display area 801B includes a CGR object that is displayed by the device 100 (e.g., using a scene camera or a device 100 that is connected to the scene camera). Fig. 8A 801B , 801C , 801D , 801E , 801F , 801F , 801B , 801C , 801D , 801D , 801D , 801B , 801C , 801D , 801D , 801B , 801B , 801C , 801D , 801B , 801B , 801B , 801C , 801D , 801F , 801B , 801B , 801B , 801C , 801D ...

[0206] Fig. 8A User input 899A directed to object add affordance 801AA is shown. In various embodiments, user input 899A corresponds to a contact (e.g., a tap) detected at the location of object add affordance 801AA.

[0207] Figure 8B CGR environment user interface 801 is shown in response to detecting user input 899A pointing to an object to add an affordance 801AA. Figure 8B In FIG. 8 , the CGR environment user interface 801 includes an object adding user interface 802 in the form of a pop-up window. Figure 8BIn the figure, the object adding user interface 802 includes a file selecting user interface 803 and a cancel enabling indication 802A, which is used to dismiss (stop displaying) the object adding user interface 802 without adding the CGR object to the CGR environment.

[0208] The file selection user interface 803 includes a plurality of file selection affordances 803A-803G respectively associated with a plurality of files. The plurality of file selection affordances 803A-803G include a first file selection affordance 803A for selecting a first image file named "landscape.jpg", a second file selection affordance 803B for selecting a second image file named "portrait.jpg" (e.g., an image with depth information), a third file selection affordance 803C for selecting a video file named "video.mov", a fourth file selection affordance 803D for selecting a presentation file named "prestation.ppt", a fifth file selection affordance 803E for selecting an electronic book file named "book.ebook", a sixth file selection affordance 802F for selecting a PDF (Portable Document Format) file named "manual.pdf", and a seventh file selection affordance 802G for selecting a three-dimensional (3D) object file named "cube.3d",

[0209] Figure 8B User input 899B is shown directed to first file selection affordance 803 A. In various embodiments, user input 899B corresponds to a contact (e.g., a tap) detected at the location of first file selection affordance 803A.

[0210] Figure 8C The CGR environment user interface 801 is shown in response to detecting a user input 899B directed to a first file selection affordance 803A. Figure 8C , based on the file type (e.g., image file) of the selected file (e.g., "landscape.jpg"), object addition user interface 802 includes image options user interface 804. Image options user interface 804 includes multiple image option selection affordances 804A-804D for selecting options for generating a 3D object based on the selected image file.

[0211] The multiple image option selection enable representations 804A-804D include a canvas enable representation 804A for generating a canvas 3D object based on a selected image file, a cut enable representation 804B for generating a cut 3D object based on a selected image file, a perspective enable representation 804C for generating a perspective 3D object based on the selected image file, and a panoramic enable representation 804D for generating a panoramic 3D object based on the selected image file.

[0212] The image selection user interface 804 includes a user interface for returning to the object adding user interface 802 to display the file selection user interface 803 (e.g., Figure 8B Therefore, in response to detecting the user input pointing to the return enable representation 804E, the image option user interface 804 is replaced with the file selection user interface 803.

[0213] Figure 8C User input 899C is shown directed to canvas-enabled representation 804A. In various embodiments, user input 899C corresponds to a contact (e.g., a tap) detected at the location of canvas-enabled representation 804A.

[0214] Fig.8D The CGR environment user interface 801 is shown in response to detecting user input 899C pointing to the canvas affordance 804A. Fig.8D , the object adding user interface 802 includes a preview user interface 805 .

[0215] Preview user interface 805 includes a return affordance 805A for returning to object add user interface 802 to display image options user interface 804 (or any options user interface that was most recently displayed). Thus, in response to detecting a user input pointing to return affordance 805A, preview user interface 805 is replaced with image options user interface 804 (or any options user interface that was most recently displayed).

[0216] The preview user interface 805 includes a view window 805D that presents a representation of the canvas 3D object 831 based on the selected image file. The selected image file represents an image including a plurality of pixels at respective pixel locations and having respective pixel values.

[0217] The canvas 3D object is defined by a wireframe (or display grid) and one or more texture images. The wireframe includes a front face, one or more edge faces, and a back face. The front face has the shape of the image represented by the selected image file (e.g., a rectangle). The front face is associated with a front texture image corresponding to the image represented by the selected image file. Therefore, the representation of the canvas 3D object 831 includes a front face 831A showing the image represented by the selected image file. The back face is the same shape as the front face, and is spaced apart from the front face in a direction perpendicular to the front face. In various embodiments, the back face is associated with a back texture image corresponding to the image represented by the selected image file. In various embodiments, the back face is associated with a back texture image corresponding to another image (e.g., an image on the back face of the canvas, a standard solid color texture, or another version of the image represented by the selected image file, such as a mirror image or blur).

[0218] The one or more edge faces connect the front side to the back side. The one or more edge faces are associated with one or more edge face texture images, which correspond to the stretching of pixels on the edge of the image represented by the selected image file. Therefore, the representation of the canvas 3D object 831 includes an edge face 831B. Each pixel on the edge of the image represented by the selected image file corresponds to a pixel on the edge of the front texture image and a pixel line of the edge face texture image connecting the front side to the back side. For example, the first edge pixel 831C of the front side 831A has a first pixel value (e.g., dark gray) corresponding to a group of pixels 831D having the same pixel value of the edge face 831B.

[0219] The preview user interface 805 includes multiple view enable representations 805B-805C, including an object view enable representation 805B and a CGR view enable representation 805C, wherein the object view enable representation is used to view the canvas 3D object 831 in the view window 805D in a scene mode (wherein user input pointing to the view window 805D changes the perspective of the view of the canvas 3D object 831), and the CGR view enable representation is used to view the canvas 3D object 831 in the view window 805D in a CGR mode (wherein movement of the device 100 changes the perspective of the view of the canvas 3D object 831).

[0220] The object adding user interface 802 includes determining an enabling representation 802B for adding a canvas 3D object to the CGR environment and dismissing (stopping displaying) the object adding user interface 802.

[0221] Fig.8D User input 899D is shown pointing to determine enable representation 802B. In various embodiments, user input 899D corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 802B.

[0222] Fig. 8E CGR environment user interface 801 is shown in response to detecting user input 899D pointing to determine affordance 802B. Fig. 8E , display area 801B includes a representation 841 of a canvas 3D object displayed against a representation 801BB of a back wall. In various implementations, device 100 detects a vertical plane corresponding to the back wall and displays the representation 841 of the canvas 3D object displayed against the representation 801BB of the back wall in response to detecting the vertical plane.

[0223] Fig. 8E User input 899E is shown directed to the representation of the canvas 3D object 841. In various embodiments, the user input 899E corresponds to a detected moving contact (eg, drag or touch and drag) with a starting location at the location of the representation 841 of the canvas 3D object.

[0224] Fig.8F A CGR environment user interface 801 is shown in response to detecting a user input 899E pointing to a representation 841 of a canvas 3D object. Fig.8F In the embodiment of the present invention, the representation 841 of the canvas 3D object is moved to be displayed against the representation 801BA of the side wall instead of the representation 801BB of the back wall. In various implementations, the device 100 detects a vertical plane corresponding to the side wall and displays the representation 841 of the canvas 3D object displayed against the representation 801BB of the side wall in response to detecting the vertical plane and the user input 899E.

[0225] Fig.8F User input 899F pointing to object add affordance 801AA is shown. In various embodiments, user input 899F corresponds to a contact (e.g., a tap) detected at the location of object add affordance 801AA.

[0226] Figure 8G CGR environment user interface 801 is shown in response to detecting user input 899F pointing to an object to add an affordance 801AA. Figure 8G In FIG. 8 , the CGR environment user interface 801 includes an object adding user interface 802 in the form of a pop-up window. Figure 8G In the example, object adding user interface 802 includes file selection user interface 803 and cancel affordance 802A. File selection user interface 803 includes the plurality of file selection affordances 803A-803G.

[0227] Figure 8GUser input 899G directed to second file selection affordance 803B is shown. In various embodiments, user input 899G corresponds to a contact (e.g., a tap) detected at the location of second file selection affordance 803B.

[0228] Figure 8H The CGR environment user interface 801 is shown in response to detecting user input 899G directed to the second file selection affordance 803B. Figure 8H In the example, based on the file type (eg, image file) of the selected file (eg, "portrait.jpg"), the object adding user interface 802 includes an image options user interface 804 including the plurality of image options selection enable representations 804A-804D.

[0229] Figure 8H User input 899H is shown directed to cut-enabled representation 804B. In various embodiments, user input 899H corresponds to a contact (e.g., a tap) detected at the location of cut-enabled representation 804B.

[0230] Figure 8I CGR environment user interface 801 is shown in response to detecting user input 899H pointing to cut enable representation 804A. Figure 8I , the object adding user interface 802 includes a preview user interface 805 .

[0231] The preview user interface 805 includes a view window 805D that presents a representation of the cut 3D object 832 based on the selected image file. The selected image file represents an image including a plurality of pixels at corresponding pixel positions and having corresponding pixel values. The image includes a foreground and a background. In various embodiments, the device 100 performs semantic segmentation to determine the foreground and the background. In various embodiments, the image file includes information (e.g., a depth map) that defines the foreground and the background.

[0232] The cut 3D object is defined by a wireframe (or display grid) and one or more texture images. The wireframe includes a front side, one or more edge faces, and a back side. The front side has the shape of the foreground of the image represented by the selected image file (e.g., longitudinally). The front side is associated with a front texture image corresponding to the foreground of the image represented by the selected image file. Therefore, the representation of the cut 3D object 832 includes a front side 832A that displays the foreground of the image represented by the selected image file. The back side is the same shape as the front side, and is spaced apart from the front side in a direction perpendicular to the front side. In various embodiments, the back side is associated with a back texture image corresponding to the foreground of the image represented by the selected image file. In various embodiments, the back side is associated with a back texture image corresponding to another image (e.g., a cardboard image).

[0233] The one or more edge faces connect the front side to the back side. The one or more edge faces are associated with one or more edge face texture images that correspond to stretches of pixels on the edge of the foreground of the image represented by the selected image file. Thus, the representation of the cut 3D object 832 includes edge faces 832B. Each pixel on the edge of the image represented by the selected image file corresponds to a pixel on the edge of the front side texture image and a line of pixels of the edge face texture image that connects the front side to the back side.

[0234] Figure 8I User input 8991 is shown pointing to determine enable representation 802B. In various embodiments, user input 8991 corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 802B.

[0235] Figure 8J The CGR environment user interface 801 is shown in response to detecting user input 899I pointing to determine the enable indication 802B. Figure 8J , display area 801B includes a representation 842 of a cutout 3D object displayed on a representation 801BC of a floor.

[0236] Figure 8J User input 899J directed to object add affordance 801AA is shown. In various embodiments, user input 899J corresponds to a contact (e.g., a tap) detected at the location of object add affordance 801AA.

[0237] Figure 8K CGR environment user interface 801 is shown in response to detecting user input 899J pointing to an object to add an affordance 801AA. Figure 8K In FIG. 8 , the CGR environment user interface 801 includes an object adding user interface 802 in the form of a pop-up window. Figure 8K In the example, object adding user interface 802 includes file selection user interface 803 and cancel affordance 802A. File selection user interface 803 includes the plurality of file selection affordances 803A-803G.

[0238] Figure 8K User input 899K is shown directed to third file selection affordance 803C. In various embodiments, user input 899K corresponds to a contact (e.g., a tap) detected at the location of third file selection affordance 803C.

[0239] Figure 8L The CGR environment user interface 801 is shown in response to detecting user input 899K directed to a third file selection affordance 803C. Figure 8L , based on the file type (e.g., video file) of the selected file (e.g., "video.mov"), the object addition user interface 802 includes a video options user interface 806. The video options user interface 806 includes a plurality of graphical option selection affordances 806A-806D for selecting options for generating a 3D object based on the selected video file.

[0240] The multiple video option selection enable representations 806A-806C include a flat-screen TV enable representation 806A for generating a flat-screen TV 3D object based on a selected video file, a computer monitor enable representation 806B for generating a computer monitor 3D object based on a selected video file, and a 60s CRT enable representation 806C for generating a 60s CRT 3D object based on a selected video file.

[0241] The video options user interface 806 includes a user interface for returning to the object adding user interface 802 to display the file selection user interface 803 (e.g., Figure 8K Therefore, in response to detecting a user input pointing to return enable representation 806D, video options user interface 806 is replaced with file selection user interface 803.

[0242] Figure 8L User input 899L is shown directed to flat-panel television affordance 806A. In various embodiments, user input 899L corresponds to a contact (e.g., a tap) detected at the location of flat-panel television affordance 806A.

[0243] Figure 8M The CGR environment user interface 801 is shown in response to detecting a user input 899L directed toward the flat-panel television affordance 806A. Figure 8M , the object adding user interface 802 includes a preview user interface 805 .

[0244] The preview user interface 805 includes a view window 805D presenting a representation 833 of the flat-panel television 3D object based on a selected video file. The selected video file represents a video including a sequence of images.

[0245] The flat-panel TV 3D object is defined by a wireframe (or display grid) and one or more texture images. The wireframe includes a front face, one or more edge faces, and a back face. The front face has a generally rectangular shape. The front face is associated with a front face texture image corresponding to the front face of the flat-panel TV having a display area including images in an image sequence of a video. Therefore, the representation 833 of the flat-panel TV 3D object includes a front face, which includes a display area 833A that displays images in an image sequence of a video. At different times, different images of the image sequence are displayed in sequence. Therefore, the display area 833A displays the video represented by the video file. The back face and the one or more edge faces are shaped so that the wireframe has a generally flat-panel TV shape. The back face and the one or more edge faces are associated with corresponding texture images corresponding to the back face and one or more edges of the flat-panel TV.

[0246] Figure 8M User input 899M is shown pointing to determine enable representation 802B. In various embodiments, user input 899M corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 802B.

[0247] Figure 8N CGR environment user interface 801 is shown in response to detecting user input 899M pointing to determine enable indication 802B. Figure 8N , display area 801B includes a representation 843 of a flat-screen television 3D object displayed against representation 801BB of a back wall.

[0248] Figure 8N A user input 899N is shown pointing to the representation 843 of the flat-panel TV 3D object. In various embodiments, the user input 899N corresponds to two contacts (e.g., a pinch or spread gesture) detected at the location of the representation 843 of the flat-panel TV CGR object that move toward or away from each other.

[0249] Fig.8O The CGR environment user interface 801 is shown in response to detecting a user input 899N directed toward the representation 843 of the flat-panel television 3D object. Fig.8O , the representation 843 of the flat screen TV 3D object is resized to be larger via user input 899N.

[0250] Fig.8O User input 899O is shown pointing to object add affordance 801AA. In various embodiments, user input 899O corresponds to a contact (e.g., a tap) detected at the location of object add affordance 801AA.

[0251] Figure 8PCGR environment user interface 801 is shown in response to detecting user input 899O pointing to an object to add an affordance 801AA. Figure 8P In FIG. 8 , the CGR environment user interface 801 includes an object adding user interface 802 in the form of a pop-up window. Figure 8P In the example, object adding user interface 802 includes file selection user interface 803 and cancel affordance 802A. File selection user interface 803 includes the plurality of file selection affordances 803A-803G.

[0252] Figure 8P User input 899P directed to sixth file selection affordance 803F is shown. In various embodiments, user input 899P corresponds to a contact (e.g., a tap) detected at the location of sixth file selection affordance 803F.

[0253] Figure 8Q The CGR environment user interface 801 is shown in response to detecting user input 899P directed to the sixth file selection affordance 803F. Figure 8Q In the example of the embodiment of the present invention, based on the file type (e.g., text file) of the selected file (e.g., "manual.pdf"), the object addition user interface 802 includes a text option user interface 806, which includes a plurality of text option selection affordances 807A-807C for selecting options for generating a 3D object based on the selected text file. In various embodiments, the text file may include a PDF file, a word processing file, or an e-book file. Therefore, in various embodiments, the text file may include more than just text, such as images, charts, videos, or other content. In various embodiments, the text file includes multiple pages (and therefore may be referred to as a page file).

[0254] The multiple text option selection enable representations 807A-807C include a book enable representation 807A for generating a book 3D object based on a selected text file, a paper stack enable representation 807B for generating a paper stack 3D object based on the selected text file, and a presentation screen enable representation 807C for generating a presentation screen 3D object based on the selected text file.

[0255] The text selection user interface 807 includes a user interface for returning to the object adding user interface 802 to display the file selection user interface 803 (e.g., Figure 8P Therefore, in response to detecting the user input pointing to the return enable representation 807D, the text option user interface 807 is replaced with the file selection user interface 803.

[0256] Figure 8QUser input 899Q is shown directed to book affordance 807A. In various embodiments, user input 899Q corresponds to a contact (e.g., a tap) detected at the location of book affordance 807A.

[0257] Figure 8R CGR environment user interface 801 is shown in response to detecting user input 899Q pointing to book affordance 807A. Figure 8R , the object adding user interface 802 includes a preview user interface 805 .

[0258] The preview user interface 805 includes a view window 805D that presents a representation 834 of a book 3D object based on a selected text file. The selected book file represents a plurality of pages, each page including text and / or other content.

[0259] The book 3D object is defined by a wireframe (or display grid) and one or more texture images. The wireframe generally includes the shape of an open book, and includes a front, a spine, a back, a left page, and a right page. The left page has a generally rectangular shape. The left page is associated with a texture image corresponding to the first page of the text file. Therefore, the representation 834 of the book 3D object includes a left page 834A showing the first page of the text file. The right page has a generally rectangular shape. The right page is associated with a texture image corresponding to the second page of the text file. Therefore, the representation 834 of the book 3D object includes a right page 834B showing the second page of the text file. In various embodiments, in response to a user input pointing to the book 3D object in a CGR environment, the left page is associated with a texture image corresponding to the third page of the text file, and the right page is associated with a texture image corresponding to the fourth page of the text file. Similarly, in response to a user input pointing to the book 3D object in a CGR environment, the left page is associated with a texture image corresponding to the fifth page of the text file, and the right page is associated with a texture image corresponding to the sixth page of the text file.

[0260] Figure 8R User input 899R is shown pointing to the representation 834 of the book 3D object. In various embodiments, the user input 899R corresponds to two contacts detected at the location of the representation 834 of the book 3D object that move about a common center (eg, a rotation gesture).

[0261] Figure 8S The CGR environment user interface 801 is shown in response to detecting a user input 899R pointing to a representation 834 of a book 3D object. Figure 8S , view window 805D includes a representation 834 of the book 3D object displayed from another perspective (eg, rotated about an axis).

[0262] As described above, the book 3D object is defined by a wireframe (or display mesh) and one or more texture images. The wireframe generally includes the shape of an open book and includes a front side, a spine side, a back side, a left page side, and a right page side. The front side has a generally rectangular shape and is associated with a texture image that includes the title of a text file (which may be the name of the text file or derived from the metadata of the file). Thus, the representation 834 of the book 3D object includes a front side 834C that displays the title of the text file.

[0263] Figure 8S User input 899S is shown pointing to determine enable representation 802B. In various embodiments, user input 899S corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 802B.

[0264] Figure 8T The CGR environment user interface 801 is shown in response to detecting user input 899S pointing to determine the enable indication 802B. Figure 8T , display area 801B includes a representation 844 of a book 3D object displayed on a representation 801BD of a table.

[0265] Figure 8T User input 899T directed to object addition affordance 801AA is shown. In various embodiments, user input 899T corresponds to a contact (e.g., a tap) detected at the location of object addition affordance 801AA.

[0266] Figure 8U CGR environment user interface 801 is shown in response to detecting user input 899T pointing to an object to add an affordance 801AA. Figure 8U In FIG. 8 , the CGR environment user interface 801 includes an object adding user interface 802 in the form of a pop-up window. Figure 8U In the example, object adding user interface 802 includes file selection user interface 803 and cancel affordance 802A. File selection user interface 803 includes the plurality of file selection affordances 803A-803G.

[0267] Figure 8U User input 899U directed to seventh file selection affordance 803G is shown. In various embodiments, user input 899U corresponds to a contact (e.g., a tap) detected at the location of seventh file selection affordance 803G.

[0268] Figure 8V The CGR environment user interface 801 is shown in response to detecting user input 899U directed to the seventh file selection affordance 803G. Figure 8VIn the example, based on the file type (e.g., 3D object file) of the selected file (e.g., "cube.3D"), the object adding user interface 802 includes a preview user interface 805. The 3D object file represents a cube 3D object. The preview user interface 805 includes a view window 805D that presents a representation 835 of the cube 3D object.

[0269] Figure 8V User input 899V is shown pointing to determine enable representation 802B. In various embodiments, user input 899V corresponds to a contact (e.g., a tap) detected at the location of determine enable representation 802B.

[0270] Figure 8W The CGR environment user interface 801 is shown in response to detecting a user input 899V pointing to the indication 802B. Figure 8W , display area 801B includes a representation 845 of a cubic 3D object displayed on a representation 801BD of a table.

[0271] Fig. 9 900 is a flowchart representation of a method 900 for generating a 3D object according to some embodiments. In various embodiments, the method 900 is performed by a device (e.g., Figure 1A The portable multifunction device 100 or Figure 7 In some embodiments, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 900 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0272] Method 900 begins at block 910, where the device receives user input via one or more input devices selecting a file representing two-dimensional (2D) content and having a file type. Figure 8B In , device 100 receives user input 899B pointing to first file selection enable representation 803A, and the user input selects a first image file representing a 2D image and having an image file type. Figure 8G In , device 100 receives user input 899G pointing to second file selection enable representation 803B, and the user input selects a second image file representing a 2D image and having an image file type. Figure 8K In , device 100 receives user input 899K directed to third file selection enable representation 803C, and the user input selects a video file representing a 2D video and having a video file type. Figure 8P, device 100 receives user input 899P directed to a sixth file selection indication 803F, the user input selecting a PDF file representing 2D page content (including text) and having a page file type.

[0273] Method 900 continues at block 920, where the device receives user input via one or more input devices requesting generation of a three-dimensional (3D) object based on the file. Figure 8C , device 100 receives user input 899C pointing to canvas affordance 804A, and in response, Fig.8D The generated canvas 3D object representation 831 is displayed in the view window 805D in FIG. Fig.8D , device 100 receives user input 899D pointing to determine enable indication 802B, and in response, Fig. 8E The generated canvas 3D object representation 841 is displayed in the view window 801B in .

[0274] Method 900 continues at block 930, where the device generates a 3D object representing the 2D content based on the file type. In various embodiments, the file type of the file is determined based on a file extension of the file. In various embodiments, the file type of the file is determined based on the 2D content represented by the file. In various embodiments, generating the 3D object representing the 2D content based on the file type includes determining that the file type is an image file type, where the 2D content includes an image, and in response to determining that the file type is an image file type, generating a canvas 3D object that includes a front face associated with a texture image representing the image. For example, in Fig.8D In response to determining that the file type of the first image file is an image file type, wherein the 2D content includes a first image, the device 100 generates and displays a representation 831 of a canvas 3D object, the canvas 3D object including a front face 831A associated with a texture image representing the first image. In various embodiments, the canvas 3D object also includes an edge face that is perpendicular to the front face and is associated with a stretched texture image corresponding to pixels on an edge of the image. For example, in Fig.8D , the representation 831 of the canvas 3D object includes stretched edge faces 831B corresponding to pixels on the edge of the first image.

[0275] In various implementations, the file of the image file type is an image (eg, a JPG file or a GIF file) or an image with depth information.

[0276] In various embodiments, generating a 3D object representing 2D content based on the file type includes determining that the file type is an image file type, wherein the 2D content includes an image, and in response to determining that the file type is the image file type, generating a cutout 3D object including a front face having a shape of a foreground of the image and being associated with a texture image representing the foreground of the image. Figure 8I In response to determining that the file type of the second image file is an image file type, wherein the 2D content includes the second image, device 100 generates and displays a representation 832 of a cut 3D object, the cut 3D object including a front face 832A having a shape of a foreground of the second image and being associated with a texture image representing the foreground of the second image.

[0277] In various embodiments, generating a 3D object representing 2D content based on the file type includes determining that the file type is a video file type, wherein the 2D content includes a video, and in response to determining that the file type is a video file type, generating a television 3D object including a front face associated with a texture image including a display area representing the video. Figure 8M In response to determining that the file type of the video file is a video file type, wherein the 2D content includes a video, the device 100 generates and displays a representation 833 of a flat-panel television 3D object including a front face associated with a texture image including a display area 833A representing the video.

[0278] In various specific implementations, the file of the video file type is a MOV file, an MPG file, or an AVI file.

[0279] In various embodiments, generating a 3D object representing 2D content based on a file type includes determining that the file type is a page file type, wherein the 2D content includes one or more content pages, and in response to determining that the file type is a page file type, generating a book 3D object including a first page face associated with a texture image representing a first page of the one or more content pages. Figure 8R In response to determining that the file type of the PDF file is a page file type, wherein the 2D content includes one or more content pages, the device 100 generates and displays a representation 834 of a book 3D object, the book 3D object including a left page surface 834A associated with a texture image of a first page representing the one or more content pages. In various embodiments, in response to user input pointing to the book 3D object in the CGR environment, the first page surface is associated with a texture image of a second page representing the one or more content pages. In various embodiments, the book 3D object includes a second page surface associated with a texture image representing the second page of the one or more content pages. For example, in Figure 8R, the representation of the book 3D object 834 includes a right page face 834B associated with a texture image representing a second page of the one or more content pages. In various embodiments, the book 3D object includes a front face associated with a texture image representing metadata of the file. For example, in Figure 8S , the representation of the book 3D object 834 includes a front face 834C associated with a texture image representing the title of the document.

[0280] In various specific implementations, the file of the page file type (or text file type) is a word processing document (e.g., a DOC file or a TXT file), an electronic book (e.g., an EPUB file or an IBOOKS file), or a slide presentation (e.g., a PPT file or a KEY file).

[0281] In various embodiments, generating a 3D object representing the 2D content based on the file type includes presenting a set of options based on the file type, receiving user input selecting one of the set of options via one or more input devices, and generating the 3D object representing the 2D content based on the selected one of the set of options. Figure 8C , device 100 presents an image option user interface 804 including the plurality of image option selection indicators 804A-804D.

[0282] In various embodiments, method 900 also includes storing the 3D object as a 3D object file.

[0283] In various embodiments, method 900 also includes displaying a 3D object representing the 2D content. Fig.8D In FIG. 8 , the device 100 displays a representation 831 of the canvas 3D object in the view window 805D. Fig. 8E , device 100 displays a representation 841 of the canvas 3D object in display area 801B.

[0284] In various specific implementations, displaying the 3D object representing the 2D content includes performing plane detection and / or object detection, and displaying the 3D object anchored to a specific type of plane and / or a specific type of object based on the file type. For example, a canvas 3D object is displayed on a vertical plane and / or a wall, a clip 3D object is displayed on a horizontal plane and / or a floor, a television 3D object is displayed on a vertical plane and / or a wall, and a book 3D object is displayed on a horizontal plane and / or a table.

[0285] In various embodiments, method 900 also includes receiving user input that spatially manipulates the 3D object and changes the display of the 3D object. Fig. 8E , device 100 detects user input 899E pointing to representation 841 of the canvas 3D object, and in response, Fig. 8E A representation 841 of the canvas 3D object is displayed at another location.

[0286] It should be understood that Fig. 9 The specific order in which the operations are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0287] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A , Figure 3 and Figure 5 described) or application-specific chips. In addition, the above reference Fig. 9 The described operations are optionally performed by Figure 1A to Figure 1B . For example, user input and user interface elements are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on touch-sensitive display system 112, and event distributor module 174 delivers event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally utilizes or calls data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A-1B How the components depicted in the figure can implement other processes.

[0288] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that other technical personnel in the field can best use the present invention and the various described embodiments with various modifications suitable for the specific purpose contemplated.

[0289] A first embodiment is a method comprising: receiving user input, via one or more input devices, selecting a file representing two-dimensional (2D) content and having a file type; receiving user input, via the one or more input devices, requesting generation of a three-dimensional (3D) object based on the file; and generating a 3D object representing the 2D content based on the file type.

[0290] The second embodiment is substantially similar to the first embodiment, wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is an image file type, wherein the 2D content comprises an image; and in response to determining that the file type is an image file type, generating a canvas 3D object, wherein the canvas 3D object comprises a front face associated with a texture image representing the image.

[0291] A third embodiment is substantially similar to the second embodiment, wherein the canvas 3D object further comprises an edge face, the edge face being perpendicular to the front face and being associated with a stretched texture image corresponding to pixels on an edge of the image.

[0292] The fourth embodiment is substantially similar to the first embodiment, wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is an image file type, wherein the 2D content comprises an image; and in response to determining that the file type is an image file type, generating a cutout 3D object, wherein the cutout 3D object comprises a front face having a shape of a foreground of the image and being associated with a texture image representing the foreground of the image.

[0293] The fifth embodiment is substantially similar to the first embodiment, wherein generating the 3D object representing the 2D content based on the file type includes: determining that the file type is a video file type, wherein the 2D content includes a video; and in response to determining that the file type is a video file type, generating a television 3D object, wherein the television 3D object includes a front face associated with a texture image including a display area representing the video.

[0294] The sixth embodiment is substantially similar to the first embodiment, wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is a page file type, wherein the 2D content comprises one or more content pages; and in response to determining that the file type is a page file type, generating a book 3D object, wherein the book 3D object comprises a first page surface associated with a texture image of a first page representing the one or more content pages.

[0295] The seventh embodiment is substantially similar to the sixth embodiment, wherein in response to a user input pointing to the book 3D object in the CGR environment, the first page face is associated with a texture image representing a second page of the one or more content pages.

[0296] The eighth embodiment is substantially similar to the sixth embodiment, wherein the book 3D object includes a second page face associated with a texture image representing a second page of the one or more content pages.

[0297] The ninth embodiment is substantially similar to any one of the sixth to eighth embodiments, wherein the book 3D object comprises a front face associated with a texture image representing metadata of the file.

[0298] The tenth embodiment is substantially similar to any one of the first to ninth embodiments, wherein generating the 3D object representing the 2D content based on the file type comprises: presenting a set of options based on the file type; receiving user input selecting one of the set of options via the one or more input devices; and generating the 3D object representing the 2D content based on the selected one of the set of options.

[0299] An eleventh embodiment is substantially similar to any one of the first to tenth embodiments, wherein the method further comprises displaying the 3D object representing the 2D content.

[0300] A twelfth embodiment is substantially similar to the eleventh embodiment, wherein the method further comprises receiving a user input that spatially manipulates the 3D object and changes the display of the 3D object.

[0301] The thirteenth embodiment is substantially similar to any one of the first to twelfth embodiments, wherein the method further comprises storing the 3D object as a 3D object file.

Claims

1. A method for generating a 3D object, comprising: receiving, via one or more input devices, a first user input selecting a file representing two-dimensional (2D) content and having a file type, wherein the first user input is directed to a first file selection affordance within a file selection user interface; In response to receiving the first user input, determining an option user interface based on the file type, wherein the image option user interface includes: In response to the file type of the 2D content being a first file type, selecting affordances for a first set of 3D objects corresponding to the first file type; In response to the file type of the 2D content being a second file type different from the first file type, a second set of 3D object selection-enabling representations corresponding to the second file type, wherein the second set of 3D object selection-enabling representations is different from the first set of 3D object selection-enabling representations; receiving, via the one or more input devices, a second user input requesting generation of a 3D object based on the file while displaying the options user interface including the one or more 3D object selection-enabling representations, wherein the second user input selects a specific 3D object selection-enabling representation from the first group of 3D object selection-enabling representations or the second group of 3D object selection-enabling representations from the options user interface, and wherein the specific 3D object selection-enabling representation is associated with a corresponding 3D object option; and A 3D object representing the 2D content is generated by displaying the 2D content on at least a portion of the corresponding 3D object option based on the file type.

2. The method of claim 1 , wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is an image file type, wherein the 2D content includes an image; and In response to determining that the file type is an image file type, a canvas 3D object is generated, the canvas 3D object including a front face associated with a texture image representing the image. 3 . The method of claim 2 , wherein the canvas 3D object further comprises an edge face, the edge face being perpendicular to the front face and being associated with a stretched texture image corresponding to pixels on an edge of the image.

4. The method of claim 1 , wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is an image file type, wherein the 2D content includes an image; and In response to determining that the file type is an image file type, a cutout 3D object is generated, the cutout 3D object including a front face having a shape of a foreground of the image and being associated with a texture image representing the foreground of the image.

5. The method of claim 1 , wherein generating the 3D object representing the 2D content based on the file type comprises: determining that the file type is a video file type, wherein the 2D content includes a video; and In response to determining that the file type is a video file type, a television 3D object is generated that includes a front face associated with a texture image that includes a display area representing the video.

6. The method of claim 1 , wherein generating the 3D object representing the 2D content based on the file type comprises: Determining that the file type is a page file type, wherein the 2D content includes one or more content pages; as well as In response to determining that the file type is a page file type, a book 3D object is generated, the book 3D object including a first page face associated with a texture image representing a first page of the one or more content pages. 7 . The method of claim 6 , wherein in response to a user input pointing to the book 3D object in a CGR environment, the first page face is associated with a texture image representing a second page of the one or more content pages. 8 . The method of claim 6 , wherein the book 3D object comprises a second page face associated with a texture image representing a second page of the one or more content pages.

9. The method of claim 6, wherein the book 3D object comprises a front face associated with a texture image representing metadata of the file.

10. The method of claim 1, further comprising displaying the 3D object representing the 2D content. 11 . The method of claim 10 , further comprising receiving user input that spatially manipulates the 3D object and changes a display of the 3D object. 12 . The method of claim 11 , further comprising storing the 3D object as a 3D object file.

13. The method of claim 1, wherein the options user interface comprises: In response to the file type of the 2D content being an image file type, a canvas affordance or a clip affordance; In response to the file type of the 2D content being a video file type, a flat-panel television affordance or a computer monitor affordance; as well as In response to the file type of the 2D content being a text file type, a book affordance, a paper stack affordance, or a presentation screen affordance.

14. The method of claim 1, wherein the 3D object is partially defined by a wireframe corresponding to the first file type.

15. The method of claim 1, wherein the 3D object is defined in part by a display mesh corresponding to the first file type.

16. The method of claim 1, wherein the first file type comprises one of an image file type, a video file type, and a page file type, and wherein the second file type comprises a different one of the image file type, the video file type, and the page file type.

17. An electronic device comprising: one or more input devices; non-transitory memory; and One or more processors configured to: receiving, via one or more input devices, a first user input selecting a file representing two-dimensional (2D) content and having a file type, wherein the first user input is directed to a first file selection affordance within a file selection user interface; In response to receiving the first user input, determining an option user interface based on the file type, wherein the image option user interface includes: In response to the file type of the 2D content being a first file type, selecting affordances for a first set of 3D objects corresponding to the first file type; In response to the file type of the 2D content being a second file type different from the first file type, a second set of 3D object selection-enabling representations corresponding to the second file type, wherein the second set of 3D object selection-enabling representations is different from the first set of 3D object selection-enabling representations; receiving, via the one or more input devices, a second user input requesting generation of a 3D object based on the file while displaying the options user interface including the one or more 3D object selection-enabling representations, wherein the second user input selects a specific 3D object selection-enabling representation from the first group of 3D object selection-enabling representations or the second group of 3D object selection-enabling representations from the options user interface, and wherein the specific 3D object selection-enabling representation is associated with a corresponding 3D object option; and A 3D object representing the 2D content is generated by displaying the 2D content on at least a portion of the corresponding 3D object option based on the file type.

18. The electronic device of claim 17, wherein the one or more processors are configured to generate the 3D object representing the 2D content based on the file type by: determining that the file type is an image file type, wherein the 2D content includes an image; and In response to determining that the file type is an image file type, a canvas 3D object is generated, the canvas 3D object including a front face associated with a texture image representing the image.

19. The electronic device of claim 18, wherein the canvas 3D object further comprises an edge face that is perpendicular to the front face and is associated with a stretched texture image corresponding to pixels on an edge of the image.

20. The electronic device of claim 17, wherein the one or more processors are configured to generate the 3D object representing the 2D content based on the file type by: determining that the file type is an image file type, wherein the 2D content includes an image; and In response to determining that the file type is an image file type, a cutout 3D object is generated, the cutout 3D object including a front face having a shape of a foreground of the image and being associated with a texture image representing the foreground of the image.

21. The electronic device of claim 17, wherein the options user interface comprises: In response to the file type of the 2D content being an image file type, a canvas affordance or a clip affordance; In response to the file type of the 2D content being a video file type, a flat-panel television affordance or a computer monitor affordance; as well as In response to the file type of the 2D content being a text file type, a book affordance, a paper stack affordance, or a presentation screen affordance.

22. The electronic device of claim 17, wherein the 3D object is partially defined by a wireframe corresponding to the first file type.

23. The electronic device of claim 17, wherein the 3D object is defined in part by a display grid corresponding to the first file type.

24. A non-transitory computer-readable medium having encoded thereon instructions that, when executed by an electronic device comprising one or more input devices and one or more processors, cause the electronic device to: receiving, via the one or more input devices, a first user input selecting a file representing two-dimensional (2D) content and having a file type, wherein the first user input is directed to a first file selection affordance within a file selection user interface; In response to receiving the first user input, determining an option user interface based on the file type, wherein the image option user interface includes: In response to the file type of the 2D content being a first file type, selecting affordances for a first set of 3D objects corresponding to the first file type; In response to the file type of the 2D content being a second file type different from the first file type, a second set of 3D object selection-enabling representations corresponding to the second file type, wherein the second set of 3D object selection-enabling representations is different from the first set of 3D object selection-enabling representations; receiving, via the one or more input devices, a second user input requesting generation of a 3D object based on the file while displaying the options user interface including the one or more 3D object selection-enabling representations, wherein the second user input selects a specific 3D object selection-enabling representation from the first group of 3D object selection-enabling representations or the second group of 3D object selection-enabling representations from the options user interface, and wherein the specific 3D object selection-enabling representation is associated with a corresponding 3D object option; and A 3D object representing the 2D content is generated by displaying the 2D content on at least a portion of the corresponding 3D object option based on the file type.

25. The non-transitory computer-readable medium of claim 24, wherein the instructions, when executed by the electronic device, cause the electronic device to generate the 3D object representing the 2D content based on the file type in the following manner: determining that the file type is an image file type, wherein the 2D content includes an image; and In response to determining that the file type is an image file type, a canvas 3D object is generated, the canvas 3D object including a front face associated with a texture image representing the image.

Images