METHOD AND DEVICE FOR DETECTING PLANES AND / OR QUADTREES FOR USE AS A VIRTUAL SUBSTRATE

Abstract

Procedure, comprehensive: on an electronic device comprising one or more processors, non-volatile memory, an image sensor, a display and one or more input devices: Displays, on the display, of a reticle element in a first appearance state superimposed in a media capture preview of objects in a field of view of the image sensor, wherein the media capture preview changes as the objects in the field of view of the image sensor change; Capturing a layer in the media capture preview; and in response to a capture of the plane, display, on the display, the reticle element is superimposed in a second appearance state in the media capture preview, wherein the reticle element corresponds to a specification of a section of the extent of the plane while it is displayed in the second appearance state.

Description

TECHNICAL AREA

[0001] This generally concerns the detection of planes and / or quadtrees within a scene, including but not limited to electronic devices that allow the detected planes and / or quadtrees to be displayed as virtual substrates. BACKGROUND

[0002] Positioning augmented reality / virtual reality (AR / VR) objects within a static or dynamic scene presents a challenge from at least one user experience perspective. If an AR / VR object is placed within a scene without a suitable virtual substrate, it cannot be anchored to a real-world surface. As a result, the AR / VR object may appear to float in mid-air, obscure a real-world object, or collide with one. This leads to a poor user experience that is neither lifelike nor believable.

[0003] In the embodiments described below, this challenge is solved by capturing planes or quadtrees within the scene and determining their extents to provide virtual substrates on which AR / VR objects can be placed.

[0004] The state-of-the-art document US 9,279,983 B1 describes methods and systems for cropping a digital image based on motion data.

[0005] The state-of-the-art document US 2012 / 0105654A1 describes methods and systems for processing video for stabilization and realignment.

[0006] The prior art document US 6,654,506 B1 describes a method and system for automatically creating cropped and zoomed versions of photographic images. SUMMARY

[0007] According to some embodiments, a method is performed on an electronic device comprising one or more processors, non-volatile memory, an image sensor, a display, and one or more input devices. The method includes displaying a reticle element in a first appearance state superimposed on a media capture preview of objects in a field of view of the image sensor, the media capture preview changing as the objects in the field of view of the image sensor change. The method further includes: capturing a plane in the media capture preview; and, in response to the capture of the plane, displaying on the display the reticle element in a second appearance state superimposed on the media capture preview, the reticle element corresponding to an indication of a portion of the extent of the plane while displayed in the second appearance state.

[0008] According to some embodiments, a method is performed on an electronic device with one or more processors and non-volatile memory. The method includes: obtaining a plurality of sets of images; synthesizing a three-dimensional point cloud for each of the plurality of sets of images; creating planes using the respective three-dimensional point clouds; and generating a composite set of quadtrees that designate a composite set of planes over the three-dimensional point clouds.

[0009] According to some embodiments, an electronic device includes a display, one or more input devices, one or more processors, non-volatile memory, and one or more programs; wherein the one or more programs are stored in the non-volatile memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of the operations of each of the methods described herein. According to some embodiments, instructions are stored in a non-volatile, computer-readable storage medium which, when executed by one or more processors of an electronic device with a display and one or more input devices, cause the device to perform or cause the performance of each of the methods described herein.According to some embodiments, an electronic device includes: a display, one or more input devices; and means for carrying out or causing the execution of each of the methods described herein. According to some embodiments, an information processing device for use in an electronic device with a display and one or more input devices includes means for carrying out or causing the execution of each of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the following description of embodiments in conjunction with the following drawings, in which the same reference numerals refer to corresponding parts throughout the figures. Fig. Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments. Fig. 1B is a block diagram illustrating exemplary components for event handling according to some embodiments. Fig. Figure 2 illustrates a portable multi-function device with a touchscreen according to some embodiments. Fig. Figure 3 is a block diagram of an exemplary multi-function device with a display and a touch-sensitive surface according to some embodiments. Fig. Figures 4A-4M illustrate exemplary user interfaces for capturing a virtual substrate and arranging objects on it according to some embodiments. Fig. Figures 5A-5B illustrate exemplary abstract block diagrams for generating a set of quadtrees according to some embodiments. Fig. Figures 6A-6G illustrate exemplary user interfaces for capturing a virtual substrate according to some embodiments. Fig. Figure 7 illustrates a flowchart of a method for capturing a virtual substrate and arranging objects on it according to some embodiments. Fig. Figure 8 illustrates a flowchart of a method for generating a combined set of quadtrees for use as a virtual substrate according to some embodiments. Fig. Figure 9 illustrates a flowchart of a method for generating a set of quadtrees for use as a virtual substrate according to some embodiments. Fig. Figure 10 illustrates a flowchart of a method for generating a combined set of quadtrees for use as a virtual substrate according to some embodiments. Fig. Figure 11 is a block diagram of a computing device according to some embodiments. DESCRIPTION OF EXECUTION FORMS

[0011] In the embodiments described below, while a media capture preview of a scene within the field of view of an image sensor is displayed, a reticle element superimposed in the media capture preview toggles from a first appearance state to a second appearance state to provide a visual indication that a layer within the scene has been captured. In the embodiments described below, while displayed in the second appearance state, the reticle element indicates the extent of the captured layer. Accordingly, the embodiments described below provide a seamless user experience that requires less time and user input when positioning augmented reality / virtual reality (AR / VR) objects within the scene relative to the captured layer, which is used as a virtual substrate.This also reduces energy consumption and improves the device's battery life by allowing the user to use the device faster and more efficiently.

[0012] In the embodiments described below, sets of planes or quadtrees for different viewpoints or fields of view are correlated across a temporal dimension to obtain a composite set of quadtrees, which are then used as virtual substrates. Therefore, the embodiments described below provide a process for identifying precise extents of virtual substrates based on different fields of view of a scene over time. Accordingly, the embodiments described below provide a seamless user experience that requires less time and user input when positioning AR / VR objects within the scene relative to the composite set of quadtrees (or planes associated with it). This also reduces power consumption and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0013] Below Fig. Sections 1A-1B, 2-3 and 11 provide a description of exemplary devices. Fig. Figure 7 illustrates a flowchart of a procedure for capturing a virtual substrate and arranging objects on it. The user interfaces in Fig. 4A-4M are used to process the process in Fig. 7 to illustrate. Fig. Figures 8-10 illustrate flowcharts of procedures for generating a set of quadtrees for use as a virtual substrate. The abstract block diagrams in Fig. 5A-5B and the user interfaces in Fig. 6A-6G are used to control the processes in Fig. 8-10 to illustrate. EXAMPLE DEVICES

[0014] Extensive reference will now be made to various embodiments, examples of which are illustrated in the accompanying drawings. The following detailed description includes numerous specific details to ensure a comprehensive understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be implemented without some of these specific details. In other cases, known methods, procedures, components, circuits, and networks have not been described in detail in order to avoid unnecessarily obscuring aspects of the embodiments.

[0015] It is also understood that although the terms "first / first / first", "second / second / second", etc. are used herein in some cases to describe different elements, these elements are not restricted by these terms. These terms are used only to distinguish one element from another. For example, a "first contact" could be referred to as a "second contact", and similarly, a "second contact" could be referred to as a "first contact" without altering the scope of protection of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0016] The terminology used in the description of the various embodiments described herein serves only to describe specific embodiments and is not intended to be restrictive. As used in the description of the various described embodiments and the accompanying claims, the singular forms "a", "an", "an" and "the", "the" are intended to include the plural forms as well, unless the context expressly indicates otherwise. It is also understood that the term "and / or", as used herein, refers to and includes any and all possible combinations of one or more of the related terms listed.It is further understood that the terms “include”, “encompassing”, “comprises” and / or “comprehensive”, when used in this patent specification, indicate the presence of listed features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more features, integers, steps, operations, elements, components and / or groups thereof.

[0017] As used here, the term "if" is optionally to be interpreted as meaning "during" or "at" or "in response to a determination" or "in response to a recording," depending on the context. Likewise, the phrase "if / when it is determined" or "if / when [a specified condition] is recorded" is optionally to be interpreted as meaning "upon determination" or "in response to a determination" or "upon recording [of the specified condition or event]" or "in response to a recording [of the specified condition or event]," depending on the context.

[0018] This document describes embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions, such as those of a PDA and / or music playback capabilities. Exemplary embodiments of portable multi-function devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc., Cupertino, California. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads), are used optionally.It should also be understood that in some embodiments the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g. a touchscreen display and / or a touchpad).

[0019] The following discussion describes an electronic device that includes a screen and a touch-sensitive surface. However, it should be understood that the electronic device may optionally include one or more physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0020] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk creation application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video application, an internet browsing application, a digital music playback application, and / or a digital video playback application.

[0021] The various applications running on the device optionally use at least one common physical user interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as corresponding information displayed on the device, are optionally adapted and / or varied from one application to another and / or within a given application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device optionally supports the various applications with user interfaces that are intuitive and recognizable to the user.

[0022] Attention is now being focused on embodiments of portable devices with touch-sensitive surfaces. Fig. Figure 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes called a "touchscreen" for simplicity and is sometimes simply called a touch-sensitive display. The device 100 includes a memory 102 (optionally including one or more computer-readable data storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a loudspeaker 111, a microphone 113, an input / output subsystem (I / O subsystem) 106, other input or control devices 116, and an external connector 124. The device 100 optionally includes one or more optical sensors 164.The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contacts on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). The device 100 optionally includes one or more touch output generators 163 for generating touch outputs on the device 100 (e.g., generating touch outputs on a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100 or the touchpad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0023] As used in the description and claims, the term “tactile output” or “touch output” refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., the housing) of the device, or a displacement of the component relative to a center of mass of the device, as detected by a user by their sense of touch. For example, in situations where the device or a component of the device comes into contact with a user’s touch-sensitive surface (e.g.,The tactile output generated by the physical movement of a device (a finger, palm, or other part of the user's hand) is interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or its component. For example, the movement of a touch-sensitive surface (such as a touchscreen or trackpad) is interpreted by the user as either a "down-click" or an "up-click" of a physical actuation button. In some cases, a user experiences a tactile sensation such as a "down-click" or an "up-click" even when no movement of a physical actuation button is associated with the touch-sensitive surface being physically depressed (e.g., moved) by the user's movements.As another example, the movement of the touch-sensitive surface is interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even if the smoothness of the touch-sensitive surface does not change. Although such interpretations of touch by a user depend on the individual user's sensory perceptions, there are many sensory perceptions of touch that are common to a large majority of users. Therefore, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an "up click," a "down click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or a component thereof that produces the described sensory perception for a typical (or average) user.

[0024] It should be understood that device 100 is only an example of a portable multifunctional device and that device 100 may optionally have more or fewer components than shown, optionally combine two or more components, or optionally have a different configuration or arrangement of the components. The various in Fig. The components shown in 1A are implemented in hardware, software, firmware or a combination thereof, including one or more signal processing and / or application-specific integrated circuits.

[0025] Memory 102 optionally includes high-speed random-access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor drive storage devices. Access to memory 102 by other components of the device 100, such as the CPU(s) 120 and the peripheral interface 118, is optionally controlled by the memory controller 122.

[0026] The peripheral interface 118 can be used to connect input and output peripherals of the device to the CPU(s) 120 and the memory 102. The one or more processors 120 execute various software programs and / or instruction sets stored in the memory 102 to perform various functions for the device 100 and to process data.

[0027] In some embodiments, the peripheral unit interface 118, the CPU(s) 120, and the memory controller 122 are optionally implemented on a single chip, such as a chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0028] The RF circuit (radio frequency circuit) 108 receives and transmits RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates via these electromagnetic signals with communication networks and other communication devices. The RF circuit 108 optionally includes well-known circuits for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module card (SIM card), memory, and the like. The RF circuit 108 optionally communicates with: a personal area network (PAN), such as a Bluetooth network; a local area network (LAN), such as an 802.11x Wi-Fi network; and / or a wide area network (WAN), such as a 4G mobile network.

[0029] The audio circuit 110, the loudspeaker 111, and the microphone 113 provide an audio interface between a user and a device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the loudspeaker 111. The loudspeaker 111 converts the electrical signal into sound waves audible to humans. The audio circuit 110 also receives electrical signals that are converted into sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. Audio data is selectively retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headphone jack (e.g., 212). Fig. 2) The headphone jack provides an interface between the audio circuit 110 and removable input / output peripherals, such as output-only headphones or a headphone set with both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0030] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch-sensitive display system 112 and other input or control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals to / from other input or control devices 116. The other input or control devices 116 optionally include physical buttons (e.g., pushbuttons, rocker switches, etc.), dials, slide switches, joysticks, click wheels, and the like.In some alternative embodiments, the input control(s) 160 is / are optionally coupled to one (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointing device such as a mouse. The one or more keys (e.g., 208, . Fig. 2) Optionally include an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g. 206, Fig. 2) one.

[0031] The touch-sensitive display system 112 provides an input interface and an output interface between the device and a user. The display controller 156 receives and / or sends electrical signals to and from the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual outputs to the user. The visual outputs optionally include graphics, text, symbols, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual outputs correspond to the user interface objects.

[0032] The touch-sensitive display system 112 has a touch-sensitive surface and a sensor or set of sensors that receives user input based on haptic / tactile contact. Thus, the touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or break in contact) on the touch-sensitive display system 112 and convert the detected contact into an interaction with user interface objects (e.g., one or more softkeys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, a contact point between the touch-sensitive display system 112 and the user corresponds to a finger of the user or a stylus.

[0033] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and the display controller 156 optionally detect contact and any movement or cessation of contact using a variety of known or yet-to-be-developed touch sensor technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrangements or other elements for determining one or more points of contact with the touch-sensitive display system 112.In one exemplary embodiment, projected mutual capacitance-based sensor technology is used, such as that found in the iPhone®, iPod Touch® and iPad® from Apple Inc. in Cupertino, California.

[0034] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touchscreen video resolution is higher than 400 dpi (e.g., 500 dpi, 800 dpi, or more). The user optionally makes contact with the touch-sensitive display system 112 by using a suitable object or element, such as a stylus, a finger, and the like. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which, due to the larger contact area of ​​a finger on the touchscreen, may be less precise than input with a stylus. In some embodiments, the device translates the coarse, finger-based input into a precise position of the mouse pointer / cursor or into a command to perform the actions desired by the user.

[0035] In some embodiments, the device 100 optionally includes a touchpad for activating or deactivating certain functions, in addition to the touchscreen. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display any visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0036] The device 100 also includes the power supply system 162 to provide power to the various components. The power supply system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with power generation, power management, and power distribution in portable devices.

[0037] The device 100 optionally includes one or more optical sensors 164 (sometimes referred to herein as the "image sensor" or "camera assembly"). Fig. Figure 1A shows an optical sensor coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensors 164 optionally include charge-coupled device (CCD) phototransistors or complementary metal-oxide semiconductors (CMOS). The optical sensors 164 receive ambient light projected through one or more lenses and convert the light into image-representing data. Together with the imaging module 143 (also referred to as the camera module), the optical sensors 164 optionally capture still images or videos. In some embodiments, an optical sensor is located on the rear of the device 100 opposite the touch-sensitive display system 112 on the front of the device, enabling the touchscreen to be used as a viewfinder for still and / or video image acquisition.In some embodiments, another optical sensor is arranged on the front of the device so that the image of the user (e.g. for selfies, for video conferences while the user sees the other participants of the video conference on the touchscreen, etc.) is obtained.

[0038] The device 100 optionally includes one or more contact intensity sensors 165. Fig. Figure 1A shows a contact intensity sensor coupled to the intensity sensor controller 159 in the I / O subsystem 106. The contact intensity sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensors 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is arranged together with or near a touch-sensitive surface (e.g., the touch-sensitive display system 112).In some embodiments, at least one contact intensity sensor is located on the rear of the device 100 opposite the touchscreen display system 112, which is located on the front of the device 100.

[0039] The device 100 optionally includes one or more proximity sensors 166. Fig. Figure 1A shows the proximity sensor 166, which is coupled to the peripheral unit interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, the proximity sensor switches off and disables the touch-sensitive display system 112 when the multifunction device is positioned near the user's ear (e.g., when the user is on a telephone call).

[0040] The device 100 optionally includes one or more touch output generators 163. Fig. Figure 1A shows a tactile output generator coupled to the haptic feedback controller 161 in the I / O subsystem 106. The tactile output generators 163 optionally include one or more electroacoustic devices, such as loudspeakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as a motor, an electromagnet, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or another tactile output generation component (e.g., a component that converts electrical signals into tactile outputs on the device). The tactile output generators 163 receive instructions from the haptic feedback module 133 to generate tactile feedback and produce tactile outputs on the device 100 that can be perceived by a user of the device 100.In some embodiments, at least one touch output generator is arranged together with or near a touch-sensitive surface (e.g., the touch-sensitive display system 112) and selectively generates a touch output by moving the touch-sensitive surface vertically (e.g., into or out of a surface of the device 100) or laterally (e.g., forwards and backwards in the same plane as a surface of the device 100). In some embodiments, at least one touch output generator sensor is located on the rear side of the device 100 opposite the touch-sensitive display system 112, which is located on the front side of the device 100.

[0041] The device 100 optionally includes one or more accelerometers 167, gyroscopes 168 and / or magnetometers 169 (e.g. as part of an inertial measurement unit (IMU)) for obtaining information concerning the position (e.g. orientation) of the device. Fig. Figure 1A shows sensors 167, 168, and 169 coupled to the peripheral unit interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, information based on an analysis of data received from the one or more accelerometers is displayed on the touchscreen in portrait or landscape orientation. The device 100 optionally includes a GPS receiver (or GLONASS receiver or receiver of another global navigation system) for obtaining information concerning the location of the device 100.

[0042] In some embodiments, the software components stored in memory 102 include the operating system 126, the communication module (or instruction set) 128, the contact / motion module (or instruction set) 130, the graphics module (or instruction set) 132, the haptic feedback module (or instruction set) 133, the text input module (or instruction set) 134, the GPS module (module of a global positioning system) (or instruction set) 135, and applications (or instruction set) 136. Furthermore, in some embodiments, memory 102 stores the device-related / global internal state 157, as described in Fig. 1A and Fig. 3 shown. The device-related / global internal state 157 includes one or more of: the active application state, which indicates whether and which applications are currently active; the display state, which indicates which applications, views, or other information occupy various areas of the touch-sensitive display system 112; the sensor state, including information received from the various sensors and other input or control devices 116 of the device; and location and / or position information relating to the location and / or position of the device.

[0043] The operating system 126 (e.g., iOS, MacOS, Darwin, LINUX, UNIX, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and supports communication between different hardware and software components.

[0044] The communication module 128 supports communication with other devices via one or more external ports 124 and also includes various software components for processing data received from the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is designed to connect directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a pin header (e.g., with 30 pins) that is identical or similar to, and / or compatible with, the 30-pin header used in some iPhone®, iPod Touch®, and iPad® devices manufactured by Apple Inc. in Cupertino, California.In some embodiments, the external connector is a Lightning connector that is identical or similar to and / or compatible with the Lightning connector used in some iPhone®, iPod Touch® and iPad® devices manufactured by Apple Inc. in Cupertino, California.

[0045] The contact / motion module 130 selectively detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or a physical click wheel). The contact / motion module 130 includes software components for performing various operations related to detecting contact (e.g., by a finger or a stylus), such as determining whether contact occurred (e.g., detecting a finger being placed or pressed down), determining the intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining whether the contact is moving and tracking the movement across the touch-sensitive surface (e.g., detecting a finger being pulled), and determining whether the contact has ceased (e.g.,(Detecting a finger lift or a break in contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Detecting movement of the contact point, represented by a series of contact data, optionally includes detecting speed (order of magnitude), velocity (order of magnitude and direction), and / or acceleration (a change in order of magnitude and / or direction) of the contact point. These operations are optionally applied to single contacts (e.g., single-finger contacts or stylus contacts) or to multiple simultaneous contacts (e.g., multi-touch / multi-finger contacts and / or stylus contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect a contact on a touchpad.

[0046] The Contact Motion Module 130 optionally captures user input via a gesture. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or intensities of captured contacts). Consequently, a gesture is optionally captured by capturing a specific contact pattern. For example, capturing a finger tap involves capturing a finger-place event followed by capturing a finger-lift or lift-off event at the same location (or substantially the same location) as the finger-place event (e.g., at the location of an icon). As another example, capturing a finger swipe gesture on the touch-sensitive surface involves capturing a finger-place event followed by capturing one or more finger-pulling events, and finally followed by capturing a finger-lift or lift-off event.Similarly, tapping, swiping, dragging and other gestures are optionally captured for a stylus by detecting a specific contact pattern for the stylus.

[0047] The Graphics Module 132 includes various known software components for rendering and displaying graphics on the Touch-Sensitive Display System 112 or any other display, including components for modifying the visual effect (e.g., brightness, transparency, saturation, contrast, or any other visual property) of displayed graphics. As used here, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0048] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications, etc., specifying graphics to be displayed, if necessary, along with coordinate data and other data relating to graphical properties, and then generates screen image data for output to the display controller 156.

[0049] The haptic feedback module 133 includes various software components for generating instructions that are used by the touch output generator(s) 163 to generate touch outputs at one or more locations on the device 100 in response to user interactions with the device 100.

[0050] The Text Input Module 134, which is optionally a component of the Graphics Module 132, provides on-screen keyboards for entering text in various applications (e.g., Contacts Module 137, Email Muddle 140, IM Module 141, Web Browser Module 147 and any other application that accepts text input).

[0051] The GPS module 135 determines the location of the device and makes this information available for use in various applications (e.g., the phone 138 for the use of location-based dialing, the camera 143 as image / video metadata, and applications that offer location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0052] The applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof: Contacts module 137 (sometimes called an address book or contact list); Telephone module 138; Videoconferencing module 139; Email client module 140; Instant messaging module (IM module) 141; Health / exercise module 142; Camera module 143 for still and / or video images; Image management module 144; Web browser module 147; Calendar module 148; Widget modules 149, optionally including one or more of: Weather widget, Stocks widget, Calculator widget, Alarm clock widget, Dictionary widget, and other widgets received from and created by the user; Widget creation module 150 for generating user-created widgets; Search module 151; Module for playing videos and music 152, which optionally consists of a video playback module and a music playback module; notes module 153; map module 154; and / or online video module 155.

[0053] Examples of other applications 136 that are optionally stored in memory 102 include other word processing applications, other image processing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, speech recognition and voice replication.

[0054] Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the functions and procedures described above (e.g., the computer-implemented procedures and other information-processing procedures described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules, and therefore different subsets of these modules are optionally combined or otherwise arranged differently in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures specified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0055] In some embodiments, the device 100 is a device in which the operation of a predefined set of functions on the device is performed exclusively by means of a touchscreen and / or a touchpad. By using a touchscreen and / or a touchpad as the primary input control device for the operation of the device 100, the number of physical input control devices (such as pushbuttons, dials, and the like) on the device 100 is optionally reduced.

[0056] The predefined set of functions, performed exclusively by a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, when touched by the user, the touchpad navigates the device 100 from any user interface displayed on the device 100 to a main, home, or main directory menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

[0057] Fig. 1B is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 (in Fig. 1A) or Fig. 370 ( Fig. 3) an event sorter 170 (e.g. in the operating system 126) and a corresponding application 136-1 (e.g. any of the applications listed above 136, 137-155, 380-390).

[0058] The event sorter 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be sent. The event sorter 170 includes an event monitor 171 and an event forwarding module 174. In some embodiments, the application 136-1 includes the internal application state 192, which indicates the current application view or views displayed on the touch-sensitive display system 112 when the application is active or running. In some embodiments, the device-related / global internal state 157 is used by the event sorter 170 to determine which application or applications are currently active, and the internal application state 192 is used by the event sorter 170 to determine the application views 191 to which event information is to be delivered.

[0059] In some embodiments, the internal application state 192 includes additional information, such as one or more of: Resume information for use when the execution of the application 136-1 is resumed; user interface status information indicating that information is already being displayed or is ready to be displayed by the application 136-1; a status queue to allow the user to return to a previous state or view of the application 136-1; and a redo / undo queue for actions previously performed by the user.

[0060] The event monitor 171 receives event information from the peripheral unit interface 118. Event information includes information about a sub-event (e.g., a user touching the touch-sensitive display system 112 as part of a multi-touch gesture). The peripheral unit interface 118 transmits information it receives from the I / O subsystem 106 or from a sensor, such as the proximity sensor 166, accelerometer(s) 167, gyroscope(s) 168, magnetometer(s) 169, and / or the microphone 113 (via the audio circuit 110). Information received by the peripheral unit interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display system 112 or from a touch-sensitive surface.

[0061] In some embodiments, the event monitor 171 sends requests to the peripheral unit interface 118 at predefined intervals. In response, the peripheral unit interface 118 transmits event information. In other embodiments, the peripheral unit interface 118 transmits event information only when a significant event occurs (e.g., receiving an input that exceeds a predefined noise threshold and / or lasts longer than a predefined duration).

[0062] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or a determination module for detecting an active event 173.

[0063] The Hit View Determination Module 172 provides software procedures to determine where a sub-event occurred within one or more views when the touch-sensitive display system 112 displays more than one view. The views consist of controls and other elements that a user can see on the display.

[0064] Another aspect of the user interface associated with an application is a set of views, sometimes referred to here as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of a given application) in which a touch is detected correspond optionally to programmatic levels within a programmatic or view hierarchy of the application. For example, the lowest level of the view in which a touch is detected is optionally referred to as the hit view, and the set of events recognized as valid inputs is optionally determined at least partially based on the hit view of the original touch that initiates a touch-based gesture.

[0065] The Hit View Determination Module 172 receives information regarding sub-events of a touch-based gesture. If an application has several views arranged in a hierarchy, the Hit View Determination Module 172 identifies a hit view as the lowest view in the hierarchy that should handle the sub-event. In most cases, the hit view is the lowest level of the view where the sub-event is initiated (for example, the first sub-event in a sequence of sub-events that constitute an event or a potential event). Once the hit view has been identified by the Hit View Determination Module, it typically receives all sub-events associated with the same touch or input source for which it was identified as the hit view.

[0066] The active event detection module 173 determines which view or views within a view hierarchy should receive a specific sequence of sub-events. In some embodiments, the active event detection module 173 determines that only the hit view should receive a specific sequence of sub-events. In other embodiments, the active event detection module 173 determines that all views encompassing the physical location of a sub-event are actively involved views, thus determining that all actively involved views should receive a specific sequence of sub-events. In other embodiments, even if contact sub-events were limited exclusively to the area associated with a particular view, higher-level views in the hierarchy would still remain actively involved views.

[0067] The event forwarding module 174 forwards the event information to an event detection device (e.g., the event detection device 180). In some embodiments that include the determination module for detecting an active event 173, the event forwarding module 174 provides the event information to an event detection device determined by the determination module for detecting an active event 173. In some embodiments, the event forwarding module 174 stores the event information in an event queue, which is retrieved by a respective event receiving module 182.

[0068] In some embodiments, the operating system 126 includes the event sorter 170. Alternatively, the application 136-1 includes the event sorter 170. In still other embodiments, the event sorter 170 is a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0069] In some embodiments, the application 136-1 includes a plurality of event handling devices 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event detection devices 180. Typically, each application view 191 includes a plurality of event detection devices 180. In other embodiments, one or more event detection devices 180 are part of a separate module, such as a user interface kit or a higher-level object, from which the application 136-1 inherits procedures and other properties.In some embodiments, each event handling device 190 includes one or more of: a data update device 176, an object update device 177, a GUI update device 178, and / or event data 179 received by the event sorter 170. The event handling device 190 selectively uses or calls the data update device 176, the object update device 177, or the GUI update device 178 to update the internal application state 192. Alternatively, one or more of the application views 191 include one or more of each event handling device 190. Also, in some embodiments, one or more of the data update device 176, object update device 177, and GUI update device 178 are included in each application view 191.

[0070] Each event detection device 180 receives event information (e.g., the event data 179) from the event sorter 170 and identifies an event from the event information. The event detection device 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event detection device 180 also includes at least a subset of: metadata 183 and event delivery instructions 188 (which optionally include partial event delivery instructions).

[0071] The event receiver 182 receives event information from the event sorter 170. The event information includes information regarding a sub-event, such as a touch or a touch movement. Depending on the sub-event, the event information may also include additional information, such as the position at which the sub-event occurs. If the sub-event involves the movement of a touch, the event information may optionally include the speed and direction of the sub-event. In some embodiments, events include the rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation or vice versa), and the event information includes corresponding information regarding the current orientation of the device (also referred to as the spatial position of the device).

[0072] The event comparator 184 compares the event information with predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event, or determines or updates the status of an event or sub-event. In some embodiments, the event comparator 184 includes event definitions 186. The event definitions 186 contain definitions of events (e.g., predefined sequences of sub-events), for example, Event 1 (187-1), Event 2 (187-2), and others. In some embodiments, sub-events within an Event 187 include, for example, touch start, touch end, touch movement, touch termination, and multiple touches. In one example, the definition for Event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises an initial touch (start of touch) on the displayed object during a predetermined phase, an initial lift-off (end of touch) during a predetermined phase, a second touch (start of touch) on the displayed object during a predetermined phase, and a second lift-off (end of touch) during a predetermined phase. In another example, the definition for event 2 (187-2) is a drag on a displayed object. The drag comprises, for example, a touch (or contact) on the displayed object during a predetermined phase, a movement of the touch across the touch-sensitive display system 112, and a lift-off of the touch (end of touch). In some embodiments, the event also includes information for one or more connected event handling devices 190.

[0073] In some embodiments, the event definition 187 includes a definition of an event for each user interface object. In some embodiments, the event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display system 112, the event comparator 184 performs a hit test when a touch is detected on the touch-sensitive display system 112 to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handling device 190, the event comparator uses the result of the hit test to determine which event handling device 190 should be activated.For example, the event comparator 184 selects an event handling device that is associated with the sub-event and the object that triggers the hit test.

[0074] In some embodiments, the definition for a given event 187 also includes delayed actions that delay the delivery of the event information until it has been determined whether the sequence of sub-events corresponds to the event type of the event detection device or not.

[0075] If an event detection device 180 determines that the sequence of sub-events does not correspond to any of the events in the event definitions 186, the respective event detection device 180 enters an event-impossible, event-failed, or event-completed state and subsequently ignores subsequent sub-events of the touch-based gesture. In this situation, any other event detection devices that remain active for the hit view continue to track and process sub-events of an ongoing touch-based gesture.

[0076] In some embodiments, each event detection device 180 includes metadata 183 with configurable properties, indicator symbols, and / or lists that indicate how the event delivery system should deliver sub-events to devices for determining actively involved events. In some embodiments, the metadata 183 includes configurable properties, indicator symbols, and / or lists that indicate how event detection devices interact or how they are enabled to interact with each other. In some embodiments, the metadata 183 includes configurable properties, indicator symbols, and / or lists that indicate whether sub-events are delivered to varying levels in the view or program hierarchy.

[0077] In some embodiments, a respective event detection device 180 activates the event handling device 190 associated with an event when one or more specific sub-events of an event are detected. In some embodiments, a respective event detection device 180 provides event information associated with the event to the event handling device 190. Activating an event handling device 190 differs from sending (and delayed sending) sub-events to a respective hit view. In some embodiments, the event detection device 180 triggers an alert symbol associated with the detected event, and the event handling device 190 associated with the alert symbol captures the alert symbol and performs a predefined process.

[0078] In some embodiments, the event delivery instructions include sub-event delivery instructions that deliver event information about a sub-event without activating an event handling device. Instead, the sub-event delivery instructions deliver event information to the event handling devices associated with the sequence of sub-events or with the actively involved views. The event handling devices associated with the sequence of sub-events or with the actively involved views receive the event information and execute a predefined process.

[0079] In some embodiments, the data update device 176 creates and updates data used in the application 136-1. For example, the data update device 176 updates the telephone number used in the contacts module 137 or saves a video file used in the video playback module 145. In some embodiments, the object update device 177 creates and updates objects used in the application 136-1. For example, the object update device 176 creates a new user interface object or updates the position of a user interface object. The GUI update device 178 updates the GUI. For example, the GUI update device 178 prepares display information and sends it to the graphics module 132 for display on a touch-sensitive display.

[0080] In some embodiments, the one or more event handling devices 190 include or have access to the data update device 176, the object update device 177, and the GUI update device 178. In some embodiments, the data update device 176, the object update device 177, and the GUI update device 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0081] It is understood that the preceding discussion regarding the event handling of user touches on touch-sensitive displays also applies to other forms of user input for operating multifunction devices 100 with input devices not all of which are initiated on touchscreens. For example, mouse movements and mouse clicks, optionally mapped to single or multiple key presses or grips; contact movements, such as tapping, dragging, scrolling, etc., on touchpads; pen inputs; device movement; verbal instructions; captured eye movements; biometric inputs; and / or any combination thereof are optionally used as inputs corresponding to sub-events that define an event to be detected.

[0082] Fig. Figure 2 illustrates a portable multifunction device 100 with a touchscreen (e.g. the touch-sensitive display system 112, Fig. 1A) according to some embodiments. The touchscreen selectively displays one or more graphics within the user interface (UI) 200. In this embodiment, as well as in other embodiments described below, a user is enabled to select one or more of the graphics by performing a gesture on the graphics, for example, with one or more fingers 202 (not shown to scale in the figure) or one or more styluses 203 (not shown to scale in the figure). In some embodiments, the selection of one or more graphics takes place when the user breaks contact with the one or more graphics.In some embodiments, the gesture optionally includes one or more tapping movements, one or more swiping movements (from left to right, right to left, up and / or down), and / or a rolling movement of a finger (from right to left, left to right, up and / or down) that has made contact with the device 100. In some embodiments or cases, accidental contact with a graphic does not result in the graphic being selected. For example, a swipe gesture across an application icon may optionally not select the corresponding application if the gesture corresponding to the selection is a tap.

[0083] The device 100 optionally includes one or more physical buttons, such as a "Home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that can optionally be run on the device 100. Alternatively, in some embodiments, the menu button is implemented as a softkey in a GUI that is displayed on the touchscreen display.

[0084] In some embodiments, the device 100 includes the touchscreen display, the menu button 204, the power button 206 for turning the device on / off and locking it, the volume control button(s) 208, the subscriber identity module card slot (SIM card slot) 210, the headphone jack 212, and an external docking / charging port 124. The power button 206 is used optionally to turn the device on and off by pressing and holding the button for a predefined time interval; to lock the device by pressing and releasing the button before the predefined time interval has elapsed; and / or to unlock the device or to initiate an unlocking process. In some embodiments, the device 100 also accepts spoken input to activate or deactivate certain functions via the microphone 113.The device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of contacts on the touch-sensitive display system 112 and / or one or more key output generators 163 for generating key outputs for a user of the device 100.

[0085] Fig. Figure 3 is a block diagram of an exemplary multifunctional device with a display and a touch-sensitive surface according to some embodiments. The device 300 need not be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home or industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for connecting these components.The communication buses 320 optionally include a circuit (sometimes referred to as a chipset) that connects and controls communication between system components. The device 300 includes the input / output (I / O) interface 330, which comprises the display 340, typically a touchscreen display. The I / O interface 330 optionally also includes a keyboard and / or mouse (or other pointing device) 350 and the touchpad 355, the keystroke output generator 357 for generating keystroke outputs on the device 300 (e.g., similar to those described above). Fig. 1A described touch output generators 163), the sensors 359 (e.g. touch-sensitive, optical, contact intensity, proximity, acceleration, position and / or magnetic sensors similar to those above in relation to Fig. 1A described sensors 112, 164, 165, 166, 167, 168 and 169). Memory 370 includes high-speed random-access memory, such as DRAM, SRAM, DDR-RAM or other random-access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state memory devices. Memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310. In some embodiments, memory 370 stores programs, modules and data structures analogous to the programs, modules and data structures stored in memory 102 of the portable multifunction device 100 ( Fig. 1A), or a subset thereof. Furthermore, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of the portable multifunction device 100. For example, memory 370 of the device 300 optionally stores the drawing module 380, the presentation module 382, ​​the word processing module 384, the website creation module 386, the disk creation module 388, and / or the spreadsheet module 390, while memory 102 of the portable multifunction device 100 ( Fig. 1A) optionally does not save these modules.

[0086] Each of the above-mentioned elements in Fig. 3 is optionally stored in one or more of the aforementioned storage devices. Each of the modules shown above corresponds to a set of instructions for performing a function described above. The modules or programs (i.e., sets of instructions) shown above need not be implemented as separate software programs, procedures, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in different embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures shown above. Furthermore, memory 370 optionally stores additional modules and data structures not described above. USER INTERFACES AND LINKED PROCESSES

[0087] Attention is now directed to embodiments of user interfaces (“UI”) and associated processes that can be implemented on an electronic device, such as a portable multifunction device (PMD) 100 with a display, a touch-sensitive surface and optionally one or more processors for detecting the intensity of contacts with the touch-sensitive surface, or a device 300 with one or more processors, a non-volatile memory, a display and an input device.

[0088] While the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger taps, finger swipes, etc.), it should be understood that in some embodiments, one or more of the finger inputs are replaced by inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a contact) followed by movement of the cursor along the swipe path (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click while the cursor is over the position of the tap gesture (e.g., instead of tapping the contact followed by releasing the tapping).

[0089] If multiple user inputs are captured simultaneously, it is understood analogously that multiple computer mice can be used either simultaneously or a mouse and finger contacts can be used simultaneously.

[0090] Fig. Figures 4A-4M illustrate exemplary user interfaces for capturing a virtual substrate and arranging objects on it according to some embodiments. The user interfaces in these figures are used to perform the processes described below, including the process in Fig. 7, to illustrate. Although some of the following examples are given with reference to inputs on a touchscreen display (where the touch-sensitive surface and the display are combined), in some embodiments the device detects inputs via an input device separate from the display (e.g., a laptop with a separate touchpad and a separate display, or a desktop with a separate mouse and a separate display).

[0091] As in Fig. As shown in Figure 4A, the device 100 displays a media capture / interaction interface 402, which is provided for capturing layers and placing augmented reality and / or virtual reality (AR / VR) objects on them. According to some embodiments, the media capture / interaction interface 402 corresponds to a media capture preview of a scene with objects in the field of view of an image sensor of the device, which changes as the field of view of the image sensor changes. For example, the media capture preview in Fig. 4A has an open entrance to a room with a table 415 inside.

[0092] In Fig. 4A includes the media capture / interaction interface 402: a snapshot affordance 404a provided to capture an image in response to its selection (e.g., with a contact or selection gesture); an AR / VR object menu affordance 404b provided to enable an object selection interface (e.g., as in Fig. 4E-4F shown) in response to their selection (e.g., with a contact or selection gesture); and an option affordance 404c provided to display an options and / or settings menu in response to its selection (e.g., with a contact or selection gesture). In Fig. 4A includes the media capture / interaction interface 402, which also contains a reticle element that is displayed in an initial appearance state 410. As in Fig. As shown in Figure 4A, the reticle element in the first appearance state 410 corresponds to a transparent rectangle with opaque corners but no edges. In some embodiments, the reticle element corresponds to a rectangle, an ellipse, a polygon, a magnifying glass, a crosshair, or the like.

[0093] Fig. Figures 4A-4B show a sequence in which the media capture / interaction interface 402 is updated based on a change in the field of view of the image sensor of the device 100. For example, the media capture preview in Fig. 4B two walls of the room containing table 415. Therefore, the perspective or viewing angle of device 100 changes between Fig. 4A-4B.

[0094] Fig. Figures 4B-4C show a sequence in which the appearance state of the reticle element changes from the first to the second appearance state in response to the detection of a nearby plane. For example, device 100 detects a plane with table 415. Continuing with this example, in response to the detection of the plane, device 100 changes the reticle element from the first appearance state 410 to the second appearance state 410'. As shown in Fig. As shown in Figure 4C, in the second appearance state 410', the reticle element corresponds to a hatched rectangle with dotted edges. In some embodiments, the edges of the reticle element flash or glow in the second appearance state 410'. In some embodiments, the edges of the reticle element flash or glow in a clockwise or counterclockwise direction. According to some embodiments, while displayed in the second appearance state 410', the reticle element provides a visual indication that the device 100 has detected a plane within the scene. According to some embodiments, while displayed in the second appearance state 410', the reticle element provides a visual indication of a portion of the extent of the detected plane that is connected to the table 415.

[0095] Fig. Figures 4C-4D show a sequence in which the appearance state of the reticular element changes from the second to the third appearance state in response to the detection of a plane orientation. For example, device 100 shows Fig. 4D the reticulate element in a third appearance state 410" by rotating and / or aligning the reticulate element to the orientation of the captured plane connected to the table 415.

[0096] Fig. Figures 4D-4E show a sequence in which the appearance state of the reticular element changes from the third to the fourth appearance state in response to the detection of a larger extent of the plane. For example, device 100 shows Fig. 4E the reticle element in a fourth appearance state 410''' by enlarging the area of ​​the reticle element in response to the detection of additional points connected to the plane, indicating that its extent is larger than previously detected.

[0097] Fig. Images 4E-4G show a sequence in which an AR / VR object is positioned within the scene relative to the captured plane. As in Fig. As shown in Figure 4E, the device 100 detects a contact 412 (e.g., a tap or selection gesture) at a location corresponding to the AR / VR object menu affordance 404b. As shown in Fig. As shown in Figure 4F, the device 100 shows the object selection interface 472 superimposed on the media detection / interaction interface 402 in response to a detection of the selection of the AR / VR object menu affordance 404b in Fig. 4E on.

[0098] In Fig. 4F includes the object selection interface 472 and a variety of AR / VR object category tabs 474a, 474b, and 474c, which are associated with shapes, animals, and transportation, respectively. Fig. Currently, the AR / VR object category tab 474a, associated with shapes, is selected. This means that the object selection interface 472 includes a variety of user-selectable AR / VR objects 476a, 476b, 476c, 476d, 476e, and 476f (sometimes referred to collectively as the "user-selectable AR / VR objects 476") associated with the shape category. In some embodiments, each of the user-selectable AR / VR objects 476 is associated with a name, a preview image, associated metadata, and / or the like. Fig. 4F includes the object selection interface 472 and also an additional category affordance 478, which is provided to display additional categories of AR / VR objects in response to their selection (e.g., with a contact or selection gesture).

[0099] As in Fig. As shown in Figure 4F, the device 100 detects a contact 414 (e.g., a tap or selection gesture) at a location corresponding to the user-selectable AR / VR object 476f (e.g., the cuboid object). As shown in Fig. As shown in 4G, the device 100 displays a cuboid AR / VR object 420 within the scene relative to the detected plane in response to the detection of the user-selectable AR / VR object 476f in Fig. 4F. In some embodiments, the device displays the cuboid AR / VR object 420 at the geometric center (e.g., the center of gravity) of the detected plane.

[0100] Fig. Images 4G-4H show a sequence in which the size of the cuboid AR / VR object increases by 420. As in Fig. As shown in Figure 4G, the device 100 detects a reverse clamping gesture with contacts 416a and 416b on the cuboid AR / VR object 420. As shown in Fig. As shown in Figure 4H, the device 100 increases the size of the cuboid AR / VR object 420 within the scene relative to the detected plane in response to the detection of the reverse pinch gesture in Fig. 4G.

[0101] Fig. Images 4H-4I show a sequence in which a cuboid AR / VR object moves 420 degrees relative to the captured plane. As in Fig. As shown in Figure 4H, the device 100 detects a tap and pull gesture with contact 418 on the cuboid AR / VR object 420. As shown in Fig. As shown in Figure 4I, the device 100 displays the cuboid AR / VR object 420 closer to the front edge 423 of the table 415 relative to the detected plane in response to the detection of the tap and pull gesture. Fig. 4H.

[0102] Fig. 4I-4J show a sequence in which the orientation of the cuboid AR / VR object 420 is changed. As in Fig. As shown in Figure 4I, the device 100 detects a counterclockwise rotation gesture with contacts 422a and 422b on the cuboid AR / VR object 420. As shown in Fig. As shown in Figure 4J, the device 100 rotates the cuboid AR / VR object 420 counterclockwise within the scene relative to the detected plane in response to the detection of a counterclockwise rotation gesture. Fig. 4I.

[0103] Fig. 4J-4K show a sequence in which the cuboid AR / VR object 420 is split into cuboid AR / VR objects 430a and 430b. As in Fig. As shown in Figure 4J, the device 100 detects a predefined interaction gesture (e.g., a single or double tap gesture) with contact 424 at a location corresponding to the middle front top edge 427 of the cuboid AR / VR object 420. As shown in Fig. As shown in 4K, the device 100 divides the cuboid AR / VR object 420 into the cuboid AR / VR objects 430a and 430b based on the location of the interaction gesture in Fig. 4H and displays the cuboid AR / VR objects 430a and 430b in relation to the captured plane.

[0104] Fig. Figures 4K-4L show a sequence in which the media capture / interaction interface 402 is updated based on a change in the field of view of the image sensor of the device 100. For example, the media capture preview in Fig. 4L encloses a single wall of the room with table 415 in it. Therefore, the perspective or viewing angle of device 100 changes, and the perspective of the cuboid AR / VR objects 430a and 430b changes accordingly.

[0105] Fig. Figures 4L-4M show a sequence in which the appearance state of the reticle element changes from the fourth to the fifth appearance state in response to the detection of user input that interacts with an edge of the reticle element. As shown in Fig. As shown in Figure 4L, the device 100 detects a tap and pull gesture with contact 426, whereby an edge 442 of the reticulate element is pulled towards the edge 444 of the table 415. For example, the device 100 is shown in Fig. 4M displays the reticle element in a fifth appearance state 410'''' by changing the size of the reticle element in response to a tap and pull gesture detection in Fig. The volume is increased by 4L.

[0106] Fig. Figure 5A illustrates an abstract block diagram associated with a process 500 for generating a set of quadtrees according to some embodiments. While relevant features are shown, the person skilled in the art will recognize from the present disclosure that various other features have been omitted for the sake of brevity, so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. For example, in some embodiments, the process 500 generates the set of quadtrees 525 by combining sets of planes 520a, 520b, ..., 520n, which are created from sets of images acquired at different reference points / viewpoints over time (e.g., camera position or fields of view).

[0107] As in Fig. As shown in 5A, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Fig. 11) a first set of images 504a (e.g. image data) related to a first reference point / viewing angle 502a. In some embodiments, the device 100 or a component thereof (e.g. the point cloud synthesis module 1156 in Fig. 11) a first three-dimensional (3D) point cloud 506a based on the first set of images 504a. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) a first set of planes 520a based on the first 3D point cloud 506a. For example, the device 100 creates the first set of planes 520a by fitting planes to the first 3D point cloud 506 according to algorithms or techniques known in the field (e.g. least matching squares, principal component analysis, simultaneous localization and mapping (SLAM), etc.).

[0108] In a similar way to how in Fig. As shown in 5A, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Fig. 11) a second set of images 504b related to a second reference point / viewing angle 502b. In some embodiments, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11) a second 3D point cloud 506b based on the second set of images 504b. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) a second set of planes 520b based on the second 3D point cloud 506b.

[0109] In a similar way to how in Fig. As shown in 5A, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Fig. 11) an nth set of images 504n related to an nth reference point / viewing angle 502n. In some embodiments, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11) an nth 3D point cloud 506n based on the nth set of images 504n. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) an nth set of planes 520n based on the nth 3D point cloud 506n.

[0110] According to some embodiments, the device 100 or a component thereof (e.g. the correlation module 1162 in Fig. 11) the first set of planes 520a, the second set of planes 520b, ... and the nth set of planes 520n to generate a combined set of planes. In some embodiments, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11) the set of quadtrees 525 based on the aggregated set of planes. For example, the device 100 generates the set of quadtrees 525 according to algorithms or techniques known in the field.

[0111] Fig. Figure 5B illustrates an abstract block diagram associated with a process 550 for generating a set of quadtrees according to some embodiments. While relevant features are shown, the person skilled in the art will recognize from the present disclosure that various other features have been omitted for the sake of brevity, so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. For example, in some embodiments, the aggregated set of quadtrees 540 is generated in process 550 by aggregating sets of quadtrees 530a, 530b, ..., 530n created from 3D point clouds associated with different reference points / viewpoints over time (e.g., camera position or fields of view).

[0112] As in Fig. As shown in Figure 5B, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Figure 5B). Fig. 11) a first set of images 504a (e.g. image data) related to a first reference point / viewing angle 502a. In some embodiments, the device 100 or a component thereof (e.g. the point cloud synthesis module 1156 in Fig. 11) a first 3D point cloud 506a based on the first set of images 504a. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) a first set of planes 520a based on the first 3D point cloud 506a and / or the first set of images 504a. In some embodiments, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11) a first set of quadtrees 530a based on the first set of planes 520a and / or the first 3D point cloud 506a. For example, the device 100 generates the first set of quadtrees 530a according to details referred to in Fig. 10 are described.

[0113] In a similar way to how in Fig. As shown in Figure 5B, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Figure 5B). Fig. 11) a second set of images 504b related to a second reference point / viewing angle 502b. In some embodiments, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11) a second 3D point cloud 506b based on the second set of images 504b. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) a second set of planes 520b based on the second 3D point cloud 506b and / or the second set of images 504b. In some embodiments, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11) a second set of quadtrees 530b based on the second set of planes 520b and / or the second 3D point cloud 506b.

[0114] In a similar way to how in Fig. As shown in Figure 5B, the device receives 100 or a component thereof (e.g., the image acquisition control module 1150 in Figure 5B). Fig. 11) an nth set of images 504n related to an nth reference point / viewing angle 502n. In some embodiments, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11) an nth 3D point cloud 506n based on the nth set of images 504n. In some embodiments, the device 100 creates a component thereof (e.g., the plane adjustment module 1158 in Fig. 11) an nth set of planes 520n based on the nth 3D point cloud 506n and / or the nth set of images 504n. In some embodiments, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11) an nth set of quadtrees 530n based on the nth set of planes 520n and / or the nth 3D point cloud 506n.

[0115] According to some embodiments, the device 100 or a component thereof (e.g. the correlation module 1162 in Fig. 11) the first set of Quadtrees 530a, the second set of Quadtrees 530b, ... and the nth set of Quadtrees 530n to obtain a combined set of Quadtrees 540. For example, the device 100 generates the combined set of Quadtrees 540 according to details referred to Fig. 10 are described.

[0116] Fig. Figures 6A-6M illustrate exemplary user interfaces for capturing a virtual substrate according to some embodiments. The user interfaces in these figures are used to perform the processes described below, including the processes in Fig. 8-10, to illustrate. Although some of the following examples refer to inputs on a touchscreen display (where the touch-sensitive surface and the display are combined), in some embodiments the device detects inputs on a touch-sensitive surface 451 that is separate from the display 450, as in Fig. 4B shown.

[0117] In some embodiments, the device 100 indicates the steps that are to be performed in Fig. 6A-6H are performed within a user interface similar to the media capture / interaction interface 652 in Fig. 6G is. In some embodiments, the device 100 performs the steps in Fig. 6A-6H, but does not display these, and instead displays the resulting levels 650a, 650b, 650c and 650d within the media capture / interaction interface 652 in Fig. 6G on.

[0118] As in Fig. As shown in Figure 6A, the device 100 captures a plurality of clusters of points 605a, 605b, 605c, and 605d within a scene. For example, the cluster of points 605a corresponds to a first table 604a within the scene. For example, the cluster of points 605b corresponds to a section of the floor within the scene. For example, the cluster of points 605c corresponds to another section of the floor within the scene. For example, the cluster of points 605d corresponds to a second table 604b within the scene. According to some embodiments, the device 100 receives a plurality of images of the scene and synthesizes a three-dimensional (3D) point cloud of points identified within the scene, which includes the clusters of points 605a, 605b, 605c, and 605d. In some embodiments, the device 100 or a component thereof (e.g., the coordinate transformation module 1154 in Figure 6A) tracks the coordinate transformation module 1154. Fig. 11) the world coordinates 602a and the device coordinates 602b to perform transformations between an image space associated with the device coordinates 602b and a 3D space (e.g. the room or scene in Fig. 6A-6G), which is connected to the world coordinates 602a, to carry out.

[0119] As in Fig. As shown in Figure 6B, the device 100 creates (e.g., identifies) a multitude of planes 610a, 610b, 610c, and 610d (e.g., infinite planes) in a 3D coordinate space connected to the 3D point cloud, based on the clusters of points 605a, 605b, 605c, and 605d in Fig. 6A. In some embodiments, the device 100 creates the plurality of planes 610a, 610b, 610c and 610d by fitting infinite planes to the clusters of points 605a, 605b, 605c and 605d in the 3D point cloud according to algorithms or techniques known in the field (e.g. least matching squares, principal component analysis, simultaneous localization and mapping (SLAM), etc.).

[0120] As in Fig. As shown in Figure 6C, the device 100 triangulates points within the clusters of points 605a, 605b, 605c and 605d in a two-dimensional (2D) coordinate space associated with the multitude of images used to synthesize the 3D point cloud. Fig. 6A.

[0121] As in Fig. As shown in Figure 6D, the device removes 100 triangles in the 2D coordinate space that have points not corresponding to a plane based on the plurality of planes 610a, 610b, 610c and 610d in Figure 6D. Fig. 6B are connected to obtain a variety of constrained triangulated regions 620a, 620b, 620c and 620d.

[0122] As in Fig. As shown in Figure 6E, the device 100 projects the plurality of constrained triangulated regions 620a, 620b, 620c and 620d back into the 3D coordinate space and squares the plurality of constrained triangulated regions 620a, 620b, 620c and 620d to obtain quadtrees 630a, 630b, 630c and 630d. Fig. 6E the quadtrees 630a, 630b, 630c and 630d are enclosed by bounding frames based on their extent.

[0123] As in Fig. 6F was shown and, due to the capture of additional points, the quadtree 630a, which is connected to the first table 604a, was enlarged to state 630a', the quadtrees 630b and 630c, which are connected to the floor, were combined into quadtree 630e, the quadtree 630d, which is connected to the second table 604b, was enlarged to state 630d' and a new quadtree 630f, which is connected to the wall of the room within the scene, was captured.

[0124] As in Fig. As shown in Figure 4G, the device 100 displays a media capture / interaction interface 652, which is provided to capture layers and arrange augmented reality and / or virtual reality (AR / VR) objects on it. According to some embodiments, the media capture / interaction interface 652 corresponds to a media capture preview of a scene with objects in the field of view of an image sensor of the device, which changes as the field of view of the image sensor changes. For example, the media capture preview in Fig. 6G enters the scene, which in Fig. 6A-6F is shown with tables 604a and 604b. According to some embodiments, the media capture / interaction interface 652 is in Fig. 6G similar to the media capture / interaction interface 402 in Fig. 4A-4M or adapted from this.

[0125] In Fig. 6G includes the media capture / interaction interface 652: a snapshot affordance 654a, provided to capture an image in response to its selection (e.g., with a contact or selection gesture); an AR / VR object menu affordance 654b, provided to display an object selection interface in response to its selection (e.g., with a contact or selection gesture); and an options affordance 654c, provided to display an options and / or settings menu in response to its selection (e.g., with a contact or selection gesture). Fig. 6G includes the user interface 602 also level extensions 650a, 650b, 650c and 650d, which define the bounding box of the quadtrees 630a', 630d', 630e and 630f respectively in Fig. 6F corresponds.

[0126] As in Fig. As shown in Figure 6G, each of the plane extensions 650a, 650b, 650c, and 650d is displayed with a unique appearance, pattern, fill, and / or the like. According to some embodiments, each of the plane extensions 650a, 650b, 650c, and 650d corresponds to a virtual substrate on which an AR / VR object is placed (e.g., as in Figure 6G). Fig. 4F-4G shown) and edited (e.g. as in Fig. 4G-4K). In some embodiments, each of the plane extents 650a, 650b, 650c, and 650d provides a visual indication that a plane has been captured within the scene. In some embodiments, each of the plane extents 650a, 650b, 650c, and 650d provides a visual indication of a portion of the extent of the associated captured quadtrees.

[0127] Fig. Figure 7 is a flowchart representation of a method 700 for sensing a virtual substrate and arranging objects on it according to some embodiments. In some embodiments (and as detailed by way of example below), the method 700 is performed by an electronic device (or a section thereof), such as the electronic device 100 in Fig. 1 or the device 300 in Fig. 3, carried out, which includes one or more processors, non-volatile memory, an image sensor or camera arrangement, a display, and one or more input devices. For example, the display and the one or more input devices are combined to form a touchscreen display. In this example, the electronic device corresponds to a smartphone or a tablet. In another example, the display and the one or more input devices are separate. In this example, the electronic device corresponds to a laptop or desktop computer. For example, the electronic device corresponds to a portable rake, a smartphone, a tablet, a laptop computer, a desktop computer, a kiosk, a set-top box (STB), a game console, and / or the like.

[0128] In some embodiments, Method 700 is performed by processing logic, including hardware, firmware, software, or a suitable combination thereof. In some embodiments, Method 700 is performed by one or more processors executing code, programs, or instructions stored in a non-volatile, computer-readable storage medium (e.g., non-volatile memory). Some operations in Method 700 are optionally combined, and / or the order of some operations is optionally changed.In short, Procedure 700 includes: changing a reticle element from a first appearance state to a second appearance state in response to the detection of a layer within a scene; positioning an augmented reality / virtual reality (AR / VR) object in the scene relative to the detected layer; and modifying / editing the AR / VR object based on user input.

[0129] Method 700 begins in block 702 with the electronic device superimposing a reticle element in a media capture preview and displaying it in a first appearance state. For example, device 100 shows Fig. 4B connects the media capture / interaction interface 402, which includes a media capture preview of a scene corresponding to a room containing a table 415 and two walls. Continuing with this example, the media capture / interaction interface 402 connects in Fig. 4B also includes a reticle element, which is displayed in a first appearance state 410. In some embodiments, the reticle element in the first appearance state 410 corresponds to a transparent rectangle with opaque corners but no edges. In some embodiments, the reticle element corresponds to a rectangle, an ellipse, a polygon, a magnifying glass, a crosshair, or the like.

[0130] Method 700 continues in block 704 by providing the electronic device with scene data. According to some embodiments, the device 100 or a component thereof (e.g., the image acquisition control module 1150 in) receives scene data. Fig. 11) Scene data (e.g. image data) by capturing two or more images of the scene from a first reference point / viewpoint (e.g. a camera position or field of view) using an image sensor or camera array.

[0131] Method 700 continues in block 706 by having the electronic device detect a plane based on the scene data. For example, the device detects a flat surface within the scene data (e.g., a floor, a wall, a tabletop, etc.). According to some embodiments, the device 100 detects or a component thereof (e.g., the plane-adaptation module 1158 in Fig. 11) at least one level by processing the scene data. For example, device 100 detects with reference to Fig. 4B a plane corresponding to the plate of table 415. In some embodiments, the entirety of the plane is located within the reticle element. In some embodiments, at least a portion of the plane is located within the reticle element. In some embodiments, the plane is larger than the extent of the reticle element. In some embodiments, the plane is smaller than the extent of the reticle element. In some embodiments, in response to the detection of two or more planes, the device displays multiple reticle elements in the second appearance state at locations near the two or more detected planes.

[0132] Method 700 continues in block 708 by having the electronic device display the reticle element in a second appearance state, which indicates a detection of the plane. For example, the device 100 changes in Fig. 4C the reticle element from the first appearance state 410 to the second appearance state 410' in response to the detection of the plane associated with the table plate 415 near the reticle element. In some embodiments, the reticle element in the second appearance state is displayed as a transparent rectangle with opaque or flashing edges. In some embodiments, the reticle element in the second appearance state is displayed as a partially transparent rectangle with opaque or flashing edges. In some embodiments, while displayed in the second appearance state, the reticle element provides a visual indication that a plane has been detected that can be used as a virtual substrate for AR / VR objects.In some embodiments, while displayed in the second appearance state, the reticle element also provides a visual cue for the boundaries of the captured plane, which can be used as a virtual substrate for AR / VR objects.

[0133] In some embodiments, the device switches the reticle element from the first appearance state to the second appearance state, provided that the detected plane is located near the reticle element while it is displayed in the first appearance state. According to some embodiments, the detected plane is located near the reticle element when the detected plane is projected onto an image space associated with the scene data (e.g., a two-dimensional space associated with the device coordinates) and there is at least a predefined number of pixels of overlap between the reticle element and the detected plane. According to some embodiments, the detected plane is located near the reticle element when the reticle element is projected onto a real space associated with the scene (e.g., a two-dimensional space associated with the device coordinates).a three-dimensional space connected to the world coordinates) and at least a predefined number of pixels overlap between the reticle element and the captured plane.

[0134] In some embodiments, the device switches the reticle element from the first appearance state to the second appearance state according to a stipulation that the detected plane is within a threshold distance while displayed in the first appearance state. According to some embodiments, the detected plane is within a threshold distance of the reticle element when the detected plane is within a predetermined distance of the device.

[0135] In some embodiments, the device aligns the reticle element, as shown by block 712, with the orientation of the detected plane. For example, device 100 shows in Fig. 4D the reticle element in a third appearance state 410" by rotating and / or aligning the reticle element with the orientation of the detected plane connected to the table 415. In some embodiments, the reticle element aligns with the yaw, pitch, and / or roll of the detected plane.

[0136] In some embodiments, the device, as shown by block 714, enlarges the reticular element. For example, device 100 shows in Fig. 4E the reticle element in a fourth appearance state 410''' by increasing the area of ​​the reticle element in response to the detection of additional points associated with the plane, indicating that its extent is larger than previously detected. In some embodiments, the reticle element expands to the detected size of the plane. In some embodiments, the reticle element shrinks to the detected size of the plane. In some embodiments, while the reticle element is displayed in the second appearance state, the size of the reticle element changes dynamically as the size of the detected plane changes based on the detection of additional points.

[0137] For example, the device detects user input that corresponds to a change in one or more dimensions of the reticulate element, such as a clamping gesture, a release gesture, a tap and pull gesture, or the like. For example, show Fig. 4L-4M a sequence in which a dimension of the reticle element is measured in response to the detection of a tap and pull gesture of the reticle element in Fig. 4L is changed (e.g., the height of the reticle element is moved towards the front edge 444 of the table 415). In some embodiments, the user input modifies the size of the reticle element within the limits of the detected plane. Therefore, in some embodiments, the user input does not change the size of the reticle element beyond the extent of the detected plane.

[0138] Method 700 continues in block 716 with the electronic device capturing an initial user input that positions an augmented reality and / or virtual reality object (AR / VR object) within the scene. For example, the device 100 captures in Fig. 4F a contact 414 (e.g., a tap or selection gesture) at a location corresponding to the user-selectable AR / VR object 476f (e.g., the cuboid object) within the object selection interface 472. In some embodiments, the device displays the object selection interface (e.g., a pop-over or pull-down menu / area) in response to the selection of a predefined affordance (e.g., the AR / VR object menu affordance 404b in).

[0139] Method 700 continues in block 718 by having the electronic device display the AR / VR object within the scene in relation to the detected plane. For example, device 100 displays in Fig. 4G a cuboid AR / VR object 420 within the scene related to the captured plane in response to a capture of the user-selectable AR / VR object 476f in Fig. 4F. In some embodiments, the AR / VR object is displayed at the geometric center (e.g., the center of gravity) of the captured plane. In some embodiments, the device 100 removes the reticle element after the AR / VR object has been positioned within the scene relative to the captured plane. In some embodiments, the device 100 removes the reticle element to reduce obscuration and element clumping when displaying the AR / VR object. In some embodiments, the device applies a rotation (e.g., yaw, pitch, and / or roll) to the AR / VR object based on an orientation of the captured plane.

[0140] Method 700 continues in block 720 with the electronic device capturing a second user input that directs interaction with the AR / VR object. As an example, device 100 captures in Fig. 4G a reverse clamping gesture with contacts 416a and 416b on the cuboid AR / VR object 420. As a further example, the device 100 captures in Fig. 4H performs a tap and pull gesture with contact 418 on the cuboid AR / VR object 420. As a further example, the device 100 captures in Fig. 4I a two-finger counterclockwise rotation gesture with contacts 422a and 422b on the cuboid AR / VR object 420. As yet another example, the device 100 captures in Fig. 4I a one-finger tap gesture with contact 424 on the cuboid AR / VR object 420.

[0141] Method 700 continues in block 722 by having the electronic device modify the AR / VR object based on one or more properties of the second user input. As an example, show... Fig. 4H-4I a sequence in which the cuboid AR / VR object 420 reacts to the detection of a tap and drag gesture on the cuboid AR / VR object 420 in Fig. 4H moved closer to the front edge 423 of the table 415. In some embodiments, one or more of the properties of the second user input correspond to the input type (e.g., a voice command, a pinch gesture, a reverse pinch gesture, a tap and drag gesture, a swipe gesture, a one-finger tap gesture, a two-finger tap gesture, a double one-finger tap gesture, a double two-finger tap gesture, etc.), the input direction, the input size, the input speed, and / or the like.

[0142] In some embodiments, one or more properties correspond to an input type. For example, if the third user input is a tap-and-drag gesture, the device modifies the location of the AR / VR object relative to the detected plane. In some embodiments, if the tap-and-drag gesture moves the AR / VR object outside the detected plane, the device displays the AR / VR object on the nearest plane (e.g., the floor plane). In some embodiments, if the tap-and-drag gesture moves the AR / VR object outside the detected plane, the device maintains a display of the AR / VR object at an edge of the detected plane. In another example, if the third user input is a pinch gesture, the device modifies the size of the AR / VR object.In another example, the device displays a predefined animation with the AR / VR object or performs a predefined operation on the AR / VR object when the third user input corresponds to a predefined gesture. In some embodiments, the device displays a predefined animation with the AR / VR object or performs a predefined operation on the AR / VR object based on the device's distance relative to the AR / VR object. In some embodiments, the device maintains the perspective of the AR / VR object (e.g., increasing / decreasing its size, displaying a different angle of the AR / VR object, etc.) when the device's position relative to the AR / VR object changes.

[0143] In some embodiments, the device, as shown in block 724, rotates the AR / VR object. For example, show Fig. 4I-4J a sequence in which the orientation of the cuboid AR / VR object 420 in response to a detection of the counterclockwise rotation gesture on the cuboid AR / VR object 420 in Fig. 4I is changed.

[0144] In some embodiments, the device, as shown by block 726, changes the size of the AR / VR object. For example, show Fig. 4G-4H a sequence in which the size of the cuboid AR / VR object 420 in response to the detection of a reverse pinch gesture on the cuboid AR / VR object 420 in Fig. 4G is increasing.

[0145] In some embodiments, the device, as shown in block 728, triggers a behavior associated with the AR / VR object. In some embodiments, the behavior corresponds to a predefined animation or operation on the AR / VR object, such as stroking an AR / VR animal to make it bark or meow, tapping an AR / VR car to make it honk or rev its engine, tapping an AR / VR cube such as a hammer to make it split in two, tapping an AR / VR volcano to make it erupt, and / or the like. For example, Fig. 4J-4K a sequence in which the cuboid AR / VR object 420 reacts to the detection of a predefined interaction gesture on the cuboid AR / VR object 420 in Fig. 4J is divided into cuboid AR / VR objects 430a and 430b.

[0146] Fig. Figure 8 is a flowchart representation of a method 800 for generating a combined set of quadtrees for use as a virtual substrate according to some embodiments. In some embodiments (and as detailed by way of example below), the method 800 is performed by an electronic device (or a section thereof), such as the electronic device 100 in Fig. 1 or the device 300 in Fig. 3, which includes one or more processors and non-volatile memory. In some embodiments, the device optionally includes a display, an image sensor or camera arrangement, and one or more input devices (e.g., a touchscreen display, a touchpad, a mouse, a keyboard, physical buttons, a microphone, etc.). For example, the display and the one or more input devices are combined to form a touchscreen display. In this example, the electronic device corresponds to a smartphone or a tablet. In another example, the display and the one or more input devices are separate. In this example, the electronic device corresponds to a laptop or desktop computer.For example, the electronic device corresponds to a portable rake, a smartphone, a tablet, a laptop computer, a desktop computer, a kiosk, a set-top box (STB), a game console and / or the like.

[0147] In some embodiments, Method 800 is performed by processing logic, including hardware, firmware, software, or a suitable combination thereof. In some embodiments, Method 800 is performed by one or more processors executing code, programs, or instructions stored in a non-volatile, computer-readable storage medium (e.g., non-volatile memory). Some operations in Method 800 are optionally combined, and / or the order of some operations is optionally changed. In short, Method 800 includes: obtaining a plurality of sets of images for different reference points / viewpoints; synthesizing a 3D point cloud for each of the plurality of sets of images; creating planes using the respective 3D point clouds; and generating a set of quadtrees that denote a consolidated set of planes over the 3D point clouds.

[0148] Method 800 begins in block 802 by having the electronic device receive a multitude of sets of images for different reference points / viewing angles. For example, the device 100 or a component thereof (e.g., the image acquisition control module 1150 in Fig. 11), as in Fig. Figures 5A-5B show sets of images 504a, 504b, ..., 504n (e.g., image data) for reference points / viewing angles 502a, 502b, ..., 502n. In some embodiments, each set of images includes two or more images. In some embodiments, each set of images corresponds to different reference points / viewing angles (e.g., different camera positions or fields of view).

[0149] Method 800 continues in Block 804 by having the electronic device synthesize a three-dimensional (3D) point cloud for each of the plurality of sets of images. For example, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11), as in Fig. Figures 5A-5B show three-dimensional (3D) point clouds 506a, 506b, ..., 506n based on the sets of images 504a, 504b, ..., 504n for the reference points / viewing angles 502a, 502b, ..., 502n. In some embodiments, the device 100 synthesizes the 3D point clouds according to algorithms or techniques known in the field by identifying a set of points for each set of images and arranging these points in a 3D space.

[0150] Method 800 continues in block 806 by having the electronic device create planes using the respective 3D point clouds. For example, the device 100 or a component thereof (e.g., the plane adjustment module 1158 in) creates Fig. 11), as in Fig. Figures 5A-5B show a set of planes 520a, 520b, ..., 520n, each based on the 3D point clouds 506a, 506b, ..., 506n. In some embodiments, the device 100 creates (e.g., adapts) the planes to the 3D point clouds according to algorithms or techniques known in the field (e.g., least matching squares, principal component analysis, simultaneous localization and mapping (SLAM), etc.).

[0151] Method 800 continues in Block 808 by having the electronic device generate a set of quadtrees that identify a summarized set of planes over the 3D point clouds. For example, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11), as in Fig. Figure 5A shows a set of quadtrees 525 that represent a aggregated set of planes over the 3D point clouds 506a, 506b, ..., 506n over time. For example, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Figure 5A) generates quadtrees 525 or a component thereof (e.g., the quadtree generation module 1160). Fig. 11), as in Fig. Figure 5B shows a composite set of quadtrees 540 that characterize a composite set of planes over the 3D point clouds 506a, 506b, ..., 506n over time. In some embodiments, the device 100 generates the set of quadtrees according to algorithms or techniques known in the field. According to some embodiments, the method 800 is also applicable for generating octrees that characterize the scene, as a person skilled in the art will recognize.

[0152] In some embodiments, the device, as shown by block 810, generates a set of planes for each of the 3D point clouds and produces the aggregated set of planes by correlating the sets of planes. In some embodiments, as shown by block 812, the device generates the set of quadtrees based on the aggregated set of planes. For example, as shown in Fig. As shown in Figure 5A, the device 100 or a component thereof (e.g., the correlation module 1162 in Figure 5A) correlates. Fig. 11) the first set of planes 520a, the second set of planes 520b, ... and the nth set of planes 520n, to create a combined set of planes. Continuing with the example in Fig. 5A in turn generates the device 100 or a component thereof (e.g. the quadtree generation module 1160 in Fig. 11) the set of quadtrees 525 based on the summarized set of planes.

[0153] In some embodiments, the device, as shown by block 814, generates an intermediate set of quadtrees for each of the 3D point clouds based on associated planes. In some embodiments, as shown by block 816, the device generates the set of quadtrees by correlating the intermediate sets of quadtrees. For example, the device generates 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11), as in Fig. Figure 5B shows a set of quadtrees 530a, 530b, ..., 530n, each based on the sets of planes 520a, 520n, ..., 520n and / or the 3D point clouds 506a, 506b, ..., 506n. Continuing with the example in Fig. 5B in turn correlates the device 100 or a component thereof (e.g. the correlation module 1162 in Fig. 11) the sets of quadtrees 530a, 530b, ..., 530n to generate a combined set of quadtrees 540.

[0154] Fig. Figure 9 is a flowchart representation of a method 900 for generating a set of quadtrees for use as a virtual substrate according to some embodiments. In some embodiments (and as detailed by way of example below), the method 900 is implemented by an electronic device (or a section thereof), such as the electronic device 100 in Fig. 1 or the device 300 in Fig. 3, carried out, which includes one or more processors, non-volatile memory, an optional image sensor or camera arrangement, an optional display, and one or more optional input devices. For example, the electronic device corresponds to a portable rake, a smartphone, a tablet, a laptop computer, a desktop computer, a kiosk, a set-top box (STB), a game console, and / or the like.

[0155] In some embodiments, Method 900 is performed by processing logic, including hardware, firmware, software, or a suitable combination thereof. In some embodiments, Method 900 is performed by one or more processors executing code, programs, or instructions stored in a non-volatile, computer-readable storage medium (e.g., non-volatile memory). Some operations in Method 900 are optionally combined, and / or the order of some operations is optionally changed. In short, Method 900 includes: generating a plurality of sets of planes for different reference points / viewpoints; correlating the sets of planes to obtain a aggregated set of planes over time; and generating a set of quadtrees based on the aggregated set of planes.

[0156] Method 900 begins in block 902 by having the electronic device obtain a three-dimensional (3D) point cloud based on a set of images for a reference point X (e.g., a time duration or camera position). As an example, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in Fig. 11) in Fig. 5A an nth 3D point cloud 506n based on the nth set of images 504n referenced to an nth reference point / viewpoint 502n (e.g., reference point X). For example, the device 100 synthesizes the nth 3D point cloud 506n by identifying points within the nth set of images 504n and arranging the points with respect to world coordinates by transforming the position of the points in an image space associated with the device coordinates into world coordinates according to algorithms or techniques known in the field.

[0157] Method 900 continues in block 904 by having the electronic device adapt planes to the 3D point cloud for the reference point X. As an example, the device 100 creates a component of this (e.g., the plane adaptation module 1158 in Fig. 11) in Fig. 5A creates an nth set of planes 520n based on the nth 3D point cloud 506n. For example, the device 100 creates the nth set of planes 520n by fitting planes to the nth 3D point cloud 506n according to algorithms or techniques known in the field (e.g. least matching squares, principal component analysis, simultaneous localization and mapping (SLAM), etc.).

[0158] Method 900 continues in block 906 by having the electronic device obtain a set of time-accumulated planes by summarizing, expanding, and / or correcting planes that are fitted to the 3D point clouds for reference points X, X-1, X-2, ..., XN. As an example, the device 100, or a component thereof (e.g., the correlation module 1162 in [reference missing]), correlates Fig. 11) in Fig. 5A correlates the first set of planes 520a (e.g., associated with reference point X-2), the second set of planes 520b (e.g., associated with reference point X-1), ... and the nth set of planes 520n (e.g., associated with reference point X) to generate a combined set of planes. For example, correlating the sets of planes fitted to the 3D point clouds for reference points X, X-1, X-2, ..., XN includes enlarging planes, combining planes, and / or correcting the size or orientation of planes over the time dimension based on identified similarities and / or differences in the position, size, and / or orientation of the planes fitted to the 3D point clouds for reference points X, X-1, X-2, ..., XN. In some embodiments, correlating the sets of planes corrects dynamic planes whose sizes change over time.In some embodiments, the correlation of sets of planes enlarges planes when additional associated points are detected over time. In some embodiments, the correlation of sets of planes combines planes when it is determined that two or more planes become part of the same plane over time.

[0159] Method 900 continues in Block 908 with the electronic device generating the set of quadtrees based on the set of time-accumulated levels. As an example, the device 100 or a component thereof (e.g., the quadtree generation module 1160 in Fig. 11) in Fig. 5A generates the set of quadtrees 525 based on the aggregated set of planes from block 906. For example, device 100 generates the set of quadtrees 525 according to algorithms or techniques known in the field.

[0160] Fig. Figure 10 is a flowchart representation of a method 1000 for generating a combined set of quadtrees according to some embodiments. In some embodiments (and as detailed by way of example below), the method 1000 is implemented by an electronic device (or a section thereof), such as the electronic device 100 in Figure 10. Fig. 1 or the device 300 in Fig. 3, carried out, which includes one or more processors, non-volatile memory, an optional image sensor or camera arrangement, an optional display, and one or more optional input devices. For example, the electronic device corresponds to a portable rake, a smartphone, a tablet, a laptop computer, a desktop computer, a kiosk, a set-top box (STB), a game console, and / or the like.

[0161] In some embodiments, Method 1000 is performed by processing logic, including hardware, firmware, software, or a suitable combination thereof. In some embodiments, Method 1000 is performed by one or more processors executing code, programs, or instructions stored in a non-volatile, computer-readable storage medium (e.g., non-volatile memory). Some operations in Method 1000 are optionally combined, and / or the order of some operations is optionally changed. In short, Method 1000 includes: generating sets of quadtrees for different reference points / viewpoints; and correlating the sets of quadtrees to obtain a composite set of quadtrees over time.

[0162] Method 1000 begins in block 1002 by having the electronic device obtain a three-dimensional (3D) point cloud based on a set of images for a reference point X (e.g., a time duration or camera position). As an example, the device 100 or a component thereof (e.g., the point cloud synthesis module 1156 in) synthesizes Fig. 11) in Fig. 5B an nth 3D point cloud 506n based on the nth set of images 504n referenced to an nth reference point / viewpoint 502n (e.g., reference point X). For example, the device 100 synthesizes the nth 3D point cloud 506n by identifying points within the nth set of images 504n and arranging the points with respect to world coordinates by transforming the position of the points in an image space associated with the device coordinates into world coordinates according to algorithms or techniques known in the field.

[0163] Method 1000 continues in block 1004 by having the electronic device adapt planes to the 3D point cloud for the reference point X. As an example, the device 100 creates a component of this (e.g., the plane adaptation module 1158 in Fig. 11) in Fig. 5B creates an nth set of planes 520n based on the nth 3D point cloud 506n (e.g., connected to the reference point X). For example, the device 100 creates the nth set of planes 520n by fitting planes to the nth 3D point cloud 506n according to algorithms or techniques known in the field (e.g., least-squares, principal component analysis, simultaneous localization and mapping (SLAM), etc.). As, for example, in Fig. As shown in Figure 6B, the device 100 creates (e.g., identifies) a multitude of planes 610a, 610b, 610c, and 610d (e.g., infinite planes) in a 3D coordinate space connected to the 3D point cloud, based on the clusters of points 605a, 605b, 605c, and 605d in Fig. 6A.

[0164] Method 1000 continues in block 1006 by having the electronic device triangulate points connected to the 3D point cloud in a two-dimensional (2D) coordinate space, which is linked to the set of images for the reference point X. As, for example, in Fig. As shown in Figure 6C, the device 100 triangulates points within the clusters of points 605a, 605b, 605c and 605d in a two-dimensional (2D) coordinate space associated with the multitude of images used to synthesize the 3D point cloud. Fig. 6A.

[0165] Method 1000 continues in block 1008 by having the electronic device remove triangles in the 2D coordinate space that do not correlate with the planes fitted to the 3D point cloud in order to obtain bounded triangulated regions for the reference point X. For example, in Fig. As shown in Figure 6D, the device removes 100 triangles in the 2D coordinate space that have points not corresponding to a plane based on the plurality of planes 610a, 610b, 610c and 610d in Figure 6D. Fig. 6B are connected to obtain a variety of constrained triangulated regions 620a, 620b, 620c and 620d.

[0166] In some embodiments, the electronic device performs the operations corresponding to blocks 1004, 1006, 1008 and 1010 sequentially according to the Fig. 10. In some embodiments, the electronic device performs the operations corresponding to blocks 1004, 1006, 1008 and 1010 sequentially in a sequence determined by the one shown in Fig. The 10 shown differs. In some embodiments, the electronic device performs the operations corresponding to blocks 1004, 1006, 1008 and 1010 in parallel.

[0167] Method 1000 continues in block 1010 by having the electronic device project the constrained triangulated regions onto the 3D coordinate space associated with the 3D point cloud. Method 1000 continues in block 1012 by having the electronic device generate a set of quadtrees based on the constrained triangulated regions for the reference point X. As an example, the device 100 generates one component thereof (e.g., the plane adjustment module 1158 in Fig. 11) in Fig. 5B generates an nth set of quadtrees 530n (e.g., connected to the reference point X) based on the nth set of planes 520n and / or the nth 3D point cloud 506n. For example, the device 100 generates an nth set of quadtrees 530n according to algorithms or techniques known in the field. As, for example, in Fig. As shown in Figure 6E, the device 100 projects the plurality of constrained triangulated regions 620a, 620b, 620c and 620d back into the 3D coordinate space and squares the plurality of constrained triangulated regions 620a, 620b, 620c and 620d to obtain quadtrees 630a, 630b, 630c and 630d. Fig. 6E the quadtrees 630a, 630b, 630c and 630d are enclosed by bounding frames based on their extent.

[0168] Method 1000 continues in block 1014 by having the electronic device summarize, expand, and / or correct sections of the set of quadtrees for reference point X based on the sets of quadtrees for reference points X-1, X-2, ..., XN. As an example, the device 100, or a component thereof (e.g., the correlation module 1162 in [reference 1000]), correlates [reference 1000] Fig. 11) in Fig. 5B the first set of quadtrees 530a (e.g. connected to reference point X-2), the second set of quadtrees 530b (e.g. connected to reference point X-1), ... and the nth set of quadtrees 530n (e.g. connected to reference point X) to generate a combined set of quadtrees 540.

[0169] For example, correlating sets of quadtrees for reference points X, X-1, X-2, ..., XN includes quadtree levels, merging quadtrees, and / or correcting the size or orientation of quadtrees over the time dimension based on identified similarities and / or differences in the position, size, and / or orientation of the sets of quadtrees for reference points X, X-1, X-2, ..., XN. In some embodiments, correlating sets of quadtrees corrects quadtrees associated with dynamic levels whose sizes change over time. In some embodiments, correlating sets of quadtrees enlarges quadtrees when additional associated points are detected over time. In some embodiments, correlating sets of quadtrees merges quadtrees when it is determined that two or more quadtrees become part of the same quadtree over time.In some embodiments, each quadtree in the aggregated set of quadtrees provides a display of one of: a substantially horizontal plane, a substantially vertical plane, or a plane angled according to one or more of three degrees of freedom. In some embodiments, each of the aggregated set of quadtrees corresponds to a virtual substrate.

[0170] Fig. Figure 11 is a block diagram of a computing device 1100 according to some embodiments. In some embodiments, the computing device 1100 corresponds to at least one section of the device 100 in Figure 11. Fig. 1 or the device 300 in Fig. 3 and performs one or more of the functionalities described above. While certain features are illustrated, the person skilled in the art will recognize from the present disclosure that various other features have not been illustrated for the sake of brevity, so as not to obscure more relevant aspects of the embodiments disclosed herein. For example, as a non-limiting example, the computing device 1100 in some embodiments includes one or more processing units (CPUs) 1102 (e.g., processors), one or more input / output interfaces (I / O interfaces) 1103 (e.g., network interfaces, input devices, output devices, and / or sensor interfaces), a memory 1110, a programming interface 1105, and one or more communication buses 1104 for connecting these and various other components.

[0171] In some embodiments, the communication buses 1104 include a circuit that connects and controls communications between system components. The memory 1110 includes high-speed random-access memory, such as DRAM, SRAM, DDR-RAM, or other random-access solid-state memory devices; and in some embodiments includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 1110 optionally includes one or more storage devices located remotely from the CPU(s) 1102. The memory 1110 comprises a non-volatile, computer-readable storage medium.Furthermore, in some embodiments, the memory 1110 or the non-volatile, computer-readable storage medium of the memory 1110 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 1120, an image acquisition control module 1150, an image processing module 1152, a coordinate transformation module 1154, a point cloud synthesis module 1156, a plane adjustment module 1158, a quadtree generation module 1160, a correlation module 1162, and an augmented reality and / or virtual reality handling module (AR / VR handling module) 1164. In some embodiments, one or more instructions are included in a combination of logic and non-volatile memory. The operating system 1120 includes procedures for handling various basic system services and for performing hardware-dependent tasks.

[0172] In some embodiments, the image acquisition control module 1150 is configured to control the functionality of an image sensor or camera array for acquiring images or obtaining image data. For this purpose, the image acquisition control module 1150 includes a set of instructions 1151a and heuristics and metadata 1151b.

[0173] In some embodiments, the image processing module 1152 is configured to preprocess raw image data from the image sensor or camera array (e.g., convert RAW image data into RGB or YCbCr image data). For this purpose, the image processing module 1152 includes a set of instructions 1153a and heuristics and metadata 1153b.

[0174] In some embodiments, the coordinate transformation module 1154 is configured to maintain world coordinates and device coordinates. In some embodiments, the coordinate transformation module 1154 is also configured to transform between coordinate sets. For this purpose, the coordinate transformation module 1154 includes a set of instructions 1155a and heuristics and metadata 1155b.

[0175] In some embodiments, the point cloud synthesis module 1156 is configured to synthesize a three-dimensional (3D) point cloud for a reference point / viewing angle (e.g., camera position) based on two or more images. For this purpose, the point cloud synthesis module 1156 includes a set of instructions 1157a and heuristics and metadata 1157b.

[0176] In some embodiments, the level fitting module 1158 is configured to create (or adapt) a set of levels for a given 3D point cloud (e.g., using least-squares matching, principal component analysis, simultaneous localization and mapping (SLAM), etc.). For this purpose, the level fitting module 1158 includes a set of instructions 1159a and heuristics and metadata 1159b.

[0177] In some embodiments, the quadtree generation module 1160 is configured to generate a set of quadtrees for a given 3D point cloud. For this purpose, the quadtree generation module 1160 includes a set of instructions 1161a and heuristics and metadata 1161b.

[0178] In some embodiments, the correlation module 1162 is configured to correlate sets of quadtrees for different reference points / viewpoints over a temporal dimension to obtain a combined set of quadtrees. In some embodiments, the correlation module 1162 is also configured to correlate sets of planes for different reference points / viewpoints over a temporal dimension to obtain a combined set of planes. For this purpose, the correlation module 1162 includes a set of instructions 1163a and heuristics and metadata 1163b.

[0179] In some embodiments, the AR / VR handling module 1164 is configured to display visual representations of captured layers and / or quadtrees. In other embodiments, the AR / VR handling module 1164 is configured to display and modify AR / VR objects. For this purpose, the AR / VR handling module 1164 includes a set of instructions 1165a and heuristics and metadata 1165b.

[0180] Although the image acquisition control module 1150, the image processing module 1152, the coordinate transformation module 1154, the point cloud synthesis module 1156, the plane adjustment module 1158, the quadtree generation module 1160, the correlation module 1162, and the AR / VR handling module 1164 are illustrated as being located on a single computing device 1100, it is understood that in other embodiments any combination of the image acquisition control module 1150, the image processing module 1152, the coordinate transformation module 1154, the point cloud synthesis module 1156, the plane adjustment module 1158, the quadtree generation module 1160, the correlation module 1162, and the AR / VR handling module 1164 may be located in different configurations in separate computing devices may be located there.For example, in some embodiments, each of the image acquisition control module 1150, image processing module 1152, coordinate transformation module 1154, point cloud synthesis module 1156, plane adjustment module 1158, quadtree generation module 1160, correlation module 1162 and AR / VR handling module 1164 is located on a separate computing device or in the cloud.

[0181] Furthermore, Fig. 11 is intended more as a functional description of the various features present in a particular implementation, as opposed to a structural scheme of the embodiments described herein. As the person skilled in the art will recognize, separately shown items could be combined and some elements separated. For example, some functional modules shown in Fig.The functions shown separately in Figure 11 may be implemented in a single module, and the various functions of individual function blocks could be implemented by one or more function blocks in different embodiments. The actual number of modules and the distribution of specific functions, and how the features are allocated among them, vary from one embodiment to another and may depend in part on the specific combination of hardware, software, and / or firmware chosen for a particular embodiment.

[0182] The present disclosure describes various features, none of which alone is responsible for the advantages described herein. It is understood that various features described herein may be combined, modified, or omitted, as is apparent to a person skilled in the art. Combinations and sub-combinations other than those specifically described herein are apparent to a person skilled in the art and are intended to form part of this disclosure. Various methods are described herein in connection with various flowchart steps and / or phases. It is understood that in many cases certain steps and / or phases may be combined in such a way that several steps and / or phases shown in the flowcharts may be carried out in a single step and / or phase.Furthermore, certain steps and / or phases can be subdivided into additional subcomponents that must be performed separately. In some cases, the sequence of steps and / or phases can be rearranged, and certain steps and / or phases can be omitted entirely. Moreover, the procedures described herein are to be understood as open-ended, so that steps and / or phases in addition to those shown and described herein may also be performed.

[0183] Some or all of the procedures and tasks described herein can be performed and fully automated by a computer system. In some cases, the computer system may include several different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and cooperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in memory or other non-volatile, computer-readable storage medium or device. The various functions disclosed herein may be implemented in such program instructions, although alternatively, some or all of the disclosed functions may be implemented in an application-specific circuit (e.g., a microcontroller, a microcontroller, or a microcontroller).The computer system may be implemented using ASICs, FPGAs, or GP-GPUs. Where the computer system includes multiple computing devices, these devices may, but need not, be arranged together. The results of the disclosed methods and tasks can be persistently stored by transforming physical storage devices, such as solid-state memory chips and / or magnetic disks, into another state.

[0184] The disclosure is not intended to be limited to the embodiments shown herein. Various modifications to the embodiments described herein are readily apparent to the person skilled in the art, and the general principles defined herein can be applied to other embodiments. The teachings of the invention provided herein can be applied to other methods and systems and are not limited to those described above, and elements and actions of the various embodiments described above can be combined to provide further embodiments. Accordingly, the novel methods and systems described herein can be implemented in a variety of other forms; furthermore, various omissions, substitutions, and modifications can be made to the form of the methods and systems described herein.

Claims

[1] Procedure, encompassing: on an electronic device comprising one or more processors, non-volatile memory, an image sensor, a display and one or more input devices: Displays, on the display, of a reticle element in a first appearance state superimposed in a media capture preview of objects in a field of view of the image sensor, wherein the media capture preview changes as the objects in the field of view of the image sensor change; Capturing a layer in the media capture preview; and in response to a capture of the plane, display, on the display, the reticle element is superimposed in a second appearance state in the media capture preview, wherein the reticle element corresponds to a specification of a section of the extent of the plane while it is displayed in the second appearance state. [2] Method according to claim 1, wherein the detected plane is located near the reticle element. [3] Method according to one of claims 1 to 2, wherein the detected plane is located within a threshold distance of the reticle element. [4] Method according to any one of claims 1 to 3, further comprising: Displaying a transition animation from the first appearance state of the reticle element to the second appearance state of the reticle element, where the transition animation corresponds to adjusting an orientation of the reticle element so that it matches an orientation of the captured plane. [5] Method according to any one of claims 1 to 4, further comprising: While the reticle element is displayed in the second appearance state, modify the size of the reticle element so that it corresponds to the size of the captured layer. [6] Method according to any one of claims 1 to 5, further comprising: While the reticle element is displayed in the second appearance state, capture, via one or more input devices, a first user input corresponding to an adjustment of the reticle element's size; and in response to the detection of the first user input, modifying the size of the reticle element based on the detected level and at least one of: a direction of the first user input, a size of the first user input, or a speed of the first user input. [7] Method according to any one of claims 1 to 6, further comprising: While the reticule element is displayed in the second appearance state, capture, via one or more input devices, a second user input corresponding to selecting an augmented or virtual reality object from an object selection interface; and in response to the capture of the second user input, displays on the display of the selected augmented or virtual reality object are overlaid in the media capture preview relative to the captured layer. [8] Method according to claim 7, wherein displaying the selected augmented or virtual reality object overlaid in the media capture preview comprises: Applying a rotation to the selected augmented or virtual reality object based on a plane orientation. [9] Method according to any one of claims 7 to 8, further comprising: In response to the capture of the second user input, the display of the reticle element overlaid in the media capture preview is stopped. [10] Method according to any one of claims 7 to 9, further comprising: Capturing, via one or more input devices, a third user input that corresponds to interacting with the augmented or virtual reality object; and in response to the detection of third-party user input, modifying the augmented or virtual reality object based at least partially on one or more properties of the third-party user input. [11] Electronic device comprising: an advertisement; an image sensor; one or more input devices; one or more processors; non-volatile memory; and one or more programs, wherein the one or more programs are stored in non-volatile memory and configured to be executed by the one or more processors, wherein the one or more programs contain instructions for: Displays, on the display, of a reticle element in a first appearance state superimposed in a media capture preview of objects in a field of view of the image sensor, wherein the media capture preview changes as the objects in the field of view of the image sensor change; Capturing a layer in the media capture preview; and in response to a capture of the plane, display, on the display, the reticle element is superimposed in a second appearance state in the media capture preview, wherein the reticle element corresponds to a specification of a section of the extent of the plane while it is displayed in the second appearance state. [12] Electronic device comprising: an advertisement; an image sensor; one or more input devices; one or more processors; non-volatile memory; a means of displaying, on the display, a reticule element in a first appearance state superimposed in a media capture preview of objects in a field of view of the image sensor, wherein the media capture preview changes as the objects in the field of view of the image sensor change; a means of capturing a layer in the media capture preview; and a means, in response to a capture of the plane, of displaying, on the display, the reticle element in a second appearance state superimposed in the media capture preview, wherein the reticle element corresponds to an indication of a section of the extent of the plane while it is displayed in the second appearance state.

Images