Method and apparatus for plot-based placement of virtual objects

Abstract

The subject of the invention is a method and apparatus for plot-based placement of virtual objects. In some implementations, a method of performing plot-based placement of a computer-generated graphical object at a device including one or more cameras and a non-transitory memory coupled to one or more processors. The method includes obtaining an input involving a content creation interface (e.g., a plotting board), where the input corresponds to a drawing of a candidate object, and where the content creation interface facilitates creation of a computer-generated graphical object renderable with the device. The method further includes obtaining a three-dimensional (3D) model using an input of a picture corresponding to the candidate object; generating a computer-generated graphical object using the acquired 3D model; and causing the computer-generated graphical object to be presented with images acquired with one or more cameras of the device.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 047823, international application date August 25, 2020, application number 202080028194.X, and invention name “Method and device for drawing-based placement of virtual objects”, which entered the Chinese national phase on October 12, 2021. Technical Field

[0002] The present disclosure relates generally to computer graphics, and in particular, to systems, methods, and apparatus for drawing-based placement of computer-generated graphical objects. Background Art

[0003] In some cases, users can populate their computer-generated rooms by selecting virtual objects from a pre-existing library. However, this limits the customizability of the computer-generated rooms. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0005] Figure 1 is a block diagram of an exemplary operational architecture according to some specific implementations.

[0006] Figure 2 is a block diagram of an example controller according to some implementations.

[0007] Figure 3 is a block diagram of an exemplary electronic device according to some implementations.

[0008] Figure 4A and Figure 4B A first computer-generated graphics rendering scene is shown according to some implementations.

[0009] Figure 5A and Figure 5B A second computer-generated graphics rendering scene is shown according to some implementations.

[0010] Fig. 6A and Figure 6B A flowchart representation of a method for drawing-based placement of computer-generated graphical objects according to some implementations is shown.

[0011] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0012] Various embodiments disclosed herein include devices, systems, and methods for placing computer-generated graphical objects (sometimes also referred to as "virtual objects," "graphic objects," or "ER objects") into a computer-generated graphical scenery (sometimes also referred to as "virtual scenery," "graphic scenery," or "ER scenery") based on a drawing. Because these objects and scenery are provided using electronic devices such as tablets, smart phones, and computers, they are sometimes also referred to as computer-generated objects or computer-generated scenery. According to some embodiments, the method is performed at a device including one or more cameras and non-volatile memory coupled to one or more processors. The method includes obtaining input involving a content creation interface (e.g., a drawing tablet), wherein the input corresponds to a drawing of a candidate object, and wherein the content creation interface facilitates creation of a computer-generated graphical object that can be presented using the device. The method also includes: obtaining a three-dimensional (3D) model using the input corresponding to the drawing of the candidate object; generating a computer-generated graphical object using the obtained 3D model; and causing the computer-generated graphical object to be presented together with an image obtained using one or more cameras of the device.

[0013] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0014] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings illustrate only some example aspects of the present disclosure and, therefore, should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein. Various examples of electronic systems and techniques for using such systems in connection with various augmented reality technologies are described.

[0015] A physical set refers to a world that individuals can sense and / or interact with without the use of electronic systems. A physical set, such as a physical park, includes physical elements, such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with a physical set, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0016] In contrast to a physical set, an augmented reality (ER) set refers to a fully (or partially) computer-generated set that various people can sense and / or otherwise interact with using electronic systems. In ER, the movement of a person is monitored in part, and in response thereto, at least one property corresponding to at least one virtual object in the ER set is changed in a manner consistent with one or more laws of physics. For example, in response to the ER system detecting that the person is looking up, the ER system may adjust various audio and graphics presented to the person in a manner consistent with the manner in which such sounds and appearances would change in a physical set. Adjustments to properties of virtual objects in the ER set may also be made, for example, in response to representations of movement (e.g., voice commands).

[0017] A person may sense and / or interact with ER objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER scenery, a person may sense and / or interact with only audio objects.

[0018] Virtual reality (VR) is an example of ER. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0019] Mixed reality (MR) is another example of ER. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0020] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0021] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0022] AR scenery may also refer to an augmented scenery in which the representation of the physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery may be graphically modified (e.g., enlarged) so that the modified portion still represents the originally captured image (but not an exact copy thereof). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0023] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0024] Various systems allow people to sense and / or interact with the ER scenery. For example, a near-eye system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into a near-eye system. The near-eye system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or translucent display may also be included in the near-eye system. A translucent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eye. The display may also include an LED, an OLED, a silicon-based liquid crystal, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. The transparent or translucent display may, for example, selectively transition between a transparent / translucent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0025] Figure 1 1 is a block diagram of an exemplary operating architecture 100 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating architecture 100 includes an optional controller 110 and an electronic device 120 (e.g., a tablet computer, a mobile phone, a laptop computer, a wearable computing device, etc.).

[0026] In some implementations, the controller 110 is configured to manage and coordinate a computer-generated graphical experience 128 for a user 150 (also sometimes referred to herein as an "ER scene," "virtual scene," "graphic scene," or "computer-generated graphical scene") and zero or more other users. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device that is located locally or remotely relative to physical set 105. For example, controller 110 is a local server located within physical set 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside of physical set 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the functionality of controller 110 is provided by electronic device 120. As such, in some implementations, components of controller 110 are integrated into electronic device 120.

[0027] In some implementations, the electronic device 120 is configured to present audio and / or video content to the user 150. In some implementations, the electronic device 120 is configured to present a computer-generated graphics experience 128 to the user 150. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 The electronic device 120 is described in more detail.

[0028] According to some implementations, the electronic device 120 presents the computer-generated graphical experience 128 to the user 150 while the user 150 is physically present within the physical set 105, wherein the physical set 105 includes the table 107 within the field of view 111 of the electronic device 120. Thus, in some implementations, the user 150 holds the electronic device 120 in one or both of his / her hands. In some implementations, when presenting the computer-generated graphical experience 128, the electronic device 120 is configured to present the computer-generated graphical content (e.g., the computer-generated graphical cylinder 109) and enable video pass-through of the physical set 105 (e.g., including the table 107) on the display 122. For example, the electronic device 120 corresponds to a mobile phone, a tablet computer, a laptop computer, a wearable computing device, etc.

[0029] In some implementations, the display 122 corresponds to an additive display that enables optical see-through of the physical scenery 105 (including the table 107). For example, the display 122 corresponds to a transparent lens, and the electronic device 120 corresponds to a pair of glasses worn by the user 150. Thus, in some implementations, the electronic device 120 presents a user interface by projecting computer-generated graphical content (e.g., the computer-generated graphical cylinder 109) onto the additive display, which in turn is superimposed on the physical scenery 105 from the perspective of the user 150. In some implementations, the electronic device 120 presents a user interface by displaying computer-generated graphical content (e.g., the computer-generated graphical cylinder 109) on the additive display, which in turn is superimposed on the physical scenery 105 from the perspective of the user 150.

[0030] In some implementations, user 150 wears electronic device 120, such as a near-eye system. Thus, electronic device 120 includes one or more displays (e.g., a single display or one display per eye) provided to display computer-generated graphical content. For example, electronic device 120 encompasses a field of view of user 150. In such implementations, electronic device 120 presents computer-generated graphical experience 128 by displaying data corresponding to computer-generated graphical experience 128 on one or more displays or by projecting data corresponding to computer-generated graphical experience 128 onto a retina of user 150.

[0031] In some implementations, the electronic device 120 includes an integrated display (e.g., a built-in display) that displays the computer-generated graphical experience 128. In some implementations, the electronic device 120 includes a head-mounted housing. In various implementations, the head-mounted housing includes an attachment area to which another device having a display can be attached. For example, in some implementations, the electronic device 120 can be attached to the head-mounted housing. In various implementations, the head-mounted housing is shaped to form a receiver for receiving another device (e.g., the electronic device 120) that includes a display. For example, in some implementations, the electronic device 120 slides / snaps into the head-mounted housing or is otherwise attached to the head-mounted housing. In some implementations, the display of the device attached to the head-mounted housing presents (e.g., displays) the computer-generated graphical experience 128. In some implementations, the electronic device 120 is replaced with an ER room, enclosure, or room configured to present computer-generated graphical content, in which the user 150 does not wear the electronic device 120.

[0032] In some implementations, the controller 110 and / or the electronic device 120 causes the computer-generated graphical representation of the user 150 to move within the computer-generated graphical experience 128 based on movement information (e.g., body pose data, eye tracking data, hand tracking data, etc.) from the electronic device 120 and / or an optional remote input device within the physical set 105. In some implementations, the optional remote input device corresponds to fixed or movable sensing equipment (e.g., image sensor, depth sensor, infrared (IR) sensor, event camera, microphone, etc.) within the physical set 105. In some implementations, each remote input device is configured to collect / capture input data while the user 150 is physically within the physical set 105 and provide the input data to the controller 110 and / or the electronic device 120. In some implementations, the remote input device includes a microphone, and the input data includes audio data (e.g., voice sample) associated with the user 150. In some implementations, the remote input device includes an image sensor (e.g., camera), and the input data includes an image of the user 150. In some implementations, the input data represents the body posture of the user 150 at different times. In some implementations, the input data represents the head posture of the user 150 at different times. In some implementations, the input data represents hand tracking information associated with the hands of the user 150 at different times. In some implementations, the input data represents the velocity and / or acceleration of a body part of the user 150 (such as his / her hand). In some implementations, the input data indicates the joint position and / or joint orientation of the user 150. In some implementations, the remote input device includes a feedback device, such as a speaker, a light, etc.

[0033] Figure 2 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0034] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0035] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and an experience engine 240.

[0036] The operating system 230 includes procedures for handling various basic system services and for performing hardware-related tasks.

[0037] In some implementations, experience engine 240 is configured to manage and coordinate one or more computer-generated graphical experiences (also sometimes referred to herein as "ER sets," "virtual sets," "graphic sets," or "computer-generated graphical sets") for one or more users (e.g., a single computer-generated graphical experience for one or more users, or multiple computer-generated graphical experiences for respective groups of one or more users). To this end, in various implementations, experience engine 240 includes data acquirer 242, mapper and locator engine 244, content manager 246, interaction and manipulation engine 248, content creation engine 250, and data transmitter 262.

[0038] In some implementations, the data acquirer 242 is configured to acquire data (e.g., presentation data, input data, user interaction data, user input, sensor data, location data, etc.) from at least one of the I / O devices 206, the electronic device 120, and the optional remote input devices 170A and 170B of the controller 110. To this end, in various implementations, the data acquirer 242 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0039] In some implementations, the mapper and locator engine 244 is configured to map the physical environment 105 and track the orientation / position of at least the electronic device 120 relative to the physical environment 105. To this end, in various implementations, the mapper and locator engine 244 includes instructions and / or logic for such instructions as well as heuristics and metadata for the heuristics.

[0040] In some implementations, content manager 246 is configured to generate (i.e., render), manage, and modify computer-generated graphical scenery (also sometimes referred to as "virtual scenery," "graphic scenery," or "computer-generated graphical experience") that is presented to a user. To this end, in various implementations, content manager 246 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0041] In some implementations, the interaction and manipulation engine 248 is configured to interpret user interactions and / or modification inputs related to the computer-generated graphical scenery. To this end, in various implementations, the interaction and manipulation engine 248 includes instructions and / or logic components for these instructions as well as heuristics and metadata for the heuristics.

[0042] In some implementations, the content creation engine 250 is configured to obtain computer-generated graphical objects (sometimes also referred to as "virtual objects", "graphic objects", or "ER objects") for placement into a computer-generated graphical scene based on user input, wherein the user input corresponds to a drawing of a candidate object. To this end, in various implementations, the content creation engine 250 includes an input interpreter 252, an optional depth inference engine 254, a model obtainer 256, and an optional 3D model library 258.

[0043] In some implementations, the input interpreter 252 is configured to obtain and interpret user input related to the content creation interface (e.g., content creation input). According to some implementations, the content creation interface corresponds to a planar 2D interface. According to some implementations, the content creation interface corresponds to a volumetric 3D interface. According to some implementations, the user input corresponds to one or more stylus inputs, touch inputs, eye tracking inputs, finger / hand tracking inputs, etc. within the content creation interface. To this end, in various implementations, the input interpreter 252 includes instructions and / or logic for these instructions as well as heuristics and metadata for the heuristics.

[0044] In some implementations, the optional depth inference engine 254 is configured to infer depth information (e.g., a depth map or mesh) corresponding to candidate objects in the drawing based on photogrammetry techniques, etc. To this end, in various implementations, the depth inference engine 254 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0045] In some implementations, the model acquirer 256 is configured to acquire a 3D model based on a user input corresponding to a drawing of a candidate object. According to some implementations, acquiring the 3D model includes matching the drawing of the candidate object to a pre-existing 3D model in the 3D model library 258. According to some implementations, the 3D model library 258 is stored locally or remotely relative to the controller 110. According to some implementations, the 3D model library 258 stores a plurality of 3D models. According to some implementations, acquiring the 3D model includes generating the 3D model based on the drawing of the candidate object and the depth information from the depth inference engine 254. In some implementations, the model acquirer 256 is also configured to generate a computer-generated graphics object using the acquired 3D model. For example, the model acquirer 256 generates a computer-generated graphics object by applying a texture or UV map to a mesh (e.g., the acquired 3D model). To this end, in various implementations, the model acquirer 256 includes instructions and / or logic components for these instructions as well as heuristics and metadata for the heuristics.

[0046] In some implementations, the data transmitter 262 is configured to transmit data (e.g., presentation data such as rendered image frames associated with a computer-generated graphic scene, position data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transmitter 262 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0047] Although the data acquirer 242, the marker and locator engine 244, the content manager 246, the interaction and manipulation engine 248, the content creation engine 250, and the data transmitter 262 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquirer 242, the marker and locator engine 244, the content manager 246, the interaction and manipulation engine 248, the content creation engine 250, and the data transmitter 262 may be located in separate computing devices.

[0048] In some implementations, the functions and / or components of the controller 110 are similar to those described below. Figure 3 The electronic device 120 shown in FIG. 1 is combined with or provided by the electronic device 120 shown in FIG. Figure 2 It serves more as a functional description of various features present in a particular implementation than as a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0049] Figure 31 is a block diagram of an example of an electronic device 120 (e.g., a mobile phone, a tablet computer, a laptop computer, a wearable computing device, etc.) according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0050] In some implementations, the one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetometer, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, a heating and / or cooling unit, a skin shear engine, one or more depth sensors (e.g., structured light, time of flight, etc.), an eye tracking engine, etc.

[0051] In some implementations, one or more displays 312 are configured to present computer-generated graphic scenery to the user. In some implementations, one or more displays 312 are also configured to present planar video content to the user (e.g., two-dimensional or "planar" AVI, FLV, WMV, MOV, MP4, etc. files associated with a TV series or movie, or real video pass-through of the physical environment 105). In some implementations, one or more displays 312 correspond to a touch screen display. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, the electronic device 120 includes a single display. As another example, the electronic device 120 includes a display for each eye of the user. In some implementations, the one or more displays 312 can present AR and VR content. In some implementations, the one or more displays 312 can present AR or VR content.

[0052] In some implementations, one or more optional internal and / or external facing image sensors 314 correspond to one or more RGB cameras (e.g., having a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), IR image sensors, event-based cameras, etc.

[0053] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and a rendering engine 340.

[0054] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the rendering engine 340 is configured to present computer-generated graphical content to a user via one or more displays 312. For this purpose, in various implementations, the rendering engine 340 includes a data acquirer 342, a renderer 344, an interaction processor 346, and a data transmitter 350.

[0055] In some implementations, the data acquirer 342 is configured to acquire data (e.g., presentation data, such as rendered image frames associated with a computer-generated graphic scene, input data, user interaction data, user input, sensor data, location data, etc.) from at least one of the I / O devices and sensors 306, the controller 110, and the remote input devices 170A and 170B of the electronic device 120. To this end, in various implementations, the data acquirer 342 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0056] In some implementations, the renderer 344 is configured to present and update computer-generated graphics content (e.g., rendered image frames associated with a computer-generated graphics scene) via the one or more displays 312. To this end, in various implementations, the renderer 344 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0057] In some implementations, the interaction processor 346 is configured to detect user interactions with the presented computer-generated graphical content. In some implementations, the interaction processor 346 is configured to detect user inputs (e.g., content creation inputs) related to the content creation interface. According to some implementations, the content creation interface corresponds to a planar 2D interface. According to some implementations, the content creation interface corresponds to a volumetric 3D interface. According to some implementations, the user input corresponds to one or more stylus inputs, touch inputs, eye tracking inputs, finger / hand tracking inputs, etc. within the content creation interface. To this end, in various implementations, the interaction processor 346 includes instructions and / or logic components for these instructions as well as heuristics and metadata for the heuristics.

[0058] In some implementations, the data transmitter 350 is configured to transmit data (e.g., presentation data, position data, user interaction data, user input, etc.) to at least the controller 110. To this end, in various implementations, the data transmitter 350 includes instructions and / or logic for such instructions as well as heuristics and metadata for such heuristics.

[0059] Although the data acquirer 342, renderer 344, interaction processor 346, and data transmitter 350 are illustrated as residing on a single device (e.g., electronic device 120), it should be understood that in other embodiments, any combination of the data acquirer 342, renderer 344, interaction processor 346, and data transmitter 350 may be located in separate computing devices.

[0060] also, Figure 3 It serves more as a functional description of various features present in a particular implementation rather than as a structural illustration of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0061] Figure 4A and Figure 4B A first computer generated graphical rendering scene according to some implementations is shown. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to not obscure more relevant aspects of the exemplary implementations disclosed herein. Figure 4A and Figure 4B A sequence of instances 400 and 450 of a first computer generated graphics rendering scene are shown, respectively.

[0062] like Figure 4A As shown, an instance 400 of a first computer-generated graphical presentation scene associated with time T1 includes a physical set 105 and a computer-generated graphical set 425 displayed on a display 122 of an electronic device 120. When a user 150 is physically present within the physical set 105, the electronic device 120 presents the computer-generated graphical set 425 to the user 150, the physical set including the table 107 within the field of view 111 of the outward-facing image sensor of the electronic device 120. Thus, in some implementations, the user 150 holds the electronic device 120 in his / her hand, similar to Figure 1 in the operating environment 100.

[0063] In other words, in some implementations, the electronic device 120 is configured to present computer-generated graphical content and enable optical or video pass-through of at least a portion of the physical scenery 105 (e.g., including the table 107) on the display 122. For example, the electronic device 120 corresponds to a mobile phone, a tablet computer, a laptop computer, a wearable computing device, etc.

[0064] like Figure 4A As shown, the electronic device 120 also displays a content creation interface 410 and a tool panel 420 on the display 122. According to some implementations, the content creation interface 410 is configured to detect / receive user input, such as a drawing or stroke of a stylus 401 held by the user 150 or a touch / finger input from the user 150. According to some implementations, the tool panel 420 includes selectable tools configured to retroactively and / or prospectively change one or more characteristics of the user input (such as line width, line color, line type, fill color, texture filler, etc.).

[0065] In some implementations, in response to detecting a user input corresponding to a drawing of a candidate object in the content creation interface 410, the electronic device 120 is configured to obtain a 3D model based on the drawing of the candidate object, and present the computer-generated graphical object in the computer-generated graphical scenery 425 based on the 3D model. Thus, based on the user drawing involving the content creation interface 410, the computer-generated graphical object is placed in the computer-generated graphical scenery 425. In some implementations, the following reference Fig. 6A and Figure 6B The method 600 in FIG. 1 describes the process for obtaining a 3D model in more detail. Figure 4B As shown, instance 450 of a first computer-generated graphics rendering scene associated with time T2 shows a computer-generated graphical object 475 (e.g., a 3D palm tree) displayed within computer-generated graphics scenery 425 in response to detecting user input 465 (e.g., a drawing of a palm tree) within content creation interface 410.

[0066] Figure 5A and Figure 5B A second computer generated graphical rendering scene according to some implementations is shown. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to not obscure more relevant aspects of the exemplary implementations disclosed herein. Figure 5A and Figure 5B Similar to and modified from Figure 4A and Figure 4B Therefore, similar reference numerals are used herein, and for the sake of brevity, only the differences will be discussed. Figure 5A and Figure 5B A sequence of instances 500 and 550 of a second computer generated graphics rendering scene are shown, respectively.

[0067] like Figure 5A As shown, the second computer-generated graphics presentation scene instance 500 associated with time T1 includes the physical scenery 105, the computer-generated graphics scenery 425 displayed on the display 122 of the electronic device 120, and the auxiliary electronic device 520. In some specific implementations, the electronic device 120 is configured to present the computer-generated graphics content and enable optical or video pass-through of at least a portion of the physical scenery 105 (e.g., including the table 107) on the display 122. Therefore, in some specific implementations, the user 150 holds the electronic device 120 in his / her hand, similar to Figure 1 The operating environment 100 in FIG. 100. For example, the electronic device 120 corresponds to a mobile phone, a tablet computer, a laptop computer, a wearable computing device, etc.

[0068] like Figure 5A As shown, the auxiliary electronic device 520 displays the content creation interface 410. According to some specific implementations, the content creation interface 410 is configured to detect / receive inputs, such as drawings or strokes of the stylus 401 held by the user 150 or touch inputs from the user 150. For example, the auxiliary electronic device 520 corresponds to a mobile phone, a tablet computer, a laptop computer, a wearable computing device, etc. According to some specific implementations, the auxiliary electronic device 520 is communicatively coupled to the electronic device 120 and / or the controller 110 via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0069] In some implementations, in response to detecting a user input corresponding to a drawing of a candidate object in the content creation interface 410, the electronic device 120 is configured to obtain a 3D model based on the drawing of the candidate object, and present the computer-generated graphical object in the computer-generated graphical scenery 425 based on the 3D model. Thus, based on the user drawing involving the content creation interface 410 associated with the auxiliary electronic device 520, the computer-generated graphical object is placed in the computer-generated graphical scenery 425 (presented by the electronic device 120). In some implementations, the following reference is made to Fig. 6A and Figure 6B The method 600 in FIG. 1 describes the process for obtaining a 3D model in more detail. Figure 5BAs shown, instance 550 of a second computer-generated graphics rendering scene associated with time T2 shows a computer-generated graphical object 475 (e.g., a 3D palm tree) displayed within a computer-generated graphics scenery 425 rendered by the electronic device 120 in response to detecting user input 465 (e.g., a drawing of a palm tree) within a content creation interface 410 associated with the auxiliary electronic device 520.

[0070] Fig. 6A and Figure 6B A flowchart representation of a method 600 for drawing-based placement of computer-generated graphical objects according to some implementations is shown. In various implementations, the method 600 is performed by a device having one or more cameras and non-transitory memory coupled to one or more processors (e.g., Figure 1 and Figure 2 A controller 110; Figure 1 and Figure 3 ; or a suitable combination thereof), or a component thereof. In some implementations, method 600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 600 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0071] As described above, in some cases, users can populate their computer-generated rooms by selecting virtual objects from a pre-existing library. However, this limits the customizability of the computer-generated room. Therefore, according to some specific implementations, in order to allow the user further customizability, a 3D model is acquired (e.g., matched against a library of computer-generated graphic objects or generated in real time) based on a drawing of a candidate object on a content creation interface (e.g., a drawing board), and the computer-generated graphic objects corresponding to the 3D model are placed into the computer-generated graphic scenery. In some specific implementations, the position of the drawing relative to the content creation interface defines the placement position of the computer-generated graphic objects within the computer-generated graphic scenery. In some specific implementations, the angle of the content creation interface defines the angle at which the computer-generated graphic objects are placed within the computer-generated graphic scenery.

[0072] As shown in block 6-1, method 600 includes obtaining input related to a content creation interface (e.g., a drawing tablet), wherein the input corresponds to a drawing of a candidate object, and wherein the content creation interface facilitates creation of a computer-generated graphical object that can be rendered using the device. In some implementations, the device or a component thereof (e.g., Figure 2 The input interpreter 252 in obtains and interprets input related to the content creation interface.

[0073] According to some implementations, the content creation interface corresponds to a flat two-dimensional (2D) content creation interface. According to some implementations, the input (e.g., content creation input) corresponds to one or more stylus inputs, touch inputs, eye tracking inputs, finger / hand tracking inputs, etc. within the content creation interface. For example, Figure 4A and Figure 4B The sequence shown in shows input detected by electronic device 120 from stylus 401. In some implementations, the content creation interface corresponds to a three-dimensional (3D) content creation interface.

[0074] In some implementations, method 600 includes generating tactile feedback when acquiring input corresponding to a drawing of a candidate object based on determining that the input is within a threshold distance from a boundary of the content creation interface. Figure 4A , when the input is near the edge of the content creation interface 410, the electronic device 120 generates tactile feedback.

[0075] In some implementations, method 600 includes causing display of a content creation interface in response to detecting an invocation command, wherein the input relates to the content creation interface. Figure 4A , the electronic device 120 causes the content creation interface 410 to be displayed simultaneously with the computer-generated graphic scenery 425 in response to an invocation command such as a hand gesture, a voice command, etc.

[0076] In some implementations, as shown in block 6-1a, method 600 includes displaying a content creation interface with an image captured using one or more cameras of the device. Thus, the device composites (e.g., video pass-through) computer-generated graphical content (including the content creation interface and any computer-generated graphical objects) with an image captured using one or more cameras of the device (e.g., an externally facing camera) to generate a computer-generated graphical scenery for presentation by the device. For example, the content creation interface is superimposed on the image captured using one or more cameras of the device and / or on the computer-generated graphical scenery.

[0077] In some implementations, as shown in block 6-1b, method 600 includes causing a content creation interface to be displayed on a second device. Figure 5A and Figure 5B The sequence shown in shows the content creation interface 410 displayed on the auxiliary electronic device 520 while the electronic device 120 presents the computer-generated graphic scenery 425.

[0078] In some implementations, as shown in block 6-1c, the content creation interface is displayed adjacent to the computer-generated graphic scenery. Figure 4A and Figure 4BThe sequence shown in shows a content creation interface 410 displayed simultaneously adjacent to a computer-generated graphical scenery 425.

[0079] As shown in block 6-2, method 600 includes obtaining a three-dimensional (3D) model using input corresponding to a drawing of a candidate object. In some implementations, a device or a component thereof (e.g., Figure 2 The model acquirer 256 in the embodiment acquires a 3D model based on the input corresponding to the drawing of the candidate object.

[0080] In some implementations, as shown in block 6-2a, obtaining the 3D model includes matching the drawing of the candidate object to a pre-existing 3D model in a 3D model library. In some implementations, the device or a component thereof (e.g., Figure 2 The model acquirer 256 in the 3D model library 258 acquires a 3D model by matching the drawing of the candidate object to a pre-existing 3D model in the 3D model library 258. For example, the model acquirer 256 acquires a 3D model from the 3D model library 258 that matches the drawing of the candidate object within a predefined confidence threshold.

[0081] In some embodiments, as shown in block 6-2b, obtaining the 3D model includes generating the 3D model based on a drawing of the candidate object. Figure 2 The model acquirer 256 in the apparatus acquires the 3D model by generating the 3D model in real time based on the drawing of the candidate object and (optionally) the depth information. According to some specific implementations, generating the 3D model includes inferring the depth information associated with the drawing of the candidate object based on photogrammetry techniques, etc. In some specific implementations, the device or its components (e.g., Figure 2 The depth inference engine 254 in the image may infer depth information (e.g., a depth map or mesh) corresponding to candidate objects in the drawing based on photogrammetry techniques, etc.

[0082] As shown in block 6-3, method 600 includes generating a computer-generated graphical object using the acquired 3D model. In some implementations, controller 110 or a component thereof (e.g., Figure 2 The model acquirer 256 in the method 600 generates a computer-generated graphics object using the acquired 3D model. In some specific implementations, the method 600 includes generating a computer-generated graphics object, including acquiring a mesh with a texture.

[0083] As indicated at block 6-4, method 600 includes causing a computer-generated graphical object to be presented along with an image acquired using one or more cameras of a device. Figure 4B, electronic device 120 displays computer-generated graphical object 475 (eg, a 3D palm tree) within computer-generated graphical scenery 425 in response to detecting user input 465 (eg, a drawing of a palm tree) within content creation interface 410 .

[0084] As an example, assuming that the functions of the controller 110 and the electronic device 120 are separated, the controller 110 or its components (eg, Figure 2 Continuing with this example, the controller 110 or a component thereof (e.g., Figure 2 The data transmitter 262 in the example transmits the rendered image frame to the electronic device 120. Finally, continuing with the example, the electronic device 120 or a component thereof (eg, Figure 3 The data acquirer 342 in the embodiment receives the rendered image frame, and the electronic device 120 or a component thereof (e.g., Figure 3 The renderer 344 in the display 312 displays the rendered image frames via one or more displays 312.

[0085] As another example, assuming that the functions of the controller 110 and the electronic device 120 are combined, the device or its components (e.g., Figure 2 The content manager 246 in the device renders image frames associated with a computer-generated graphics scene including computer-generated graphics objects, and the device or a component thereof (e.g., Figure 3 The renderer 344 in the display 312 displays the rendered image frames via one or more displays 312.

[0086] In some implementations, as shown in block 6-4a, method 600 includes determining a display location of a computer-generated graphical object using the obtained location of the input corresponding to the depiction of the candidate object, and wherein causing the computer-generated graphical object to be presented includes causing the computer-generated graphical object to be presented at the determined display location. In some implementations, the location of the input within the content creation interface determines a translation coordinate associated with placing the computer-generated graphical object into the computer-generated graphical scenery. In other words, the transformation maps the location of the input within the content creation interface to a placement location within the computer-generated graphical scenery. Thus, as an example reference Figure 5A If an input associated with a drawing of a candidate object is detected within an upper portion of the 3D content creation interface 410 relative to the y-axis, the device places the computer-generated graphical object into the computer-generated graphical scenery based on the position of the input, so that the computer-generated graphical object can float in the air when displayed within the computer-generated graphical scenery.

[0087] In some embodiments, as shown in block 6-4b, at least one rotational dimension of the computer-generated graphical object corresponds to the angle of the content creation interface. In some embodiments, one or more rotational dimensions (e.g., pitch, roll, and / or yaw) of placing the computer-generated graphical object into the computer-generated graphical scenery is based on the angle of the content creation interface. Thus, as an example, if the content creation interface is pitched at a 45° angle, the device places the computer-generated graphical object into the computer-generated graphical scenery at a 45° pitch. For example, a user can rotate and / or translate the content creation interface. In some embodiments, one or more rotational dimensions (e.g., pitch, roll, and / or yaw) of placing the computer-generated graphical object into the computer-generated graphical scenery is based on the angle of the input relative to the content creation interface.

[0088] In some implementations, as shown in block 6-5, method 600 includes: obtaining additional input related to the presented computer-generated graphic object; and modifying the presented computer-generated graphic object according to the additional input. In some implementations, as shown in block 6-5a, modifying the presented computer-generated graphic object includes scaling the computer-generated graphic object, rotating the computer-generated graphic object, translating the computer-generated graphic object, animating the computer-generated graphic object, etc. In some implementations, as shown in block 6-5b, modifying the presented computer-generated graphic object includes modifying at least one of the color, texture, shading, shading, shape, etc. of the computer-generated graphic object.

[0089] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0090] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0091] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0092] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device including one or more cameras and non-transitory memory coupled to one or more processors: Get input related to content creation areas within the user interface, wherein the input corresponds to a drawing of a candidate object, wherein the content creation area facilitates creation of computer-generated graphical objects capable of being rendered using the device; as well as wherein the user interface includes the content creation area and a presentation area for computer-generated content; as well as In response to obtaining the input related to the content creation area: presenting the drawing within the content creation area; obtaining a three-dimensional (3D) model using the input corresponding to the drawing of the candidate object; generating a computer-generated graphical object using the acquired 3D model; as well as The computer-generated graphical object is caused to be presented within the presentation area along with images acquired using the one or more cameras of the device while maintaining presentation of the drawing within the content creation area.

2. The method of claim 1, wherein the user interface further comprises a tool area having a plurality of selectable tools configured to change one or more characteristics of the drawing presented in the content creation area that are associated with the input.

3. The method of claim 1, wherein the user interface further comprises a tool area having a plurality of selectable tools configured to change one or more characteristics of a subsequent drawing associated with a subsequent user input. The method of claim 1 , wherein obtaining the 3D model comprises matching the depiction of the candidate object to a pre-existing 3D model in a 3D model library. The method of claim 1 , wherein acquiring the 3D model comprises generating the 3D model based on the drawing of the candidate object.

6. The method of claim 1, wherein the input corresponds to at least one of one or more stylus inputs, one or more touch inputs, one or more finger tracking inputs, one or more hand tracking inputs, or one or more eye tracking inputs involving the content creation interface.

7. The method of claim 1, wherein generating the computer-generated graphics object comprises acquiring a textured mesh.

8. The method according to claim 1, further comprising: A display position of the computer-generated graphical object is determined within the rendering area using the position of the input obtained corresponding to the drawing of the candidate object, and wherein causing the rendering of the computer-generated graphical object includes causing the computer-generated graphical object to be rendered within the rendering area at the determined display position.

9. The method of claim 1, wherein at least one rotational dimension of the computer-generated graphical object corresponds to an angle of the content creation interface.

10. The method according to claim 1, further comprising: Based on determining that the input is within a threshold distance from a boundary of the content creation interface, haptic feedback is generated when the input corresponding to the drawing of the candidate object is acquired.

11. The method according to claim 1, further comprising: obtaining additional input related to the computer-generated graphical object being rendered; as well as The rendered computer-generated graphical object is modified based on the additional input.

12. The method of claim 11, wherein modifying the computer-generated graphical object comprises at least one of scaling, rotating, translating, or animating the computer-generated graphical object.

13. The method of claim 11, wherein modifying the computer-generated graphical object comprises modifying at least one of a color, a texture, a shading, a shading, or a shape of the computer-generated graphical object.

14. A device comprising: one or more processors; non-transitory memory; one or more cameras; as well as one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: Get input related to content creation areas within the user interface, wherein the input corresponds to a drawing of a candidate object, wherein the content creation area facilitates creation of computer-generated graphical objects capable of being rendered using the device; as well as wherein the user interface includes the content creation area and a presentation area for computer-generated content; as well as In response to obtaining the input related to the content creation area: presenting the drawing within the content creation area; obtaining a three-dimensional (3D) model using the input corresponding to the drawing of the candidate object; generating a computer-generated graphical object using the acquired 3D model; as well as The computer-generated graphical object is caused to be presented within the presentation area along with images acquired using the one or more cameras of the device while maintaining presentation of the drawing within the content creation area.

15. The device of claim 14, wherein the user interface further comprises a tool area having a plurality of selectable tools configured to change one or more characteristics of the drawing presented in the content creation area that are associated with the input.

16. The device of claim 14, wherein the user interface further comprises a tool area having a plurality of selectable tools configured to change one or more characteristics of a subsequent drawing associated with a subsequent user input.

17. The device of claim 14, wherein the input corresponds to at least one of one or more stylus inputs, one or more touch inputs, one or more finger tracking inputs, one or more hand tracking inputs, or one or more eye tracking inputs involving the content creation interface.

18. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device having one or more cameras, cause the device to: Get input related to content creation areas within the user interface, wherein the input corresponds to a drawing of a candidate object, wherein the content creation area facilitates creation of computer-generated graphical objects capable of being rendered using the device; as well as wherein the user interface includes the content creation area and a presentation area for computer-generated content; as well as In response to obtaining the input related to the content creation area: presenting the drawing within the content creation area; obtaining a three-dimensional (3D) model using the input corresponding to the drawing of the candidate object; generating a computer-generated graphical object using the acquired 3D model; as well as The computer-generated graphical object is caused to be presented within the presentation area along with images acquired using the one or more cameras of the device while maintaining presentation of the drawing within the content creation area.

19. A non-volatile memory according to claim 18, wherein the user interface further comprises a tool area having a plurality of selectable tools, the plurality of selectable tools being configured to change one or more characteristics of the drawing presented in the content creation area that are associated with the input.

20. The non-transitory memory of claim 18, wherein the user interface further comprises a tool area having a plurality of selectable tools, the selectable tools configured to change one or more characteristics of a subsequent drawing associated with a subsequent user input.

21. A non-volatile memory according to claim 18, wherein the input corresponds to at least one of one or more stylus inputs, one or more touch inputs, one or more finger tracking inputs, one or more hand tracking inputs, or one or more eye tracking inputs involving the content creation interface.

22. A method comprising: At a device including a display, one or more cameras, and non-transitory memory coupled to one or more processors: causing a content creation area to be presented on a first portion of the display; Obtaining input related to a content creation region of a drawing corresponding to a candidate object; as well as In response to obtaining the input: presenting the drawing within the content creation area; Using the input to obtain a three-dimensional 3D model; generating a computer-generated graphical object using the 3D model; as well as The rendering of the computer-generated graphical object on the second portion of the display is caused to be superimposed on an image acquired using the one or more cameras of the device.

23. The method of claim 22, wherein the computer-generated graphical object is rendered within the content creation area simultaneously with the drawing.

24. The method of claim 22, wherein the content creation area is presented superimposed on the image acquired using the one or more cameras of the device.

25. The method of claim 22, wherein the content creation area is a two-dimensional content creation area.

26. The method of claim 22, wherein the content creation area is a three-dimensional content creation area.

27. The method of claim 22, wherein the input corresponds to at least one of one or more stylus inputs, one or more touch inputs, one or more finger tracking inputs, one or more hand tracking inputs, or one or more eye tracking inputs involving the content creation interface.

28. The method of claim 22, wherein obtaining the 3D model comprises matching the depiction of the candidate object to a pre-existing 3D model in a 3D model library.

29. The method of claim 22, wherein acquiring the 3D model comprises generating the 3D model based on the drawing of the candidate object.

30. The method of claim 22, wherein generating the computer-generated graphics object comprises acquiring a textured mesh.

31. The method of claim 22, further comprising: obtaining additional input related to rendering of the computer-generated graphical object; as well as The rendering of the computer-generated graphical object is modified based on the additional input.

32. The method of claim 31 , wherein modifying the computer-generated graphical object comprises at least one of scaling, rotating, translating, or animating the computer-generated graphical object.

33. The method of claim 31 , wherein modifying the computer-generated graphical object comprises modifying at least one of a color, a texture, a shading, a shading, or a shape of the computer-generated graphical object.

34. An apparatus comprising: monitor; one or more processors; non-transitory memory; one or more cameras; as well as one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: causing a content creation area to be presented on a first portion of the display; Obtaining input related to a content creation region of a drawing corresponding to a candidate object; as well as In response to obtaining the input: presenting the drawing within the content creation area; Using the input to obtain a three-dimensional 3D model; generating a computer-generated graphical object using the 3D model; as well as The rendering of the computer-generated graphical object on the second portion of the display is caused to be superimposed on an image acquired using the one or more cameras of the device.

35. The apparatus of claim 34, wherein the computer-generated graphical object is rendered within the content creation area simultaneously with the drawing.

36. The device of claim 34, wherein the content creation area is presented superimposed on the image acquired using the one or more cameras of the device.

37. The device of claim 34, wherein the content creation area is a two-dimensional content creation area.

38. The device of claim 34, wherein the content creation area is a three-dimensional content creation area.

39. The device of claim 34, wherein the one or more programs cause the device to generate the computer-generated graphics object by acquiring a mesh having a texture.

40. The device of claim 34, wherein the one or more programs further cause the device to: obtaining additional input related to rendering of the computer-generated graphical object; and The rendering of the computer-generated graphical object is modified based on the additional input.

41. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device having a display and one or more cameras, cause the device to: causing a content creation area to be presented on a first portion of the display; Obtaining input related to a content creation region of a drawing corresponding to a candidate object; as well as In response to obtaining the input: presenting the drawing within the content creation area; Using the input to obtain a three-dimensional 3D model; generating a computer-generated graphical object using the 3D model; as well as The rendering of the computer-generated graphical object on the second portion of the display is caused to be superimposed on an image acquired using the one or more cameras of the device.

Images