System and method for interactive three-dimensional preview

Abstract

The present disclosure relates to a system and method for interactive three-dimensional preview. Three-dimensional data can be synchronized between a first electronic device and a second electronic device. A content creation application can be run on the first electronic device and can utilize a data file describing a three-dimensional content item. A two-dimensional representation of the content item can be displayed on the first electronic device. A user can request a preview of the two-dimensional representation of the content item in three dimensions. The first electronic device can initiate a data transfer with the second electronic device. The three-dimensional data of the data file can be transferred from the content creation application of the first electronic device to a three-dimensional graphics rendering application at the second electronic device via a communication link. The three-dimensional graphics rendering application can generate a three-dimensional preview of the content item based on the received three-dimensional data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 226,723, filed on July 28, 2021, U.S. Patent Application No. 17 / 814,455, filed on July 22, 2022, and U.S. Patent Application No. 17 / 814,462, filed on July 22, 2022, the entire contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to systems and methods for generating previews of three-dimensional content, and more particularly to systems and methods for generating and updating previews of three-dimensional content using an application running on an electronic device. Background Art

[0004] Some computer graphics environments provide two-dimensional and / or three-dimensional environments in which at least some of the objects displayed for user observation are virtual and computer-generated. For example, in some use cases, a user can create or modify an XR environment, such as by editing, generating, or otherwise manipulating XR virtual objects using a content generation environment (such as a graphics editor or graphics editing interface running on a content creation application). In some embodiments, creation or modification of an XR environment, including content items (e.g., two-dimensional and / or three-dimensional objects) within the XR environment, may include generating previews of the content items and presenting the previews of the content items to the user at various intermediate stages of the content creation process. However, such previews of content items generated in two dimensions and presented to the user are limited by the two-dimensional display and graphics processing characteristics of the device on which the content creation application is running. Summary of the Invention

[0005] Some embodiments of the present disclosure relate to a first electronic device that communicates with a second electronic device, wherein the first electronic device and the second electronic device are configured to exchange data (e.g., extended reality (XR) content data) therebetween. In some embodiments, the first electronic device may be configured with a content creation application. The content creation application can be used to create XR content (also referred to herein as XR content items) and can be used to present a two-dimensional representation of three-dimensional XR content. In some embodiments, the second electronic device may be configured with a three-dimensional graphics rendering application. The three-dimensional graphics rendering application can be used to display a three-dimensional preview of three-dimensional XR content created using the content creation application at the first electronic device. The content creation application can form a communication link with the three-dimensional graphics rendering application so that the two-dimensional XR content displayed on the first electronic device can be presented as a three-dimensional preview on the second electronic device. In some embodiments, the communication link may include one or more intermediaries (e.g., one or more supporting applications) running on the first electronic device and / or the second electronic device. In some embodiments, the content creation application and the three-dimensional graphics rendering application can be provided within a single electronic device.

[0006] Some embodiments of the present disclosure relate to the synchronization of three-dimensional data between a first electronic device and a second electronic device (e.g., a computer and a head-mounted display, respectively). A content creation application may be run on the first electronic device and may utilize a data file describing a three-dimensional content item. The data file describing the content item may be launched in the content creation application using a communication link between the content creation application and an integrated design environment. In some embodiments, a two-dimensional representation of the content item may be displayed on the first electronic device. In some embodiments, the data file may be edited while running on the content creation application. In some embodiments, a user (e.g., using the first electronic device) may request a preview of a two-dimensional representation of the three-dimensional content item. The first electronic device may (e.g., in response to the request) initiate a data transfer with the second electronic device. In some embodiments, the three-dimensional data of the data file may be transferred to a first intermediary application running on the second electronic device. The first intermediary application may launch a second intermediary application that communicates with the first intermediary application via the communication link. The second intermediary application may use the communication link to transfer user code from the data file to a three-dimensional graphics rendering application. In some embodiments, the three-dimensional graphics rendering application may receive a request to generate content from a preview proxy application and generate a three-dimensional virtual object. The three-dimensional virtual object may be presented to the user as a three-dimensional representation or a two-dimensional representation. The 3D preview of the content item can be displayed on the second electronic device as a 3D content item rendered in a 3D XR environment. The 3D graphical rendering of the content item can be simultaneously presented as a 2D representation of the content item displayed on the first electronic device (e.g., having a 2D display).

[0007] Some embodiments of the present disclosure relate to user interaction with and / or manipulation of a three-dimensional preview displayed on an electronic device. In some embodiments, a two-dimensional representation of an XR content item displayed on a first electronic device can be displayed simultaneously with a three-dimensional preview of the XR content item on a second electronic device. In some embodiments, user interaction (e.g., user input, such as a touch, tap, motion, redirection, etc.) with the three-dimensional preview of the XR content item received at the second electronic device can cause the display of the three-dimensional preview of the XR content item to be updated based on the input. In some embodiments, user interaction with the three-dimensional preview of the XR content item received at the second electronic device can cause the display of the two-dimensional representation of the XR content item to be updated at the first electronic device based on the input. In some embodiments, the user input received at the second electronic device is transmitted to the first electronic device in real time (e.g., within a time less than a threshold amount), such that the display of the two-dimensional representation of the XR content item and the three-dimensional preview of the XR content item are optionally updated simultaneously or nearly simultaneously (e.g., within less than 50 ms of each other). The accompanying drawings and detailed description provide a comprehensive description of these embodiments, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals tend to designate corresponding parts throughout the several views.

[0009] Figure 1 An electronic device displaying an augmented reality environment according to an embodiment of the present disclosure is shown.

[0010] Figure 2 A block diagram illustrating an exemplary architecture of a system or device according to an embodiment of the present disclosure.

[0011] Figure 3 A block diagram illustrating an exemplary system architecture for generating and presenting content items in three dimensions according to an embodiment of the present disclosure is shown.

[0012] Figure 4 A first electronic device displaying a two-dimensional representation of a content item and a second electronic device displaying a three-dimensional representation of the content item are shown according to an embodiment of the present disclosure.

[0013] Figure 5A A preview of a content item in three dimensions displayed using an electronic device according to an embodiment of the present disclosure is shown.

[0014] Figure 5B An exemplary modification to the appearance of a content item that appears in three dimensions is shown according to an embodiment of the present disclosure.

[0015] Figures 6A to 6B A flow chart illustrating a process for generating and presenting a content item in three dimensions is shown, according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner that complies with at least one law of physics. For example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) that presents the XR environment, and in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in the physical environment. As another example, the XR system may detect interaction with one or more objects in the XR environment (e.g., (virtual) touch, tap, pinch, etc.), and in response, adjust and / or update the graphical content presented to the person in a manner similar to how such objects or views of such objects would change in the physical environment. In some embodiments, the XR system may adjust features of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

[0017] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems (including hologram-based systems), heads-up displays (HUDs), head-mounted displays (HMDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some embodiments, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology that projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as a hologram or onto a physical surface.

[0018] In some embodiments, XR content may be presented to a user via an XR data file (data file) (including scripts, executable code, etc.), which includes data representing the XR content and / or data describing how to present the XR content. In some embodiments, the XR file includes data representing one or more XR scenes and one or more triggers for presenting the one or more XR scenes. For example, the XR scene may be anchored to a horizontal, flat surface so that when a horizontal, flat surface is detected (e.g., in the field of view of one or more cameras), the XR scene may be presented. The XR file may also include data about one or more virtual objects associated with the XR scene, and / or associated triggers and actions involving the XR virtual objects.

[0019] Typically, to simplify the generation of XR files and / or the editing of computer-generated graphics, a content creation application that includes a content generation environment (e.g., an editing environment GUI) can be used. In some embodiments, the content generation environment itself is an XR environment (e.g., a two-dimensional and / or three-dimensional environment). For example, the content generation environment may include one or more virtual objects and one or more representations of real-world objects. In some embodiments, the virtual objects are superimposed on the physical environment or its representation. In some embodiments, the physical environment is captured via one or more cameras of the electronic device and is actively displayed in the XR environment (e.g., via a display generation component). In some embodiments, the physical environment is provided (e.g., passively) by the electronic device, for example, if the display generation component includes a translucent or transparent element, the user is able to see the physical environment through the translucent or transparent element.

[0020] In such a content generation environment, users can create virtual objects from scratch (including the appearance of the virtual object, the behavior / action of the virtual object, and / or the triggering of the behavior / action of the virtual object). Additionally or alternatively, virtual objects can be created by other content creators and imported into the content generation environment, where the virtual objects can be placed in an XR environment or scene. In some embodiments, virtual objects generated in the content generation environment or the entire environment (for example, via generating an XR file and importing or opening the XR file in a content creation application or an XR viewer application) can be exported to other environments or XR scenes.

[0021] Some embodiments of the present disclosure relate to a first electronic device that communicates with a second electronic device, wherein the first electronic device and the second electronic device are configured to exchange data (e.g., XR content data) therebetween. In some embodiments, the first electronic device may be configured with a content creation application, via which XR content (also referred to herein as XR content items) is created, wherein the content creation application includes a two-dimensional representation of the XR content. In some embodiments, the second electronic device may be configured with a three-dimensional graphics rendering application, via which a three-dimensional preview of the XR content created by the content creation application of the first electronic device is displayed. The content creation application may form a communication link with the three-dimensional graphics rendering application, so that the two-dimensional XR content displayed on the first electronic device may be presented as a three-dimensional preview on the second electronic device. In some embodiments, the communication link may include one or more intermediaries (e.g., one or more supporting applications) that run one or both of the first electronic device and the second electronic device. In some embodiments, the content creation application and the three-dimensional graphics rendering application may be provided within a single electronic device.

[0022] Some embodiments of the present disclosure relate to the synchronization of three-dimensional data between a first electronic device and a second electronic device (e.g., a computer and a head-mounted display, respectively). A content creation application may be run on the first electronic device and may utilize a data file describing a three-dimensional content item. The data file describing the content item may be launched in the content creation application using a communication link between the content creation application and an integrated design environment. In some embodiments, a two-dimensional representation of the content item may be displayed on the first electronic device. In some embodiments, the data file may be edited while running on the content creation application. In some embodiments, a user (e.g., using the first electronic device) may request a preview of a two-dimensional representation of the three-dimensional content item. The first electronic device may (e.g., in response to the request) initiate a data transfer with the second electronic device. In some embodiments, the three-dimensional data of the data file may be transferred to a first intermediary application running on the second electronic device. The first intermediary application may launch a second intermediary application that communicates with the first intermediary application via the communication link. The second intermediary application may use the communication link to transfer user code from the data file to a three-dimensional graphics rendering application. In some embodiments, the three-dimensional graphics rendering application may receive a request to generate content from a preview proxy application and generate a three-dimensional virtual object. The three-dimensional virtual object may be presented to the user as a three-dimensional representation or a two-dimensional representation. The 3D preview of the content item can be displayed on the second electronic device as a 3D content item rendered in a 3D XR environment. The 3D graphical rendering of the content item can be simultaneously presented as a 2D representation of the content item displayed on the first electronic device (e.g., having a 2D display).

[0023] Some embodiments of the present disclosure relate to user interaction with and / or manipulation of a three-dimensional preview displayed on an electronic device. In some embodiments, a two-dimensional representation of an XR content item displayed on a first electronic device can be displayed simultaneously with a three-dimensional preview of the XR content item on a second electronic device. In some embodiments, user interaction (e.g., user input, such as a touch, tap, motion, redirection, etc.) with the three-dimensional preview of the XR content item received at the second electronic device can cause the display of the three-dimensional preview of the XR content item to be updated based on the input. In some embodiments, user interaction with the three-dimensional preview of the XR content item received at the second electronic device can cause the display of the two-dimensional representation of the XR content item to be updated at the first electronic device based on the input. In some embodiments, the user input received at the second electronic device is transmitted to the first electronic device in real time (e.g., within a time less than a threshold amount), such that the display of the two-dimensional representation of the XR content item and the three-dimensional preview of the XR content item are optionally updated simultaneously or nearly simultaneously (e.g., within less than 50 ms of each other). The accompanying drawings and detailed description provide a comprehensive description of these embodiments, and it should be understood that this summary does not limit the scope of the present disclosure in any way.

[0024] Figure 1 An electronic device 100 is shown that displays an XR environment (e.g., a computer-generated environment) according to an embodiment of the present disclosure. In some embodiments, the electronic device 100 is a handheld device or a mobile device, such as a tablet, laptop, smartphone, or head-mounted display. Figure 2 An example of the device 100 is described with reference to a block diagram of the structure of FIG. Figure 1 As shown, the electronic device 100 and the desktop 120 are located in the physical environment 110. In some embodiments, the electronic device 100 can be configured to capture an area of ​​the physical environment 110 including the desktop 120 and the plant 156 (shown in the field of view of the electronic device 100). In some embodiments, in response to a trigger, the electronic device 100 can be configured to capture an area of ​​the physical environment 110 including the desktop 120 and the plant 156 (shown in the field of view of the electronic device 100). Figure 1 , a virtual object 130 is displayed in the computer-generated environment 100 that is not present in the physical environment 110 but is displayed in the computer-generated environment positioned on top of (e.g., anchored to) the computer-generated representation 120′ of the real-world desktop 120. For example, the virtual object 130 may be displayed on the surface of the desktop 120′ in the computer-generated environment displayed via the device 100 in response to detecting the planar surface of the desktop 120 in the physical environment 110. Figure 1As shown in the example of , the computer-generated environment may include representations of additional real-world objects, such as representation 156' of real-world plants 156. It should be understood that the virtual object 130 is a representative virtual object, and one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or three-dimensional virtual objects) may be included and rendered in a three-dimensional computer-generated environment. For example, a virtual object may represent an application or user interface displayed in a computer-generated environment. In some embodiments, the application or user interface may include a display of content items (e.g., photos, videos, etc.) of a content application. In some embodiments, the virtual object is optionally configured to be interactive and responsive to user input, so that the user can virtually touch, tap, move, rotate, or otherwise interact with the virtual object. In addition, it should be understood that the 3D environment (or 3D virtual object) described herein may be a representation of a 3D environment (or three-dimensional virtual object) projected or presented at an electronic device.

[0025] In the following discussion, an electronic device is described that communicates with a display generation component and one or more input devices. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in this disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device for display on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received on an electronic device (e.g., touch input received on the touch-sensitive surface of the electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0026] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application. In addition, the device may support a content creation application (e.g., an application with a content generation environment) for generating or editing computer-generated graphics and / or content of an XR environment. In addition, the device may support a three-dimensional graphics rendering application for generating XR content and / or an XR environment and displaying it in three dimensions.

[0027] Figure 2 A block diagram of an exemplary architecture of a system or device 250 according to an embodiment of the present disclosure is shown. In some embodiments, device 250 is a mobile device, such as a mobile phone (e.g., a smart phone), a tablet computer, a laptop computer, a desktop computer, a head-mounted display, an auxiliary device for communicating with another device, etc. Device 250 optionally includes various sensors (e.g., one or more hand tracking sensors, one or more position sensors, one or more image sensors, one or more touch-sensitive surfaces, one or more motion and / or orientation sensors, one or more eye tracking sensors, one or more microphones or other audio sensors, etc.), one or more display generation components, one or more speakers, one or more processors, one or more memories and / or communication circuits. One or more communication buses are optionally used for communication between the above-mentioned components of device 250.

[0028] In some embodiments, as Figure 2As shown, the system / device 250 can be divided between multiple devices. For example, the first device 260 optionally includes a processor 218A, one or more memories 220A, communication circuitry 222A, and a display generation component 214A that optionally communicates via a communication bus 208A. The second device 270 (e.g., corresponding to device 200) optionally includes various sensors (e.g., one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206, one or more touch-sensitive surfaces 208, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, etc.), one or more display generation components 214B, one or more speakers 216, one or more processors 218B, one or more memories 220B, and / or communication circuitry 222B. One or more communication buses 208B are optionally used for communication between the above-mentioned components of the device 270. First device 260 and second device 270 optionally communicate via a wired or wireless connection between the two devices (eg, via communication circuitry 222A-222B).

[0029] The communication circuits 222A, 222B optionally include circuits for communicating with electronic devices, networks such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuits 222A, 222B optionally include circuits for communicating using near field communication (NFC) and / or short range communication such as A circuit for communication.

[0030] Processors 218A, 218B include one or more general-purpose processors, one or more graphics processors and / or one or more digital signal processors. In some embodiments, memory 220A, 220B is a non-transient computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) that stores computer-readable instructions, which are configured to be executed by processors 218A, 218B to perform the technology, process and / or method described below. In some embodiments, memory 220A, 220B may include more than one non-transient computer-readable storage medium. A non-transient computer-readable storage medium may be any medium (e.g., not including a signal) that can tangibly contain or store computer-executable instructions for use by an instruction execution system, device and apparatus or in combination with the instruction execution system, device and apparatus. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transient computer-readable storage medium. A non-transient computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like.

[0031] In some embodiments, the display generation components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some embodiments, the display generation components 214A, 214B include multiple displays. In some embodiments, the display generation components 214A, 214B may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, etc. In some embodiments, the device 270 includes a touch-sensitive surface 208 for receiving user input such as tap input and swipe input or other gestures. In some embodiments, the display generation components 214B and the touch-sensitive surface 208 form a touch-sensitive display (e.g., a touch screen integrated with the device 270 or a touch screen external to the device 270 that communicates with the device 270).

[0032] Device 270 optionally includes an image sensor 206. Image sensor 206 optionally includes one or more visible light image sensors (such as a charge-coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain images of physical objects from a real-world environment. Image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. Image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of physical objects from device 270. In some embodiments, information from the one or more depth sensors may allow the device to identify objects in the real-world environment and distinguish them from other objects in the real-world environment. In some embodiments, the one or more depth sensors may allow the device to determine the texture and / or topography of objects in the real-world environment.

[0033] In some embodiments, device 270 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around device 270. In some embodiments, image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information about physical objects in the real-world environment. In some embodiments, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some embodiments, device 270 uses image sensor 206 to detect the position and orientation of device 270 and / or display generation component 214 in the real-world environment. For example, device 270 uses image sensor 206 to track the position and orientation of display generation component 214B relative to one or more fixed objects in the real-world environment.

[0034] In some embodiments, device 270 includes microphone 213 or other audio sensor. Device 270 uses microphone 213 to detect sounds from the user and / or the user's real-world environment. In some embodiments, microphone 213 includes an array of microphones (multiple microphones) that optionally operate in concert to identify ambient noise or locate a sound source in the space of the real-world environment.

[0035] Device 270 includes a location sensor 204 for detecting the location of device 270 and / or display generating component 214B. For example, location sensor 204 may include a GPS receiver that receives data from one or more satellites and allows device 270 to determine its absolute location in the physical world.

[0036] Device 270 includes an orientation sensor 210 for detecting the orientation and / or movement of device 270 and / or display generation component 214B. For example, device 270 uses orientation sensor 210 to track changes in the position and / or orientation of device 270 and / or display generation component 214B, such as changes relative to physical objects in the real-world environment. Orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0037] In some embodiments, the device 270 includes a hand tracking sensor 202 and / or an eye tracking sensor 212. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214B, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214B. In some embodiments, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214B. In some embodiments, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214B.

[0038] In some embodiments, the hand tracking sensor 202 can use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more hands (e.g., one or more hands of a human user). In some embodiments, the hands can be resolved with sufficient resolution to distinguish fingers and their corresponding positions. In some embodiments, one or more image sensors 206 are positioned relative to the user to define a field of view and interaction space of the image sensor 206, in which the finger / hand position, orientation and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) can be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0039] In some embodiments, the eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source, such as an LED) that emits light toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some embodiments, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and focus / gaze can be determined by tracking both eyes. In some embodiments, one eye (e.g., the dominant eye) is tracked by respective eye tracking cameras / illumination sources.

[0040] Device 270 and system 250 are not limited to Figure 2 Components and configurations, but may include fewer components, other components, or additional components in multiple configurations. In some embodiments, system 250 can be implemented in a single device. A person using system 250 is optionally referred to herein as a user of the device. Now focus on an exemplary system architecture for generating and presenting three-dimensional previews of content and associated processes implemented on electronic devices such as electronic device 260 and electronic device 270. The system architecture can describe the data flow and data transmission from a content creation application running on a first electronic device to a three-dimensional graphics rendering application running on a second electronic device. In some embodiments, the processing of the system described below can be operated by processors 218A, 218B of devices 260 and 270.

[0041] Figure 3 A block diagram of an exemplary system architecture 300 for generating and presenting three-dimensional content items according to an embodiment of the present disclosure is shown. As described above, a content item (e.g., an XR content item, such as a 3D content item) created in a content creation application running on a first electronic device is Figure 1 The virtual object 130 in the content creation application can be previewed in three dimensions via a three-dimensional graphics rendering application running on a second electronic device, which is optionally in communication with the first electronic device. As described above, the first electronic device can be a mobile or non-mobile computing device, such as a mobile phone (e.g., a smartphone), a tablet computer, a laptop computer, or a desktop computer, and the second electronic device can be a head-mounted display, a projector, a hologram generator, an auxiliary device that communicates with another device, etc. As described below, the content creation application can, in response to a user's request to preview the content item in three dimensions, transmit graphical data corresponding to the content item to the three-dimensional graphics rendering application via a communication link, and the three-dimensional graphics rendering application can generate and present a three-dimensional preview of the content item.

[0042] While some aspects of a content item created in a content creation application can be captured in two dimensions (e.g., color, two-dimensional dimensions such as height and width, plan views, etc.), other aspects cannot be captured. Specifically, if a content item created in a content creation application is intended and / or explicitly designed to be displayed in a three-dimensional environment, the two-dimensional preview can provide the designer (i.e., user) with incomplete information about the appearance of the three-dimensional content. Alternative views (e.g., side and rear views), surface textures, lighting effects, etc. may not be visible or captured within the two-dimensional preview. Furthermore, in order to view alternative views of the content, for example, the user may need to generate a new preview for each alternative view, which adds time and effort to the workflow of designing, previewing, and modifying the content item, and thus increases complexity. Therefore, as described below, providing a three-dimensional preview can be particularly useful during the design phase of the digital content creation process.

[0043] In some embodiments, as Figure 3 As shown, using the first electronic device 360 ​​(eg, corresponding to Figure 2 260) can work in a content creation application 310 ("CC application"). The content creation application optionally communicates with an integrated design environment 312 (IDE). The content creation application 310 and / or IDE 312 utilize data files (e.g., including scripts, executable code, etc.) that describe content items (e.g., that define the appearance, actions, reactivity, etc. of content items (e.g., one or more virtual objects) designed for use with a three-dimensional operating system (e.g., designed for a three-dimensional graphics environment). In some embodiments, data files describing content items can be uploaded to the content creation application 310 and / or launched in the content creation application using a communication link 320 between the IDEs 312. In some embodiments, the content creation application can be launched on a first electronic device (e.g., via a computer program). Figure 2The display generation component 214A in the first electronic device displays a two-dimensional representation of the content item. It should be understood that the two-dimensional representation is a function of the two-dimensional display of the first electronic device, but the two-dimensional representation can represent three-dimensional content. In some embodiments, the two-dimensional representation of the content item can be displayed in the display UI of the content creation application. In some embodiments, the two-dimensional representation of the content item can be displayed in a display UI separate from the content creation application (for example, a display UI running on a second application that communicates with the content creation application). It should be understood that the content creation application is running on the first electronic device, and the data files uploaded to the content creation application can be stored on the first electronic device (for example, stored in memory 220A or downloaded and accessed from a web-based storage). In some embodiments, the data files can be edited while the content creation application is running. In some such embodiments, scripts, executable code, etc. can be displayed in a portion of the display UI of the content creation application, allowing the user to directly edit portions of the scripts, executable code, etc. at the first electronic device. If applicable, editing the data file can update the two-dimensional representation of the content item displayed at the first electronic device. As described herein, in some embodiments, editing the data file can be implemented in an IDE 312 that communicates with the content creation application 310.

[0044] In some embodiments, a user may request (e.g., using a first electronic device) to preview a two-dimensional representation of a content item in three dimensions. As an example, the content creation application may include an optional user interface element (e.g., displayed on a display UI or displayed in the content creation application or at some other UI in communication with the content creation application) that generates a request when selected. In some embodiments, the request may be entered using one or more input devices in communication with the first electronic device, such as by pressing one or more keys on a keyboard or a button on a mouse. In response to the request, Figure 3 As shown, the first electronic device may initiate a communication with the second electronic device 370 (eg, corresponding to Figure 2 Data transmission of the device 270). Figure 3 As shown, the second electronic device 370 may be configured with one or more applications for generating a three-dimensional preview of the content item. The content item may be presented to the user as a three-dimensional representation of the content item using the second electronic device, and a two-dimensional representation of the content item may be displayed at the first electronic device. The first electronic device may communicate with the second electronic device using a communication link 322. The communication link 322 may be any suitable wired or wireless communication means, including but not limited to a universal serial bus (USB), Wi-Fi, or (e.g., wireless transmission), or, if the disclosed architecture 300 is implemented on a single electronic device, via cross-process communication (e.g., inter-processor communication).

[0045] In some embodiments, the three-dimensional data of a data file (e.g., all or part of a script / executable code within the data file) can be transmitted to a first intermediary application 314 running on a second electronic device. As an example, the first intermediary application can be a preview shell application that is configured to receive all or part of a script, executable code, etc. of a data file transmitted from a content creation application of the first electronic device and, in response to receiving the data file (or part thereof), launch the preview shell. In some embodiments, the preview shell application is optionally a trusted application (e.g., a first-party application) for the second electronic device, such that the data, script, executable code, etc. loaded within the preview shell application is protected. The preview shell application 314 can launch a second intermediary application 316, which can be a preview proxy application that communicates with the preview shell application 314 via a communication link 324. In some embodiments, the preview proxy application 316 running on the second electronic device 370 can be configured to execute code (e.g., a three-dimensional graphical data representation of a script, executable code of a data file, etc.) provided from the first electronic device and transmitted to the second electronic device via the preview shell application 314. The preview proxy application can transfer the executable code to the three-dimensional graphics rendering application 318 using the communication link 326. In some embodiments, a preview shell application 314 and a preview proxy application 316 can be included to allow untrusted applications or unauthorized applications (e.g., third-party applications) that can be executed by the preview proxy application 318 to communicate securely between the first electronic device and the second electronic device without security risks or the risk of crashing the hardware or software of the second electronic device. In some embodiments, the three-dimensional graphics rendering application 318 (3D graphics rendering application) can receive a request to generate content from the preview proxy application 314 and generate a virtual object in three dimensions. The three-dimensional virtual object can be presented to the user in a three-dimensional representation or a two-dimensional representation. The three-dimensional preview of the content item can be presented as a three-dimensional content item rendered in a three-dimensional XR environment (e.g., via Figure 2The three-dimensional graphical rendering of the content item can be presented as a two-dimensional representation of the content item displayed on the first electronic device (e.g., a device having a two-dimensional display). Thus, as described above, one or more communication channels or links can be established between the first and second electronic devices and between applications running on the first and second electronic devices, and more specifically, between the content creation application 310 of the first electronic device and the three-dimensional graphics rendering application 318 of the second electronic device.

[0046] As an example, a three-dimensional content preview of code within a data file can provide a user with useful visual feedback about the appearance of the content in the XR environment. In some embodiments, edits or modifications to a data file running in a content creation application can create corresponding changes to the appearance of the three-dimensional preview, which can be rendered and presented to the user at a second electronic device and / or at a first electronic device. As an example, a user may wish to edit or modify one or more features of a content item and view a new three-dimensional preview of the content item based on the edits or modifications. For example, a user can rewrite portions of a script, executable code, etc. of a data file via one or more input devices (e.g., a keyboard) in communication with a first electronic device, optionally while simultaneously displaying a two-dimensional representation of the content item on the first electronic device and simultaneously presenting a three-dimensional preview of the content item to the user using a second electronic device. The user can finalize the edit or modification (e.g., by saving changes to the data file) and can request a new preview of the content item representing the data file. Additionally or alternatively, a new preview can be automatically requested once the edit or modification is completed by the user. In some embodiments, the two-dimensional representation of the content item displayed on the first electronic device can be updated accordingly.

[0047] As described above, the preview shell application may be a trusted application that executes only trusted code, and the preview proxy application may be an application configured to execute untrusted application code (eg, generated by a third-party application developer). Figure 3The system architecture 300 shown allows scripts, executable code, etc. corresponding to the creation of content (and animation of the content) in a data file to be executed within a preview agent application on a second electronic device, but may include untrusted code. Therefore, one advantage of the disclosed three-dimensional preview system is that untrusted user data and code written at a first electronic device can be securely transmitted to a second electronic device and executed on the second electronic device to generate and present a three-dimensional preview of the data on the second device (and represented in two dimensions on the first device), while maintaining secure communication between the first and second electronic devices without compromising the hardware and / or software of the second electronic device. For example, in the event that the user code within the data file crashes during execution, the first device and / or the second electronic device can be protected from the possible impact of the untrusted code (e.g., preventing the device from crashing).

[0048] As described above, a three-dimensional preview of the content item can be displayed on the second electronic device while a two-dimensional representation of the content item is also displayed simultaneously (e.g., partially or completely coextensive in time) on the first electronic device. Figure 3 The system architecture 300 shown can be provided in a single electronic device (e.g., a laptop computer, desktop computer, mobile device, etc.) rather than being implemented and distributed between two electronic devices. For example, a three-dimensional graphics rendering application can be provided within a simulator or at least partially provided as a simulator that is configured to generate content for a computer-generated environment in three dimensions, without presenting the content in three dimensions due to the two-dimensional display characteristics (e.g., two-dimensional display) of the electronic device. In some such embodiments, in response to receiving a request to display a content item in three dimensions, the three-dimensional graphics rendering application can generate and present a preview of the three-dimensional content as a two-dimensional representation within the simulator's computer-generated environment (e.g., in different windows on the display of the electronic device). Additionally or alternatively, in some embodiments, a content creation application can be directly or indirectly communicatively linked to a three-dimensional graphics rendering application running on the same electronic device as the content creation application. In some such embodiments, all or part of a script, executable code, etc. of a data file can be transferred to the three-dimensional graphics rendering application 318 on the device. In such embodiments, the three-dimensional graphics rendering application can optionally be configured with at least some functionality of, for example, a preview shell application and a preview proxy application. In some embodiments, one or more of the intermediary applications between the content creation application and the 3D graphics rendering application may be omitted. In some such embodiments, the functionality of one or more of the intermediary applications may be implemented as part of the content creation application or the 3D graphics rendering application. Additionally, it should be understood that in some embodiments, the request may be received from, for example, Figure 3 The content creation application is shown propagating to the 3D graphics rendering application, but the rendered 3D content from the 3D graphics rendering application can be communicated more directly back to the content creation application to reduce latency between generating the 3D content and displaying it on the first electronic device.

[0049] It should be understood that although referred to herein as applications, the intermediary system components (e.g., the preview shell and the preview agent) may be provided or implemented as one or more software that executes individually or in combination within the processor of the second electronic device (e.g., processor 218B), rather than being provided as separate applications. As described above, in some embodiments, the graphics data transmitted along the communication channel between the first electronic device and the second electronic device may be synchronized. In some such embodiments, the communication channel between the content creation application and the three-dimensional rendering application may be bidirectional, allowing data to be transmitted in either direction between the two. Thus, as described with reference to Figures 5A to 5B As discussed in detail, a user may interact with a three-dimensional preview of a content item such that the user interaction affects not only the appearance of the three-dimensional preview of the content item, but also the appearance of a two-dimensional representation of the content item at the content creation application.

[0050] Attention is now directed to an exemplary simultaneous display of a two-dimensional representation of a content item and a corresponding three-dimensional preview of the content item. As described above, the two-dimensional representation of the content item can be displayed on a first electronic device (e.g., via a content creation application), and the three-dimensional preview of the content item can be simultaneously displayed on a second electronic device (e.g., via a three-dimensional graphics rendering application). Figure 4 Shown are a first electronic device 460 displaying a two-dimensional representation of a content item 464 and a second electronic device displaying a three-dimensional representation of a content item (eg, one or more virtual objects 430 ), in accordance with an embodiment of the present disclosure.

[0051] like Figure 4 As shown, the first electronic device 460 in the environment 468 may include a content creation application 462 showing an editing environment GUI. As shown, the editing environment GUI may be displayed on the first electronic device 460, which may be, for example, a desktop computing device, and may communicate with one or more input devices 466. Figure 4 As shown, the second electronic device (ie, the second electronic device (eg, corresponding to Figure 2A viewpoint associated with an electronic device 270 (e.g., a second electronic device such as a head-mounted display) can present a three-dimensional environment 468 or a representation of the environment 468 (e.g., a 3D computer-generated environment defined by X, Y, and Z axes) that includes a three-dimensional preview application 434. As described herein, the three-dimensional preview application 434 can include content 432, and the corresponding content can be displayed in two dimensions in a content creation application represented by the two-dimensional content 464.

[0052] exist Figure 4 In the example of , three-dimensional content 432 may include one or more virtual objects 430, such as a chair, a table, a cup, etc., but it should be understood that these objects are merely representative and one or more different virtual objects (e.g., 2D or 3D objects) may be imported or designed within the content creation application (including a plurality of shapes, objects, symbols, text, numbers, etc.) and included in the 3D preview application 434.

[0053] Additionally, it should be understood that the 3D environment (or 3D virtual object) described herein and presented using a second electronic device (e.g., a head-mounted display) may be a three-dimensional representation of a two-dimensional (2D) environment (or 2D virtual object) simultaneously displayed on a first electronic device (e.g., displayed on a 2D screen of the first electronic device). In some embodiments, the 3D preview application 434 may display grid lines or other indicators to provide the content creator with information about the placement and / or size of the virtual object in the 3D environment.

[0054] As described above, the user may wish to preview a content item designed and / or edited in a content creation application or in an IDE on a first electronic device because the content is intended to appear in three dimensions. Figure 4 In an example, a user may be designing an XR scene that includes one or more virtual objects (e.g., a chair, a table, a cup, etc.). The scene may include virtual objects that are specifically designed to be displayed within a three-dimensional XR environment (e.g., a computer game, an application, a movie, a television program, etc.). Thus, it may be particularly advantageous for a content creator to be able to (e.g., in response to input) preview the current state (e.g., current appearance, form, perspective, etc.) and / or behavior of one or more virtual objects within the scene in three dimensions. As described above, when referring to Figure 3 When the user requests to preview the content in three dimensions within the content creation application 462 (or IDE) on the first electronic device 460 (e.g., by selecting a “preview” user interface element displayed within the content creation application). Figure 4As shown, a second electronic device can be used to present a three-dimensional preview of content 432. Additionally, a two-dimensional representation of the generated content (e.g., two-dimensional content 464) can be displayed on the first electronic device 460. In some embodiments, a second electronic device can be used to simultaneously present a representation of the first electronic device 406 and a two-dimensional preview of its content, such that a user of the second electronic device is simultaneously presented with the two-dimensional content 464 and the three-dimensional content 432 in the environment 468. It should be understood that the generation and presentation of the preview of the three-dimensional content 432 can be performed by the above-mentioned and Figure 3 Thus, one advantage of the disclosed 3D preview generation system and method is that a 3D preview of a content item being created can be presented to provide the user with immediate and accurate feedback on how the content item will look and / or behave in an actual implementation (e.g., this may only be possible using the hardware and software of the second electronic device or a simulator of the second electronic device).

[0055] In some embodiments, a user may wish to interact with a three-dimensional preview of content displayed on a second electronic device, e.g., to obtain additional perspectives and feedback about the content being designed via the content creation application and / or the behavior of the content. As an example, at the second electronic device, the user can scale, zoom in / out, rotate, and / or move the three-dimensional content 432 (e.g., by selecting a user interface element displayed within the 3D environment 468 and / or by performing gestures (e.g., finger taps and pinches)). Additionally or alternatively, in some embodiments, the user can scale, zoom in / out, rotate, and / or move the three-dimensional preview using one or more input devices 466 in communication with the first electronic device 460 (e.g., by selecting a user interface element displayed within the content creation application 462 via a mouse click or entering a command sequence via a keyboard). Manipulating the three-dimensional content 432 in the preview can allow the user to obtain more comprehensive (and more useful) visual feedback in three dimensions, such as alternative views of the content item from any direction / perspective (e.g., side and / or rear views), surface texture and / or color of the content item, spatial relationships between one or more virtual objects 430 of the content, depth perspective of one or more virtual objects 430, and other possibilities. This visual feedback may be difficult or impossible to obtain via a 2D representation. Using this visual feedback, the user can then edit the content in real time.

[0056] As an example, a user may edit a script, executable code, or the like running in the content creation application 462 via one or more input devices 466 to change the appearance of one or more virtual objects 430 of the content 432 being previewed using the second electronic device. For example, the user may wish to change the color, size, surface texture, etc. of one or more virtual objects 430 of the content 432, add, delete, or reposition one or more virtual objects of the content item, and / or add, delete, or modify different behaviors associated with the input. After the representation of the two-dimensional content item has been edited or modified, the second electronic device may be used to update the three-dimensional preview of the content 432 so that the user can visually analyze the new appearance of the content in three dimensions, as previously described with reference to FIG. Figure 3 Thus, one advantage of the disclosed 3D preview generation system and method is that a 3D preview of a content item being created can be updated in real time (e.g., in 50 ms or less) to provide the user with immediate and accurate feedback regarding changes to the appearance and form of the content item being created.

[0057] In some embodiments, the presentation of the three-dimensional preview of the content can be changed and / or updated in response to a change in the viewpoint of the second electronic device (e.g., a change in the viewpoint associated with a head-mounted display). In some such embodiments, a user wearing, holding, or otherwise viewing through the display of the second electronic device can reorient the viewpoint associated with the second electronic device relative to the 3D environment 468 being displayed. For example, the user can "walk around" the content 432 displayed within the 3D environment 468 to obtain an alternative side view of one or more virtual objects 430 (or a view of the content from a higher or lower angle (e.g., a top view)). As the user moves the viewpoint of the second electronic device, the preview of the three-dimensional content 432 in the preview application 434 can remain fixed within the 3D environment 468 (e.g., fixed in position and orientation relative to the real-world environment).

[0058] Additionally or alternatively, it may be particularly advantageous for a user to be able to edit content currently being previewed in three dimensions by interacting with the three-dimensional preview of the content. Now, attention is focused on an exemplary three-dimensional preview of a content item displayed via an electronic device (e.g., a head-mounted display). As described below, user input received at an electronic device (e.g., corresponding to the second electronic device described above) can change the appearance of the three-dimensional preview displayed on the electronic device, which can simultaneously change the appearance of a two-dimensional representation of the content item displayed in a content creation application (e.g., on a desktop computer).

[0059] Figure 5A A preview of a three-dimensional content item displayed using an electronic device according to an embodiment of the present disclosure is shown. Figure 4When the content is created in a content creation application on the first electronic device, it can be previewed in three dimensions on the second electronic device to provide visual feedback on the appearance and form of the content in three dimensions. Figure 4 For example, Figure 5A One or more virtual objects, such as a chair 531, a cup 533, and a table 535, are shown displayed within a 3D preview application 534 on an electronic device (e.g., corresponding to the second electronic device described above). As described below, a user can interact with content items (e.g., one or more virtual objects) 531, 533, and 535 at the electronic device to change the appearance or behavior of such objects. In some embodiments, changes in the appearance and / or behavior of the objects may result in modifications to the two-dimensional representation of the content (e.g., corresponding to the second electronic device described above). Figure 4 464) and thereby edit or modify the data files that define / describe the content.

[0060] Re-reference Figure 3 , three-dimensional content data can be exchanged between a content creation application on a first electronic device and a 3D graphics rendering application on a second electronic device, wherein the 3D content data can be exchanged bidirectionally. As described above, changes to the scripts, executable code, etc. of the data files that define the content running in the content creation application can result in corresponding changes to the two-dimensional representation of the content on the first electronic device, and therefore result in corresponding changes to the three-dimensional preview of the content on the second electronic device. In addition, in some embodiments, changes to the three-dimensional preview of the content running in the 3D graphics rendering application on the second electronic device can result in corresponding changes to the two-dimensional representation of the content on the first electronic device, and therefore result in corresponding changes to the scripts, executable code, etc. of the data files that define the content. For example, with reference to Figure 5A , the user may wish to change the position of chair 531 relative to the position of cup 533 and table 535 of 3D preview application 534 .

[0061] Figure 5B , an exemplary modification of the appearance of a content item in three dimensions according to an embodiment of the present disclosure is shown. As described above, input can be received at the electronic device, where the input is a user interaction with the three-dimensional preview of the content item. As an example, the user can virtually select chair 531 (e.g., via a finger tap / touch) and, while continuing to display 3D preview application 534, drag chair 531 to a new position relative to the initial position of the chair within the 3D environment, such as Figure 5B As shown in the figure, the three-dimensional previews of content items 531, 533 and 535 are relative to Figure 5AThe initial positioning of the content items 531, 533 and 535 shown in the example of has changed. As described above, such a change to the three-dimensional preview of the content items at the electronic device may change / modify the data files defining the content items. For example, referring back to Figure 3 , the repositioning of chair 531 can change the 3D data running in 3D graphics rendering application 318. This change to the 3D data can then be propagated to the preview proxy application, preview shell application, etc., back to the content creation application. Content creation application 310 can update the two-dimensional representation of the content item based on the repositioning of chair 531, which can then be converted into corresponding changes in the script, executable code, etc. of the data file in IDE 312 (e.g., corresponding changes to the coordinate definitions listed in the script, executable code, etc. of the data file). In this way, as described above, user input corresponding to interaction with the three-dimensional preview of the content item received at a (second) electronic device (e.g., a head-mounted display) can be captured by changes to the data file defining the content item in the content creation application at the first electronic device. Therefore, the synchronization of 3D data between the first and second electronic devices is advantageous in that synchronization allows the user to work with the 3D data and interact with user code on either or both of the first or second electronic devices. In addition, the disclosed process of modifying the appearance of a content item by manipulating the content item in three-dimensional space can improve the user experience by providing a simple and intuitive operation for editing scripts, executable code, etc. of a graphics data file. Thus, another advantage is that user interaction with one or more virtual objects of a three-dimensional preview may provide for easier and more precise object manipulation and / or modification.

[0062] In some embodiments, content items can be displayed simultaneously in two dimensions and three dimensions to provide a multi-user experience. For example, according to some embodiments described herein, a first user can operate a first electronic device (e.g., a desktop computer) that includes a content creation application, and a second user can operate a second electronic device (e.g., a head-mounted display) that includes at least a 3D graphics rendering application. As described above, the content creation application can include content items that include one or more virtual objects (e.g., 531, 533, and 535) displayed as two-dimensional representations on the first electronic device. The 3D graphics rendering application can be configured to display a preview of the content item in three dimensions on the second electronic device (e.g., in a 3D preview application 534). Thus, the first user can view a two-dimensional representation of the content item on the first electronic device, and the second user can simultaneously view a three-dimensional preview of the content item on the second electronic device. The first user and the second user can then collaboratively view, modify, and update the content item by separately operating the first and second electronic devices. As described above, editing or modifying the content item on the respective electronic devices causes the corresponding representation of the content item on the other electronic device to be updated based on the edit or modification. Thus, the first user and the second user can actively interact with the two-dimensional representation and the three-dimensional representation of the content item, respectively, to collaboratively modify and ultimately complete the appearance and form of the content item. Therefore, another advantage is that simultaneously displaying the content item in two dimensions on the first electronic device and in three dimensions on the second electronic device can provide efficient and more convenient user collaboration between the first user operating the first electronic device and the second user operating the second electronic device.

[0063] It should be understood that Figures 5A to 5B The virtual objects shown in (e.g., 531, 533, and 535) are exemplary, and alternative and / or additional virtual objects may be provided within the 3D preview application 534. It should also be understood that while Figures 5A to 5B, but the two-dimensional representation of the virtual object can be displayed simultaneously within the user's view (e.g., on the first electronic device), as discussed throughout this disclosure. It should also be understood that, as described herein, the two-dimensional representation of the content item and the corresponding three-dimensional preview of the content item can be displayed simultaneously on a single electronic device (e.g., using a simulator application running a 3D graphics rendering application). It should also be understood that when a change to the three-dimensional preview of the content is received at the second electronic device, as described above, real-world objects within the viewpoint of the second electronic device (e.g., a real-world table, lighting / shadow effects from a lamp, etc.) may not be captured by the two-dimensional representation of the content (e.g., the two-dimensional representation of the real-world table is not rendered and displayed on the first electronic device). Therefore, in addition to the second electronic device that can provide a three-dimensional representation in response to environmental input, a comparison of the two-dimensional representation on the first electronic device with the three-dimensional representation on the second electronic device can be used to distinguish between behaviors caused by three-dimensional content (e.g., lighting effects, etc.) that may not reflect changes to the content itself.

[0064] Figures 6A to 6B 6 shows a flow chart illustrating a process 600 for generating and presenting a content item in three dimensions according to an embodiment of the present disclosure. Figure 6A As shown, at 602, a first electronic device (e.g., a desktop computer, a laptop computer, a tablet computer, a mobile device, etc.) displays a content creation application that may include a two-dimensional representation of a content item (e.g., one or more objects). In some embodiments, a request to preview the content item in three dimensions may be received at 604 at the first electronic device (e.g., via one or more input devices in communication with the first electronic device). The request optionally corresponds to a selection of a "preview" user interface element displayed within the content creation application. In some embodiments, the request may be received from the execution of code in an IDE. At 606, a second electronic device (e.g., a head-mounted display or other projection system / device in communication with the first electronic device) may be used to generate and present a preview of the content item in three dimensions.

[0065] In some embodiments, as described above, the disclosed process 600 may utilize a content preview system (e.g., corresponding to Figure 3 The system 300 shown in FIG. 3 is used to generate and present content items in three dimensions. Figure 6B As shown, generating and presenting a content item in three dimensions may include establishing a communication link between a first electronic device and a second electronic device at 622, and transmitting graphics data (e.g., 3D graphics data) corresponding to a two-dimensional representation of the content item to a three-dimensional graphics rendering application at the second electronic device at 630. In some embodiments, as Figure 6BAs shown, establishing a communication link between the first electronic device and the second electronic device optionally includes: establishing communication between a content creation application at the first electronic device and a preview shell application at the second electronic device at 624; establishing communication between the preview shell application and the preview proxy application at the second electronic device at 626; and establishing communication between the preview proxy application and the 3D graphics rendering application at the second electronic device at 628. As shown, transmitting graphics data corresponding to a two-dimensional representation of the content item to the 3D graphics rendering application optionally includes: transmitting the graphics data from the content creation application to the preview shell application at the second electronic device (e.g., via a wired or wireless connection) at 632; transmitting the graphics data from the preview shell application to the preview proxy application at the second electronic device (e.g., via cross-process communication) at 634; and transmitting the graphics data from the preview proxy application to the 3D graphics rendering application at the second electronic device (e.g., via cross-process communication) at 636.

[0066] refer to Figure 6A At 608, when the content item is presented in three dimensions using the second electronic device and a two-dimensional representation of the content item is simultaneously displayed using the first electronic device, at 610, input is optionally received at the second electronic device. In some embodiments, the input is optionally an interaction with the three-dimensional content item presented at the second electronic device (e.g., a finger tap or finger touch, a redirection of a viewpoint associated with the second electronic device, or a drag-and-drop interaction with one or more objects of the content item). Based on the input, at 612, the appearance of the content item presented in three dimensions using the second electronic device is optionally updated at 614 (e.g., the appearance of the three-dimensional preview changes based on the input), and the appearance of the two-dimensional representation of the content item displayed on the first electronic device is updated at 616 (e.g., the appearance of the two-dimensional representation of the content item changes based on the input).

[0067] It should be understood that process 600 is an example, and more, fewer, or different operations may be performed in the same or different order. In addition, the operations in the above process 600 are optionally performed by running an information processing device such as a general-purpose processor (e.g., as described with respect to FIG. Figure 2 said) or one or more functional modules in a dedicated chip and / or by Figure 2 other components to achieve this.

[0068] Therefore, in accordance with the above, some embodiments of the present disclosure relate to a method for generating a three-dimensional preview. The method may include: displaying a content creation application including a two-dimensional representation of a first content item on a first electronic device; receiving a request to preview the first content item in three dimensions on the first electronic device; generating and presenting the first content item in three dimensions using a second electronic device; and while presenting the first content item in three dimensions using the second electronic device and simultaneously displaying the two-dimensional representation of the first content item using the first electronic device: receiving input on the second electronic device; and updating the appearance of the first content item in three dimensions presented using the second electronic device and updating the appearance of the two-dimensional representation of the first content item displayed on the first electronic device based on the input.

[0069] Additionally or alternatively, in some embodiments, the first electronic device may be a laptop computer, a desktop computer, or a tablet computer. Additionally or alternatively, in some embodiments, the second electronic device may be a head-mounted display.

[0070] Additionally or alternatively, in some embodiments, the method may further include capturing, using the second electronic device, at least a portion of the real-world environment using one or more input devices of the second electronic device. Additionally or alternatively, in some embodiments, generating and presenting the first content item in three dimensions may include presenting a representation of the captured portion of the real-world environment.

[0071] Additionally or alternatively, in some embodiments, capturing the portion of the real-world environment may include capturing the first electronic device and the content creation application, and generating and presenting the first content item in three dimensions may include presenting a representation of the first electronic device and the content creation application that includes a two-dimensional representation of the first content item.

[0072] Additionally or alternatively, in some embodiments, the method may further include capturing, using the second electronic device, at least a portion of a real-world environment including a light source using one or more input devices of the second electronic device. Additionally or alternatively, in some embodiments, generating and presenting the first content item in three dimensions may include presenting a lighting effect due to an interaction between the first content item and the light source, and updating the appearance of the three-dimensional first content item presented using the second electronic device may include changing the appearance of the lighting effect. Additionally or alternatively, in some embodiments, the appearance of the two-dimensional representation of the first content item displayed on the first electronic device may not include a lighting effect.

[0073] Additionally or alternatively, in some embodiments, the input received at the second electronic device can be an interaction with the first content item in three dimensions, the interaction comprising selecting the first content item in three dimensions using one or more fingers or using one or more input devices of the second electronic device. Additionally or alternatively, in some embodiments, the input received at the second electronic device can be a reorientation of the second electronic device relative to the first content item in three dimensions.

[0074] Additionally or alternatively, in some embodiments, the content creation application may include a two-dimensional representation of the second content item, and the method may further include: generating and presenting the second content item in three dimensions; while presenting the first content item and the second content item in three dimensions using the second electronic device and simultaneously displaying the two-dimensional representations of the first content item and the second content item using the first electronic device: receiving input at the second electronic device; and based on the input, updating the appearance of the first content item and the second content item presented in three dimensions using the second electronic device, and updating the appearance of the two-dimensional representations of the first content item and the second content item displayed on the first electronic device.

[0075] Additionally or alternatively, in some embodiments, the method may also include: establishing a communication link between the first electronic device and the second electronic device; and transmitting content item data corresponding to the first content item from the first electronic device to a three-dimensional graphics rendering application at the second electronic device via the communication link, and transmitting the first content item generated in a three-dimensional manner to the first electronic device.

[0076] Additionally or alternatively, in some embodiments, updating the appearance of the first content item presented in three dimensions using the second electronic device and updating the appearance of the two-dimensional representation of the first content item displayed on the first electronic device may include: modifying content data corresponding to the first content item presented in three dimensions based on input; and transmitting the modified content data corresponding to the first content item presented in three dimensions from the three-dimensional graphics rendering application to the content creation application via a communication link.

[0077] Additionally or alternatively, in some embodiments, establishing the communication link may include: establishing a first communication link between a content creation application at the first electronic device and a preview shell application at the second electronic device; establishing a second communication link between the preview shell application and the preview proxy application at the second electronic device; and establishing a third communication link between the preview proxy application and the three-dimensional graphics rendering application at the second electronic device.

[0078] Additionally or alternatively, in some embodiments, transmitting content item data from a first electronic device to a three-dimensional graphics rendering application at a second electronic device may include: transmitting the content item data from a content creation application at the first electronic device to a preview shell application at the second electronic device via a first communication link; transmitting the content item data from the preview shell application to a preview proxy application at the second electronic device via a second communication link; and transmitting the content item data from the preview proxy application to the three-dimensional graphics rendering application at the second electronic device via a third communication link.

[0079] Some embodiments of the present disclosure relate to a system including a first electronic device and a second electronic device. The system may include: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors. The one or more programs may include instructions for performing any of the above methods.

[0080] Some embodiments of the present disclosure relate to a non-transitory computer-readable storage medium that can store one or more programs comprising instructions that, when executed by one or more processors of a system comprising a first electronic device and a second electronic device, cause the system to perform any of the above methods.

[0081] Some embodiments of the present disclosure relate to a system including a first electronic device and a second electronic device. The system may also include: one or more processors; a memory; and means for executing any of the above methods.

[0082] Some embodiments of the present disclosure relate to an information processing device for use in a system including a first electronic device and a second electronic device. The information processing device may also include means for performing any of the above methods.

[0083] Some embodiments of the present disclosure relate to a method for generating and updating a three-dimensional preview of content. The method may include: at a first electronic device in communication with a display and one or more input devices: displaying, via the display, a content creation application including a two-dimensional representation of a first content item; while displaying the content creation application including the two-dimensional representation of the first content item, receiving, via the one or more input devices, input corresponding to a request to preview the first content item in three dimensions; and, based on the input, transmitting to a second electronic device a request to generate a three-dimensional representation of the first content item at the second electronic device; receiving, at the first electronic device, information corresponding to the three-dimensional representation of the first content item; and updating the appearance of the two-dimensional representation of the first content item displayed at the first electronic device.

[0084] Additionally or alternatively, in some embodiments, the first electronic device may be a laptop computer, a desktop computer, or a tablet computer. Additionally or alternatively, in some embodiments, the second electronic device may be a head-mounted display.

[0085] Additionally or alternatively, in some embodiments, transmitting a request to a second electronic device to generate a three-dimensional representation of a first content item at the second electronic device may include: establishing a communication link between the first electronic device and the second electronic device; and transmitting content item data corresponding to the first content item from the first electronic device to a three-dimensional graphics rendering application at the second electronic device via the communication link.

[0086] Additionally or alternatively, in some embodiments, receiving information corresponding to a three-dimensional representation of the first content item at the first electronic device may include receiving modified content item data corresponding to the three-dimensional representation of the first content item from a three-dimensional graphics rendering application at the content creation application via a communication link.

[0087] Some embodiments of the present disclosure relate to an electronic device. The electronic device may include: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors. The one or more programs may include instructions for performing any of the above methods.

[0088] Some embodiments of the present disclosure relate to a non-transitory computer-readable storage medium that can store one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the aforementioned methods.

[0089] Some embodiments of the present disclosure relate to an electronic device. The electronic device may further include: one or more processors; a memory; and means for executing any of the above methods.

[0090] Some embodiments of the present disclosure relate to an information processing device for use in an electronic device. The information processing device may also include means for performing any of the above methods.

[0091] Some embodiments of the present disclosure relate to a method for generating and updating a three-dimensional preview of content. The method may include: at a first electronic device in communication with a display and one or more input devices: capturing, via the one or more input devices, at least a portion of a real-world environment of a second electronic device including a display content creation application; receiving, from the second electronic device, a request to preview a first content item in three dimensions; and based on the request, generating a three-dimensional representation of the first content item and presenting the three-dimensional representation of the first content item within a three-dimensional environment; transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device; receiving input via the one or more input devices; and based on the input: updating an appearance of the three-dimensional representation of the first content item in the three-dimensional environment; and transmitting the updated appearance of the three-dimensional representation of the first content item to the second electronic device.

[0092] Additionally or alternatively, in some embodiments, the first electronic device may be a head-mounted display. Additionally or alternatively, in some embodiments, the second electronic device may be a laptop computer, a desktop computer, or a tablet computer.

[0093] Additionally or alternatively, in some embodiments, the method may further include presenting a representation of the captured portion of the real-world environment. The representation of the captured portion of the real-world environment may include: a representation of the second electronic device including a representation of the content creation application; and a representation of the first content item displayed in the content creation application.

[0094] Additionally or alternatively, in some embodiments, the method may further include capturing a light source of the real-world environment via one or more input devices. Additionally or alternatively, in some embodiments, presenting the three-dimensional representation of the first content item may include presenting a lighting effect caused by an interaction between the first content item and the light source, and updating the appearance of the three-dimensional representation of the first content item in the three-dimensional environment may include changing the appearance of the lighting effect.

[0095] Additionally or alternatively, in some embodiments, transmitting the updated appearance of the three-dimensional representation of the first content item to the second electronic device may not include information about the lighting effect.

[0096] Additionally or alternatively, in some embodiments, the input received via the one or more input devices can be an interaction with the three-dimensional representation of the first content item, the interaction comprising selecting the three-dimensional representation of the first content item using one or more fingers and / or using the one or more input devices.

[0097] Additionally or alternatively, in some embodiments, the input received via the one or more input devices can be a reorientation or repositioning of the first electronic device within the real-world environment.

[0098] Additionally or alternatively, in some embodiments, the method may further include establishing a communication link between the first electronic device and the second electronic device. Additionally or alternatively, in some embodiments, receiving a request from the second electronic device to preview the first content item in three dimensions may include receiving, at the first electronic device, content item data corresponding to the first content item from the second electronic device at a three-dimensional graphics rendering application via the communication link. Additionally or alternatively, in some embodiments, transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device may include transmitting the first content item generated in three dimensions to the second electronic device via the communication link.

[0099] Additionally or alternatively, in some embodiments, establishing a communication link may include: establishing a first communication link between a preview shell application at the first electronic device and a content creation application at the second electronic device; establishing a second communication link between the preview shell application and the preview proxy application at the first electronic device; and establishing a third communication link between the preview proxy application and the three-dimensional graphics rendering application at the first electronic device.

[0100] Additionally or alternatively, in some embodiments, receiving content item data corresponding to the first content item from the second electronic device at a three-dimensional graphics rendering application may include: receiving the content item data from a content creation application at the second electronic device at a preview shell application at the first electronic device via a first communication link; transmitting the content item data from the preview shell application to the preview proxy application at the first electronic device via a second communication link; and transmitting the content item data from the preview proxy application to the three-dimensional graphics rendering application at the first electronic device via a third communication link.

[0101] Some embodiments of the present disclosure relate to an electronic device. The electronic device may include: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors. The one or more programs may include instructions for performing any of the above methods.

[0102] Some embodiments of the present disclosure relate to a non-transitory computer-readable storage medium that can store one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the aforementioned methods.

[0103] Some embodiments of the present disclosure relate to an electronic device. The electronic device may further include: one or more processors; a memory; and means for executing any of the above methods.

[0104] Some embodiments of the present disclosure relate to an information processing device for use in an electronic device. The information processing device may also include means for performing any of the above methods.

[0105] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments have been chosen and described in order to best illustrate the principles of the invention and its practical application so as to enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suitable for the specific purposes contemplated.

Claims

1. A method comprising: At a first electronic device in communication with a display and one or more input devices: capturing, via the one or more input devices, at least a portion of a real-world environment, the real-world environment including a second electronic device displaying a content creation application and a first content item; receiving a request from the second electronic device to preview the first content item in three dimensions; as well as generating a three-dimensional representation of the first content item in accordance with the request, and presenting the three-dimensional representation of the first content item within a three-dimensional environment; transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device; receiving input via the one or more input devices; Based on the input: updating an appearance of the three-dimensional representation of the first content item in the three-dimensional environment; as well as The updated appearance of the three-dimensional representation of the first content item is transmitted to the second electronic device.

2. The method according to claim 1, wherein: The first electronic device is a head-mounted display; and The second electronic device is a laptop computer, a desktop computer or a tablet computer.

3. The method according to claim 1, further comprising: presenting a representation of the captured portion of the real-world environment, wherein the representation of the captured portion of the real-world environment comprises: a representation of the second electronic device including a representation of the content creation application; and A representation of the first content item is displayed in the content creation application.

4. The method according to claim 1, further comprising: capturing a light source of the real-world environment via the one or more input devices; in: Rendering the three-dimensional representation of the first content item includes rendering a lighting effect resulting from an interaction between the first content item and the light source; Updating the appearance of the three-dimensional representation of the first content item in the three-dimensional environment comprises changing the appearance of the lighting effect. 5 . The method of claim 4 , wherein transmitting the updated appearance of the three-dimensional representation of the first content item to the second electronic device does not include information about the lighting effect.

6. A method according to claim 1, wherein the input received via the one or more input devices is an interaction with the three-dimensional representation of the first content item, the interaction comprising selecting the three-dimensional representation of the first content item using one or more fingers and / or using the one or more input devices. 7 . The method of claim 1 , wherein the input received via the one or more input devices is a reorientation or repositioning of the first electronic device within the real-world environment.

8. The method according to claim 1, further comprising: establishing a communication link between the first electronic device and the second electronic device; Wherein receiving the request for previewing the first content item in three dimensions from the second electronic device comprises: receiving, at a three-dimensional graphics rendering application at the first electronic device via the communication link, content item data corresponding to the first content item from the second electronic device; and wherein transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device comprises: The first content item generated in three dimensions is transmitted to the second electronic device via the communication link.

9. The method of claim 8, wherein establishing the communication link comprises: establishing a first communication link between a preview shell application at the first electronic device and the content creation application at the second electronic device; establishing, at the first electronic device, a second communication link between the preview shell application and the preview proxy application; as well as A third communication link is established at the first electronic device between the preview proxy application and the three-dimensional graphics rendering application.

10. The method of claim 9, wherein receiving, at the three-dimensional graphics rendering application, the content item data corresponding to the first content item from the second electronic device comprises: receiving, at the preview shell application at the first electronic device, the content item data from the content creation application at the second electronic device via the first communication link; transmitting, at the first electronic device, the content item data from the preview shell application to the preview proxy application via the second communication link; as well as The content item data is transmitted from the preview agent application to the three-dimensional graphics rendering application via the third communication link at the first electronic device.

11. An electronic device comprising: one or more processors; Memory; as well as One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing a method including the following operations: capturing, via one or more input devices, at least a portion of a real-world environment, the real-world environment including a second electronic device displaying a content creation application and a first content item; receiving a request from the second electronic device to preview the first content item in three dimensions; as well as generating a three-dimensional representation of the first content item in accordance with the request, and presenting the three-dimensional representation of the first content item within a three-dimensional environment; transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device; receiving input via the one or more input devices; Based on the input: updating an appearance of the three-dimensional representation of the first content item in the three-dimensional environment; as well as The updated appearance of the three-dimensional representation of the first content item is transmitted to the second electronic device.

12. The electronic device of claim 11, wherein the method further comprises: presenting a representation of the captured portion of the real-world environment, wherein the representation of the captured portion of the real-world environment comprises: a representation of the second electronic device including a representation of the content creation application; and A representation of the first content item is displayed in the content creation application.

13. The electronic device of claim 11, wherein the method further comprises: capturing a light source of the real-world environment via the one or more input devices; in: Rendering the three-dimensional representation of the first content item includes rendering a lighting effect resulting from an interaction between the first content item and the light source; Updating the appearance of the three-dimensional representation of the first content item in the three-dimensional environment comprises changing the appearance of the lighting effect. 14 . The electronic device of claim 13 , wherein transmitting the updated appearance of the three-dimensional representation of the first content item to the second electronic device does not include information about the lighting effect.

15. An electronic device according to claim 11, wherein the input received via the one or more input devices is an interaction with the three-dimensional representation of the first content item, the interaction comprising using one or more fingers and / or using the one or more input devices to select the three-dimensional representation of the first content item.

16. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of a first electronic device, cause the first electronic device to perform a method comprising the following operations: capturing, via one or more input devices, at least a portion of a real-world environment, the real-world environment including a second electronic device displaying a content creation application and a first content item; receiving a request from the second electronic device to preview the first content item in three dimensions; as well as generating a three-dimensional representation of the first content item in accordance with the request, and presenting the three-dimensional representation of the first content item within a three-dimensional environment; transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device; receiving input via the one or more input devices; Based on the input: updating an appearance of the three-dimensional representation of the first content item in the three-dimensional environment; as well as The updated appearance of the three-dimensional representation of the first content item is transmitted to the second electronic device.

17. The non-transitory computer-readable storage medium of claim 16, wherein: The first electronic device is a head-mounted display; and The second electronic device is a laptop computer, a desktop computer or a tablet computer.

18. The non-transitory computer-readable storage medium of claim 16, wherein the method further comprises: establishing a communication link between the first electronic device and the second electronic device; Wherein receiving the request for previewing the first content item in three dimensions from the second electronic device comprises: receiving, at a three-dimensional graphics rendering application at the first electronic device via the communication link, content item data corresponding to the first content item from the second electronic device; and wherein transmitting the three-dimensional representation of the first content item to the second electronic device for display at the second electronic device comprises: The first content item generated in three dimensions is transmitted to the second electronic device via the communication link.

19. The non-transitory computer-readable storage medium of claim 18, wherein establishing the communication link comprises: establishing a first communication link between a preview shell application at the first electronic device and the content creation application at the second electronic device; establishing, at the first electronic device, a second communication link between the preview shell application and the preview proxy application; as well as A third communication link is established at the first electronic device between the preview proxy application and the three-dimensional graphics rendering application.

20. The non-transitory computer-readable storage medium of claim 19, wherein receiving, at the three-dimensional graphics rendering application, the content item data corresponding to the first content item from the second electronic device comprises: receiving, at the preview shell application at the first electronic device, the content item data from the content creation application at the second electronic device via the first communication link; transmitting, at the first electronic device, the content item data from the preview shell application to the preview proxy application via the second communication link; as well as The content item data is transmitted from the preview agent application to the three-dimensional graphics rendering application via the third communication link at the first electronic device.

Images