displaying rendered volume representations according to different display modes

Abstract

The present disclosure relates to displaying rendered volumetric representations according to different display modes. A method is performed at an electronic device that includes one or more processors, non-transitory memory, and a display. The method includes rendering a first volumetric object so as to generate first object data. The method includes displaying the first object data on the display according to a first display mode. The first display mode includes displaying the first object data within a two-dimensional (2D) content area. The method includes detecting a request to change from the first display mode to a second display mode. The method includes, in response to detecting the request, displaying the first object data on the display according to the second display mode. The second display mode includes displaying the first object data within a representation of a physical environment.

Description

Technical Field

[0001] This disclosure relates to the display of content, and more particularly to the display of volumetric content according to various display modes. Background Technology

[0002] Devices can display two-dimensional (2D) content items within a 2D content area of ​​a display. In some cases, the device generates and displays a volumetric representation of the 2D content item. Generating a volumetric representation of a 2D content item involves a computationally expensive and time-consuming complex rendering process. In various cases, the device initiates the rendering process in response to receiving a display request. Therefore, relatively large latency is associated with the transition between receiving a display request, generating a volumetric representation, and displaying the volumetric representation. Relatively large latency often results in unwanted visual artifacts, thus degrading the user experience. Summary of the Invention

[0003] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, and a display. The method includes rendering a first volumetric object to generate first object data. The method includes displaying the first object data on the display according to a first display mode. The first display mode includes displaying the first object data within a 2D content area. The method includes detecting a request to change from the first display mode to a second display mode. The method includes, in response to the detection request, displaying the first object data on the display according to the second display mode. The second display mode includes displaying the first object data within a representation of the physical environment.

[0004] According to some embodiments, an electronic device includes one or more processors, non-transitory memory, and a display. One or more programs are stored in the non-transitory memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing operations to be performed in any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the electronic device, cause the device to perform or cause operations to be performed in any of the methods described herein. According to some embodiments, an electronic device includes means for performing or causing operations to be performed in any of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device includes means for performing or causing operations to be performed in any of the methods described herein. Attached Figure Description

[0005] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0006] Figure 1This is a block diagram of an example of a portable multi-functional device according to some embodiments.

[0007] Figures 2A to 2J This is an example of rendering a volume object before performing a corresponding display mode transition, according to some embodiments.

[0008] Figure 3 This is an example of a block diagram of a system for displaying rendered volumetric objects according to different display modes, based on some embodiments.

[0009] Figure 4 This is an example of a flowchart illustrating a method for displaying a rendered volumetric object according to different display modes, based on some embodiments.

[0010] Figure 5 This is an example of a flowchart illustrating a method for manipulating a rendered volumetric object according to some embodiments. Detailed Implementation

[0011] Devices can display 2D content items within a 2D content area of ​​a display, such as displaying 2D thumbnails within a webpage. In some cases, the device generates and displays a volumetric representation of the 2D content item. For example, when displaying a 2D content item within a 2D content area, the device receives input requesting the display of a volumetric (e.g., three-dimensional (3D)) representation of the 2D content item. Based on this input, the device generates and displays the volumetric representation of the 2D content item. Generating the volumetric representation of the 2D content item involves a computationally expensive and time-consuming rendering process. Therefore, relatively large latency is associated with the transition between receiving input, generating the volumetric representation, and displaying the volumetric representation. Relatively large latency often results in unwanted visual artifacts, thus degrading the user experience.

[0012] In contrast, the various embodiments disclosed herein include methods, systems, and electronic devices for rendering volumetric objects before performing a display mode transition associated with the display volumetric object, thereby providing a more seamless transition between display modes. To this end, the electronic device renders the volumetric object to generate object data. The object data indicates a volumetric representation of the volumetric object. Furthermore, the electronic device displays the object data according to a first display mode, including displaying the first object data within a 2D content area. For example, the electronic device displays a rendered 3D model of a sofa within a webpage. Subsequently, the electronic device detects a request to change from the first display mode to a second display mode. Based on this request, the electronic device displays the first object data according to the second display mode, including displaying the first object data (e.g., a rendered 3D model of the sofa) within a representation of the physical environment. The representation of the physical environment provides a 3D representation of the physical environment. Compared to other devices, generating object data before detecting a request enables a faster and more seamless (e.g., fewer visual artifacts) display mode transition.

[0013] describe Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0014] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0015] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and in the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0016] As used herein, depending on the context, the term "if" is optionally interpreted as meaning "when," "at," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is optionally interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected," or "in response to the detection of [the stated condition or event]."

[0017] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

[0018] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some embodiments, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0019] Figure 1This is a block diagram of an example of a portable multi-functional device 100 (sometimes referred to herein as "electronic device 100" for brevity) according to some embodiments. Electronic device 100 includes a memory 102 (e.g., one or more non-transitory computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an input / output (I / O) subsystem 106, a display system 112, an inertial measurement unit (IMU) 130, an image sensor 143 (e.g., a camera), a contact strength sensor 165, an audio sensor 113 (e.g., a microphone), an eye-tracking sensor 164 (e.g., included within a head-mounted device (HMD)), a limb-tracking sensor 150, and other input or control devices 116. In some embodiments, electronic device 100 corresponds to one of a mobile phone, tablet computer, laptop computer, wearable computing device, head-mounted device (HMD), head-mounted housing (e.g., electronic device 100 slides to or is otherwise attached to a head-mounted housing), etc. In some embodiments, the head-mounted housing is shaped to form a receiver for receiving an electronic device 100 having a display.

[0020] In some embodiments, the peripheral device interface 118, one or more processing units 120, and memory controller 122 are optionally implemented on a single chip, such as chip 103. In some other embodiments, they are optionally implemented on separate chips.

[0021] I / O subsystem 106 couples input / output peripherals on electronic device 100, such as display system 112 and other input or control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, image sensor controller 158, intensity sensor controller 159, audio controller 157, eye-tracking controller 160, one or more input controllers 152 for other input or control devices, IMU controller 132, limb tracking controller 180, and privacy subsystem 170. One or more input controllers 152 receive electrical signals from / send electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 152 may optionally be coupled (or not coupled) to any of the following: a keyboard, an infrared port, a universal serial bus (USB) port, a stylus, a finger wearable device, and / or a pointing device such as a mouse. One or more buttons may optionally include push-buttons. In some embodiments, other input or control devices 116 include a positioning system (e.g., GPS) that obtains information about the position and / or orientation of the electronic device 100 relative to a particular object. In some embodiments, other input or control devices 116 include depth sensors and / or time-of-flight sensors that obtain depth information characterizing physical objects within the physical environment. In some embodiments, other input or control devices 116 include an ambient light sensor that senses ambient light from the physical environment and outputs corresponding ambient light data.

[0022] Display system 112 provides input and output interfaces between electronic device 100 and user. Display controller 156 receives electrical signals from display system 112 and / or sends electrical signals to display system 112. Display system 112 displays visual output to user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (sometimes referred to herein as "computer-generated content"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "enabled representation" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0023] Display system 112 may have a touch-sensitive surface, sensor, or sensor array to accept input from a user based on tactile and / or tactile contact. Display system 112 and display controller 156 (and any associated modules and / or instruction sets in memory 102) detect contact on display system 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on display system 112. In an exemplary embodiment, the contact point between display system 112 and the user corresponds to the user's finger or a finger-wearable device.

[0024] Display system 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Display system 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed thereafter, as well as other proximity sensor arrays or other elements for determining one or more points of contact with display system 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies.

[0025] Users may optionally use any suitable object or accessory, such as a stylus, a finger wearable device, or a finger, to interact with the display system 112. In some embodiments, the user interface is designed to work with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on a touchscreen. In some embodiments, the electronic device 100 translates coarse finger-based input into precise pointer / cursor positions or commands to perform the actions desired by the user.

[0026] The audio circuitry also receives electrical signals converted from sound waves by the audio sensor 113 (e.g., a microphone). The audio circuitry converts the electrical signals into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved by the peripheral interface 118 from and / or transmitted to the memory 102 and / or the RF circuitry. In some embodiments, the audio circuitry also includes a headset jack. This headset jack provides an interface between the audio circuitry and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a mono or binaural headset) and input (e.g., a microphone).

[0027] The inertial measurement unit (IMU) 130 includes an accelerometer, a gyroscope, and / or a magnetometer to measure various force, angular rate, and / or magnetic field information relative to the electronic device 100. Therefore, according to various embodiments, the IMU 130 detects one or more positional change inputs of the electronic device 100, such as the electronic device 100 being rocked, rotated, or moved in a particular direction.

[0028] Image sensor 143 captures still images and / or video. In some embodiments, the optical sensor is located on the back of the electronic device 100, opposite to the touchscreen on the front of the electronic device 100, allowing the touchscreen to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another image sensor 143 is located on the front of the electronic device 100, enabling the acquisition of images of the user (e.g., for selfies, for video conferencing while the user is viewing other video conferencing participants on the touchscreen, etc.). In some embodiments, the image sensor is integrated within the HMD. For example, image sensor 143 outputs image data representing physical objects (e.g., physical agents) within the physical environment.

[0029] A contact strength sensor 165 detects the strength of a contact on electronic device 100 (e.g., a touch input on a touch-sensitive surface of electronic device 100). The contact strength sensor 165 is coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the physical environment. In some embodiments, at least one contact strength sensor 165 is arranged juxtaposed with or adjacent to the touch-sensitive surface of electronic device 100. In some embodiments, at least one contact strength sensor 165 is located on the side of electronic device 100.

[0030] Eye-tracking sensor 164 detects the eye gaze of a user of electronic device 100 and generates eye-tracking data indicating the user's gaze location. In various embodiments, the eye-tracking data includes data indicating a fixed point (e.g., a point of attention) of the user on a display panel, such as a display panel within a head-mounted device (HMD), a head-mounted housing, or a head-up display.

[0031] The limb tracking sensor 150 acquires limb tracking data indicating the position of a user's limbs. For example, in some embodiments, the limb tracking sensor 150 corresponds to a hand tracking sensor that acquires hand tracking data indicating the position of a user's hand or fingers within a particular object. In some embodiments, the limb tracking sensor 150 utilizes computer vision techniques to estimate limb pose based on camera images.

[0032] In various embodiments, electronic device 100 includes a privacy subsystem 170 that includes one or more privacy setting filters associated with user information, such as user information included in limb tracking data, eye gaze data, and / or body position data associated with the user. In some embodiments, privacy subsystem 170 selectively prevents and / or restricts electronic device 100 or portions thereof from acquiring and / or transmitting user information. To this end, privacy subsystem 170 receives user preferences and / or choices from the user in response to prompting the user to make user preferences and / or choices. In some embodiments, privacy subsystem 170 prevents electronic device 100 from acquiring and / or transmitting user information unless and until privacy subsystem 170 obtains informed consent from the user. In some embodiments, privacy subsystem 170 anonymizes (e.g., scrambles or obfuscates) certain types of user information. For example, privacy subsystem 170 receives user input specifying which types of user information privacy subsystem 170 anonymizes. As another example, privacy subsystem 170 independently (e.g., automatically) anonymizes certain types of user information that may include sensitive and / or identifiable information.

[0033] Figures 2A to 2J This is an example of rendering a volumetric object before performing a corresponding display mode transition, according to some embodiments. For example... Figure 2A As shown, the physical environment 200 includes a physical wall 202, a physical sideboard 204, and a user 50 holding an electronic device 210. The electronic device 210 includes a display 212 associated with a visible area 214 of the physical environment 200. The visible area 214 includes a portion of the physical wall 202 and the physical sideboard 204. In some embodiments, the electronic device 210 corresponds to a mobile device, such as a smartphone, tablet, wearable device, or others. In some embodiments, the electronic device 210 corresponds to a head-mounted device (HMD) that may include one or more opaque or see-through displays.

[0034] In some embodiments, electronic device 210 includes an image sensor, such as a scene camera. The image sensor can capture image data characterizing the physical environment 200. The image data can correspond to images or image sequences (e.g., video streams). See reference... Figure 3The electronic device 210 may include a compositing system that combines image data with object data, wherein the object data corresponds to a rendered volumetric (e.g., 3D) object.

[0035] like Figure 2B As shown, on display 212, electronic device 210 displays various object data within 2D content area 220 according to a first display mode. The various object data include first object data corresponding to the volume of a car 222, second object data corresponding to the volume of a fruit bowl 224, and third object data corresponding to the volume of a table 226. To this end, electronic device 210 renders the first volume object to generate first object data, renders the second volume object to generate second object data, and renders the third volume object to generate third object data. For example, electronic device 210 includes a graphics processing unit (GPU) to perform the rendering. Specific volume objects can be defined in a scene file, which describes the geometry, viewpoint, texture, lighting, and shadow information characterizing the specific volume object.

[0036] The 2D content area 220 may correspond to an application window, such as the canvas of a webpage or drawing application. In some embodiments, the 2D content area 220 includes 2D content in addition to including volume objects 222 to 226 (“3D objects”). For example, as Figure 2B As shown, the 2D content area 220 includes column header text ("3D Object;", "Description;", and "Place Objects in Your Space") and text describing the volumetric objects ("Virtual Sports Car;", "Virtual Fruit Bowl;", and "Virtual Table"). As another example, such as... Figure 2B As shown, the 2D content area 220 includes a first power indicator 230 associated with the volume of the car 222, a second power indicator 232 associated with the volume of the fruit bowl 224, and a third power indicator 234 associated with the volume of the table 226. Details regarding the power indicators are provided below.

[0037] In some embodiments, when various object data are displayed according to a first display mode, the electronic device 210 does not display a representation of the physical environment 200. Therefore, in some embodiments, when various object data are displayed according to the first display mode, the electronic device 210 does not activate its image sensor, and thus the image sensor does not capture image data characterizing the physical environment 200.

[0038] like Figure 2CAs shown, electronic device 210 receives a first manipulation input 240 relating to volume table 226 via one or more input devices. Specifically, the first manipulation input 240 corresponds to a 90-degree clockwise rotation of volume table 226. For example, one or more input devices may include a limb tracker that tracks the clockwise rotation of user 50's hand. Based on the first manipulation input 240, electronic device 210 updates the rendering of a third volume object to generate updated third object data. Figure 2D As shown, with Figure 2C Compared to the volume table 226 shown, on the display 212, the electronic device 210 displays updated third object data corresponding to the volume table 226 rotated 90 degrees clockwise.

[0039] like Figure 2E As shown, electronic device 210 receives a first request 242 relating to a third display representation 234. The first request 242 requests electronic device 210 to change a first display mode to a second display mode. Furthermore, the first request 242 requests electronic device 210 to display updated third object data according to the second display mode. For example, electronic device 210 tracks the user 50's finger spatially relating to the third display representation 234. As another example, electronic device 210 tracks the user 50's eye gaze spatially relating to the third display representation 234. In some embodiments, the second display mode includes a representation of the physical environment. For example, the second display mode is characterized by an augmented reality (AR) environment or a mixed reality (MR) environment. Those skilled in the art will understand that the request to change the display mode can correspond to various input types, such as hardware input (e.g., pressing a button), voice input from user 50 (e.g., "Show the virtual table placed in my room"), or limb / gaze selection of a specific volumetric object (rather than a selection of the corresponding display representation).

[0040] like Figure 2F As shown, in response to receiving the first request 242, the electronic device 210 displays updated third object data on the display 212 according to a second display mode. It is noteworthy that the second display mode includes the updated third object data displayed within a representation of the physical environment 200. For example, the representation of the physical environment 200 generally corresponds to the visible area 216 associated with the display 212. Therefore, the representation of the physical environment 200 includes a representation of the physical sideboard 204 and a portion of the physical wall 202. To this end, in some embodiments, in response to receiving the first request 242, the electronic device 210 activates an image sensor that captures image data characterizing the physical environment 200. Therefore, the image data corresponds to the representation of the physical environment 200. Furthermore, the electronic device 210 synthesizes the image data with the updated third object data to generate display data and sends the synthesized result to the display 212 for display.

[0041] Compared to other devices, rendering the volumetric object before receiving a request (e.g., first request 242) allows the electronic device 210 to provide a more seamless transition between display modes. For example, the volumetric object can be pre-rendered as 3D content and displayed alongside 2D content, such as a 2D webpage or a 2D canvas. Based on receiving the first request 242, the electronic device 210 can transition from a first display mode to a second display mode without re-rendering the volumetric object, thus providing a seamless transition. For example, the electronic device 210 continuously displays the volumetric object across display modes, so that the volumetric object does not disappear and reappears during the transition. As described above, in some embodiments, the first display mode includes volumetric objects that are not directly superimposed on image data of the physical environment (e.g., a live camera feed), while the second display mode includes volumetric objects that are directly superimposed on at least a portion of the image data. In contrast, other devices may display a 2D representation of a table (e.g., a thumbnail of the table) within a content area and will not render a 3D object of the table until a display mode change request is received. Therefore, other devices introduce a delay between receiving the display mode change request and displaying the rendered 3D table. Furthermore, other devices may display unwanted visual artifacts before rendering is complete.

[0042] like Figure 2G As shown, although in the first display mode, the electronic device 210 receives a second request 244 concerning the second display representation 232 within the 2D content area 220. The second request 244 requests the electronic device 210 to change from the first display mode to the second display mode. Furthermore, the second request 244 requests the electronic device 210 to display second object data according to the second display mode. In response to receiving... Figure 2G In the second request 244, on the display 212, the electronic device 210 displays the second object data overlaid on the physical sideboard 204, such as... Figure 2H As shown. The second object data corresponds to the volume of the fruit bowl 224. To this end, in some embodiments, the electronic device 210 performs computer vision techniques relative to the image data to identify the physical cutlery cabinet 204 within the image data and overlays the second object data on the portion of the image data representing the physical cutlery cabinet 204.

[0043] like Figure 2IAs shown, electronic device 210 receives a second manipulation input 246 relating to a volumetric fruit bowl 224 via one or more input devices. The second manipulation input 246 corresponds to a request to move the volumetric fruit bowl 224 to the left along the surface of the physical sideboard 204. For example, the second manipulation input 246 is a leftward movement of the user 50's hand, originating from a position corresponding to the volumetric fruit bowl 224. Electronic device 210 may include a limb tracker to track the movement of the user 50's hand. Based on the second manipulation input 246, electronic device 210 updates the rendering of the second volumetric object to generate updated second object data. The updated second object data is displayed on display 212, such as... Figure 2J As shown.

[0044] Figure 3 This is an example of a block diagram of a system 310 for displaying rendered volumetric objects according to different display modes, based on some embodiments. In various embodiments, system 310 or a portion thereof is integrated into an electronic device, such as a reference numeral. Figures 2A to 2J The described electronic device 210.

[0045] System 310 includes a rendering subsystem 352 that renders one or more volume objects to generate object data 354. Each portion of the object data 354 provides a volume (e.g., 3D) representation of a corresponding volume object within the volume objects. For example, refer to... Figure 2B The first portion of object data 354 corresponds to the volume of car 222, and the second portion of object data 354 corresponds to the volume of fruit bowl 224. In some embodiments, the rendering subsystem 352 obtains volume objects from a volume object data repository 350 (such as a buffer or other non-transitory memory). In some embodiments, the rendering subsystem 352 includes a GPU that performs rendering. According to various embodiments, the system 310 includes one or more input devices 360 that detect manipulation input 368, and the rendering subsystem 352 updates the rendering based on the manipulation input 368.

[0046] For example, input device 360 ​​includes a limb tracker 362 that receives manipulation input 368. Based on the manipulation input 368, the limb tracker 362 can detect when a user's limb is involved with a specific displayed volumetric object, such as when the user's hand is less than a threshold distance from the specific displayed volumetric object. Therefore, system 310 selects the specific displayed volumetric object. Furthermore, the limb tracker 362 can detect limb movement, and rendering subsystem 352 updates the rendering of the specific displayed volumetric object accordingly. As an example, such as Figure 2I As shown, the limb tracker 362 receives a second manipulation input 246 to select the volume fruit bowl 224, and the rendering subsystem 352 updates the rendering accordingly to move the volume fruit bowl 224 to the left across the physical sideboard 204 on the display 212.

[0047] As another example, input device 360 ​​includes position sensor 364, such as an IMU, a touch sensor (e.g., included on a touch-sensitive surface), a magnetic sensor, and / or others. Position sensor 364 receives manipulation input 368 and detects position changes of system 310 based on manipulation input 368. For example, refer to... Figure 2F Based on the leftward rotation of the electronic device 210, the rendering subsystem 352 updates the rendering of the volume table 226, making the volume table 226 appear to move to the right across the display 212 accordingly.

[0048] In some embodiments, system 310 includes a demultiplexer 370. The demultiplexer 370 switches between a first state and a second state based on a display mode change request 372. The first state is associated with a first display mode, while the second state is associated with a second display mode. In the first state, the demultiplexer 370 outputs object data 354 to a compositing subsystem 322, which generates first display data 324 associated with the first display mode. In the second state, the demultiplexer 370 outputs object data 354 to a compositing subsystem 340 (or optionally first to a processing subsystem 356), which generates second display data 342 associated with the second display mode. For example, the display mode change request 372 corresponds to an input spatially relating to a volumetric object displayed within a 2D content area, or spatially relating to a volumetric object (such as...) Figure 2E The first request 242 shown is an input associated with a power indication. As another example, the display mode change request 372 corresponds to a hardware input (e.g., pressing a button). As another example, the display mode change request 372 corresponds to a predetermined position change of the system 310, such as an oscillation of the system 310 detected by the position sensor 364.

[0049] According to various embodiments, on display 380, system 310 displays object data 354 according to a first display mode. The first display mode includes displaying the first object data 354 within a 2D content area. For example, the 2D content area includes the outer boundary of an application window, such as a web page window or a drawing application window. The 2D content area is indicated within 2D content data 320. The 2D content data 320 may also indicate 2D content. For example, based on a request for a specific web page, system 310 obtains 2D content data 320 indicating a specific web page from a web server. The 2D content data 320 may also indicate 2D content, such as text or thumbnails within a specific web page. As another example, system 310 obtains 2D content data 320 from local storage, such as from a buffer utilized by a valid drawing application. To display object data 354 according to the first display mode, system 310 includes a combination subsystem 322. The combination subsystem 322 combines the 2D content data 320 with the object data 354 to generate first display data 324. For example, refer to... Figure 2B The first display data 324 indicates the volume of car 222 within the 2D content area 220.

[0050] Compared to the first display mode, the second display mode includes displaying object data 354 within a representation of the physical environment 300 on the display 380. For this purpose, in some embodiments, the system 310 includes an image sensor 330 that captures image data 332 characterizing the physical environment 300. The image data 332 indicates the representation of the physical environment 300. Further for this purpose, the system 310 includes a compositing subsystem 340 that combines the image data 332 with the object data 354 to generate second display data 342. The second display data 342 indicates one or more volumetric objects within the representation of the physical environment 300. For example, refer to... Figure 2H The second display data 342 indicates the volume of the table 226 and the volume of the fruit bowl 224 within the representation of the physical environment 200, which includes the representation of the physical sideboard 204.

[0051] In some embodiments, before compositing object data 354, system 310 processes at least a portion of object data 354. For this purpose, system 310 includes a processing subsystem 356. For example, processing subsystem 356 processes object data 354 to change the appearance of the corresponding rendered volumetric object, such as resizing, repositioning, or changing the color of the rendered volumetric object. For example, system 310 resizes the rendered volumetric object before displaying it within a representation of the physical environment. As another example, see reference... Figure 2GThe electronic device 210 processes second object data corresponding to the volume of the fruit bowl 224. Furthermore, the electronic device 210 synthesizes the processed second object data with a representation of the physical environment 200 to generate... Figure 2H The corresponding display data in. Therefore, Figure 2H The volume of the fruit bowl 224 shown (second display mode) is smaller than Figure 2G The volume of the fruit bowl 224 shown is in the first display mode.

[0052] Figure 4 This is an example of a flowchart illustrating a method 400 for displaying a rendered volumetric object according to different display modes, based on some embodiments. In various embodiments, method 400 or a portion thereof is performed by an electronic device (e.g., electronic device 210). In various embodiments, method 400 or a portion thereof is performed by... Figure 3 The system 310 shown is executed. In various embodiments, method 400 or a portion thereof is executed by a mobile device, such as a smartphone, tablet, or wearable device. In various embodiments, method 400 or a portion thereof is executed by a head-mounted device (HMD) including a display. In some embodiments, method 400 is executed by processing logic components, including hardware, firmware, software, or a combination thereof. In some embodiments, method 400 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0053] As indicated by box 402, method 400 includes rendering a first volumetric object to generate first object data. In some embodiments, the first object data indicates multiple images (e.g., a video stream) representing the first volumetric object at different times. For example, each of the multiple images represents a first volumetric object from a corresponding perspective among multiple viewing perspectives. As an example, such as Figure 2C As shown, electronic device 210 generates and displays a first image of the volume table 226 from a first viewing perspective view, while electronic device 210 from... Figure 2D The second perspective view generates and displays a second image of volume table 226.

[0054] As indicated by box 404, method 400 includes displaying first object data according to a first display mode. As indicated by box 406, the first display mode includes displaying the first object data within a 2D content area, such as simultaneously displaying the first object data within a 2D web application window. Therefore, in some embodiments, method 400 includes obtaining 2D content data indicating the 2D content area, combining the 2D content data with the first object data to generate first display data, and sending the first display data to a display. For example, see reference... Figure 3The combination subsystem 322 generates first display data 324 by combining object data 354 with 2D content data 320. The first display data indicates a first volumetric object within the 2D content area. For example, as... Figure 2G As shown, on display 212, electronic device 210 displays second object data (corresponding to the volume of fruit bowl 224) within a 2D content area 220. In some embodiments, displaying the first object data according to a first display mode includes displaying a specific image of the first object data.

[0055] As indicated by box 408, in some embodiments, 2D content data includes 2D content within a 2D content region. For example, refer to... Figure 2B The 2D content area 220 includes 2D tables, various 2D texts (e.g., "3D object;", "virtual sports car"), and 2D display representations 230 to 2D display representations 234. In some embodiments, as indicated by box 410, displaying the first object data according to a first display mode includes arranging the first object data relative to the 2D content. For example, arranging the first object data corresponds to displaying the first object data next to the 2D content within the 2D content area, rather than displaying a text link or 2D representation (e.g., thumbnail) of the volumetric object. For example, refer to... Figure 2B The electronic device 210 displays third object data (corresponding to volume table 226) next to the corresponding descriptive text (“virtual table”).

[0056] As indicated by box 412, method 400 includes detecting a request to change from a first display mode to a second display mode. For example, the second display mode corresponds to an augmented reality (AR) environment or a mixed reality (MR) environment. As indicated by box 414, in some embodiments, detecting the request includes receiving input relating to a first volumetric object displayed within a 2D content area. For example, refer to... Figure 3 The limb tracker 362 detects a user's limbs involving a first volume object. As another example, a touch sensor detects touch input from the user involving the first volume object. In some embodiments, the detection request is based on the input involving the first volume object for at least a threshold amount of time. In some embodiments, the request corresponds to input involving an indication associated with the first volume object. For example, refer to... Figure 2E Electronic device 210 receives a first request 242 relating to a third power indicator 234 associated with volume table 226.

[0057] As indicated by box 416, in some embodiments, in response to a detection request, method 400 includes processing first object data. For example, processing includes changing the appearance of the first object data, such as a reset operation or a repositioning operation. As an example, in response to a detection... Figure 2EIn the first request 242, electronic device 210 processes third object data to generate a larger version of volume table 226. Therefore, the volume table 226 is displayed according to the second display mode. Figure 2F The table 226 (shown in the image) is larger than the volume displayed according to the first display mode. Figure 2E (as shown in the figure). In some embodiments, the processing is performed by Figure 3 The processing subsystem 356 is executed.

[0058] As indicated by box 418, in response to detecting a request, method 400 includes displaying first object data according to a second display mode. As indicated by box 420, the second display mode includes displaying the first object data within a representation of the physical environment. For this purpose, in some embodiments, method 400 comprises an image sensor (e.g., Figure 3 The method is performed by an electronic device (image sensor 330) that captures image data characterizing the physical environment. The image data indicates a representation of the physical environment. Further for this purpose, in some embodiments, method 400 includes synthesizing the image data with first object data (e.g., via...). Figure 3 The synthesis subsystem 322 in the middle is used to generate second display data. The second display data indicates a first volumetric object within a representation of the physical environment. Further for this purpose, in some embodiments, method 400 includes sending the second display data to a display. For example, in response to detection... Figure 2E In the first request 242, electronic device 210 activates the camera to capture image data of the physical environment 200. Continuing this example, on display 212, electronic device 210 displays a representation of the physical environment 200, including a representation of the physical sideboard 204, as shown below. Figure 2F As shown. In some embodiments, the image data includes an image sequence. In some embodiments, the image data corresponds to transitive image data. In some embodiments, displaying the first object data according to a second display mode includes stopping the display of the 2D content area while maintaining the display of the first object data.

[0059] The physical environment is associated with the electronic device performing method 400; for example, the physical environment includes the electronic device. In some embodiments, when displaying first object data according to a second display mode, method 400 includes locking the rendered first volumetric object world to points or regions represented by the physical environment. For example, method 400 includes utilizing Simultaneous Localization and Mapping (SLAM) to perform world locking. Compared to a 2D content area, the representation of the physical environment may correspond to a 3D representation of the physical environment.

[0060] In some embodiments, displaying the first object data according to the second display mode includes displaying a specific image of the first object data. To this end, in some embodiments, method 400 includes storing the specific image in non-transitory memory in response to a detection request, and retrieving the specific image from the transient memory to display the specific image within a representation of the physical environment. By retrieving the specific image from non-transitory memory instead of re-rendering the first volumetric object, the electronic device provides a faster and more seamless transition from the first display mode to the second display mode.

[0061] As indicated by box 422, in some embodiments, method 400 includes detecting completion of processing of the first object data before transitioning to a second display mode. For example, refer to... Figure 3 Before sending the second display data 342 to the display 380, the compositing subsystem 340 obtains a processing termination indicator from the processing subsystem 356. As an example, the processing termination indicator is part of a fencing process (e.g., implemented at the operating system (OS) level) and is used to facilitate the scheduling of multiple animation processes relative to each other. As an example, based on detecting a request to change from a first display mode to a second display mode, method 400 includes determining that a first animation process that increases the 2D content area is ready to start, and determining that a second animation process that increases the size of the first object data is also ready to start. To provide a seamless transition from the first display mode to the second display mode, one animation process in the animation process registers a fencing with the OS and pauses its animation. Furthermore, the animation process sends a processing termination indicator (e.g., a fencing identifier) ​​to other animation processes, thereby signaling the other animation processes to also register a fencing with the OS and pause the start of their animation. Once the OS receives both fencings, the OS can fencing and simultaneously start the animation processes to achieve a seamless transition from the first display mode to the second display mode.

[0062] Figure 5 This is an example flowchart of a method 500 for manipulating a rendered volumetric object according to some embodiments. In various embodiments, method 500 or a portion thereof is performed by an electronic device (e.g., electronic device 210). In various embodiments, method 500 or a portion thereof is performed by... Figure 3 The system 310 shown is executed. In various embodiments, method 500 or a portion thereof is executed by a mobile device, such as a smartphone, tablet, or wearable device. In various embodiments, method 500 or a portion thereof is executed by an HMD including a display. In some embodiments, method 500 is executed by processing logic components, including hardware, firmware, software, or a combination thereof. In some embodiments, method 500 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0063] As indicated by box 502, method 500 includes rendering a first volume object to generate first object data, and rendering a second volume object to generate second object data. For example, see reference... Figure 3 The rendering subsystem 352 obtains a first volume object and a second volume object from the volume object data repository 350. Furthermore, the rendering subsystem 352 generates a first portion of object data 354 corresponding to the first volume object, and generates a second portion of object data 354 corresponding to the second volume object.

[0064] As indicated by box 504, method 500 includes displaying first object data and second object data according to a first display mode. As indicated by box 506, the first display mode includes displaying the first object data and second object data within a 2D content area. For example, refer to... Figure 2B The electronic device 210 displays first object data (corresponding to the volume of car 222) and second object data (corresponding to the volume of fruit bowl 224) within the 2D content area 220.

[0065] As indicated by box 508, in some embodiments, method 500 includes manipulating the display of at least one of a first volumetric object and a second volumetric object within a 2D content area. To this end, method 500 includes updating the rendering of the first object based on manipulation input to generate updated first object data, and replacing the first object data with the updated first object data on a display. For example, an electronic device receives manipulation input via one or more input devices.

[0066] As an example, as shown in box 510 and referenced Figure 3 The limb tracker 362 receives manipulation input 368 and outputs corresponding limb tracking data to the rendering subsystem 352. The limb tracker 362 tracks the user's limbs, such as the user's fingers or hands. In some embodiments, the manipulation input spatially relates to a specific volumetric object on the display. For example, the distance between the manipulation input and the specific volumetric object is less than a threshold distance. As an example, refer to... Figure 2C and Figure 2D The electronic device 210 receives a first manipulation input 240 relating to the volume table 226, and thus updates the rendering of the volume table 226 in order to rotate the volume table 226.

[0067] As indicated in box 512, method 500 includes detecting a request to change a first display mode to a second display mode, such as referencing Figure 4The method 400 shown is described in block 412. As indicated in block 514, in response to a detection request, method 400 includes displaying first object data and second object data according to a second display mode. As indicated in block 516, the second display mode includes displaying the first object data and second object data within a representation of the physical environment. For example, in response to receiving... Figure 2G In the second request 244, electronic device 210 displays second object data (corresponding to the volume of fruit bowl 224) and third object data (corresponding to the volume of table 226) within the representation of physical environment 200. (See reference) Figure 4 Box 420 of method 400 shown describes additional details regarding the display of object data within a representation of the physical environment.

[0068] As indicated by box 518, in some embodiments, method 500 includes manipulating at least one of a first volumetric object and a second volumetric object displayed within a representation of the physical environment. For example, as indicated by box 520, method 500 includes utilizing limb tracking to manipulate a specific volumetric object. As an example, refer to... Figure 2I and Figure 2J The electronic device 210 receives a second manipulation input 246 relating to the volume fruit bowl 224, and thus updates the rendering of the volume fruit bowl 224 to move the volume fruit bowl 224 to the left across the surface of the physical sideboard 204.

[0069] As another example, as shown in box 522, method 500 includes detecting changes in the position of an electronic device in order to manipulate a specific volumetric object. For example, refer to... Figure 3 Position sensor 364 receives manipulation input 368 and outputs corresponding device position data to rendering subsystem 352. For example, the position data indicates rotational or translational movement of the electronic device. For instance, position sensor 364 corresponds to an IMU that detects rotational and / or positional changes in system 310. As another example, refer to... Figure 2J If user 50 walks closer to physical wall 202, electronic device 210 will update the rendering of the second object data to enlarge the appearance of the volume fruit bowl 224, and update the rendering of the third object data to enlarge the appearance of the volume table 226.

[0070] This disclosure describes various features, none of which alone can achieve the benefits described herein. It should be understood that the various features described herein can be combined, modified, or omitted, as will be apparent to those skilled in the art. Other combinations and sub-combinations beyond those specifically described herein will be apparent to those skilled in the art and are intended to form part of this disclosure. Various methods are described herein in conjunction with various flowchart steps and / or stages. It should be understood that in many cases, certain steps and / or stages can be combined such that multiple steps and / or stages shown in the flowchart can be performed as a single step and / or stage. Additionally, certain steps and / or stages can be divided into additional sub-components to be performed independently. In some cases, the order of steps and / or stages can be rearranged, and certain steps and / or stages can be omitted entirely. Furthermore, the methods described herein should be understood to be broadly interpretable, such that additional steps and / or stages beyond those shown and described herein can also be performed.

[0071] Some or all of the methods and tasks described herein can be performed and fully automated by a computer system. In some cases, the computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate via a network to perform the functions described herein. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in memory or other non-transitory computer-readable storage media or devices. The various functions disclosed herein may be implemented in such program instructions, but alternatively, some or all of the disclosed functions may be implemented in the computer system's dedicated circuitry (e.g., ASIC, FPGA, or GP-GPU). In cases where the computer system includes multiple computing devices, these devices may be located in the same location or not. The results of the disclosed methods and tasks can be persistently stored by converting physical storage devices such as solid-state memory chips and / or disks into different states.

[0072] The various processes defined herein take into account options for obtaining and using users' personal information. For example, such personal information may be used to provide improved privacy screens on electronic devices. However, the extent to which such personal information is collected should be based on the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0073] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0074] However, users can limit the extent to which parties can access or otherwise obtain their personal information. For example, settings or other preferences can be adjusted so that users can decide whether their personal information can be accessed by various entities. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0075] This disclosure is not intended to be limited to the embodiments shown herein. Various modifications to the embodiments described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of this disclosure. The teachings of the invention provided herein can be applied to other methods and systems, and are not limited to those described above, and elements and actions of the various embodiments described above can be combined to provide further embodiments. Therefore, the novel methods and systems described herein can be implemented in many other forms; furthermore, various omissions, substitutions, and changes can be made to the form of the methods and systems described herein without departing from the spirit of this disclosure. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of this disclosure.

Claims

1. A method for rendering an object model, comprising: In electronic devices that include one or more processors, non-transitory memory, and displays: Render the three-dimensional 3D object model of the first object in order to generate the first 2D image data of the first object; Displaying the first 2D image data of the first object on the display according to a first display mode, wherein the first display mode includes displaying the first 2D image data of the first object within a 2D content area; and While displaying the first 2D image data of the first object according to the first display mode: Receive first control input via one or more input devices; as well as Render the 3D object model of the first object based on the first manipulation input to generate a second 2D image data of the first object; The second 2D image data of the first object is displayed on the display according to the first display mode; as well as In response to a request to change from the first display mode to the second display mode, the second 2D image data of the first object is displayed on the display according to the second display mode, wherein the second display mode includes displaying the second 2D image data of the first object within a representation of the physical environment.

2. The method according to claim 1, wherein displaying the first 2D image data of the first object according to the first display mode comprises: Obtain 2D content data indicating the 2D content area; The 2D content data is combined with the first 2D image data to generate first display data, wherein the first display data indicates the first object within the 2D content area; and The first display data is sent to the display.

3. The method of claim 2, wherein the electronic device includes an image sensor, and wherein displaying the second 2D image data of the first object according to the second display mode comprises: Image data characterizing the physical environment is captured via the image sensor, wherein the image data indicates the representation of the physical environment; The image data is combined with the second 2D image data of the first object to generate second display data, wherein the second display data indicates the first object within the representation of the physical environment; and The second display data is sent to the display.

4. The method of claim 3, wherein the 2D content data further indicates 2D content within the 2D content area.

5. The method of claim 4, wherein the first 2D image data of the first object displayed within the 2D content area comprises the first 2D image data of the first object arranged relative to the 2D content.

6. The method according to claim 1, further comprising: Further in response to the detection request, the second 2D image data of the first object is processed; as well as The completion of the process is detected before the second 2D image data of the first object is displayed according to the second display mode.

7. The method of claim 1, wherein displaying the second 2D image data of the first object according to the second display mode includes stopping the display of the 2D content area while maintaining the display of the second 2D image data of the first object.

8. The method of claim 1, wherein displaying the second 2D image data of the first object according to the first display mode includes displaying an image of the first object, and wherein displaying the second 2D image data of the first object according to the second display mode includes displaying the image of the first object.

9. The method of claim 8, wherein the one or more input devices include a position sensor that receives the manipulation input, and wherein the position sensor detects a position change of the electronic device based on the manipulation input.

10. The method of claim 8, wherein the manipulation input spatially relates to the first 2D image data of the first object on the display.

11. The method of claim 1, further comprising: Render the 3D object model of the second object to generate the first 2D image data of the second object; as well as When the first 2D image data of the first object is displayed in the 2D content area in the first display mode, the 2D image data of the second object is displayed in the 2D content area.

12. The method of claim 1, wherein detecting the request includes receiving input of the second 2D image data relating to the first object displayed within the 2D content area.

13. The method of claim 1, wherein the second display mode corresponds to an augmented reality (AR) environment or a mixed reality (MR) environment.

14. A system for rendering an object model, comprising: monitor; The rendering subsystem is used to render the three-dimensional 3D object model of the first object in order to generate the first 2D image data of the first object. as well as Demultiplexer, used for: The first display data is transmitted to the display, wherein the first display data is characterized by a first display mode, and wherein the first display mode includes displaying the first 2D image data of a first object within a 2D content area; and While displaying the first 2D image data of the first object according to the first display mode, the rendering subsystem is also configured to receive a first manipulation input via one or more input devices and render the 3D object model of the first object based on the first manipulation input to generate second 2D image data of the first object, and the demultiplexer is further configured to transmit the second 2D image data of the first object to the display for display according to the first display mode; and In response to a request to change from the first display mode to the second display mode, second display data is transmitted to the display, wherein the second display data is characterized by the second display mode, and wherein the second display mode includes displaying the second 2D image data of the first object within a representation of the physical environment.

15. The system of claim 14, wherein the system includes a combination system that generates the first display data by combining the first 2D image data with 2D content data, and wherein the system includes a synthesis system that generates the second display data by synthesizing the second 2D image data of the first object with image data.

16. The system of claim 15, wherein the system includes an image sensor that captures the image data.

17. The system of claim 14, wherein the system includes a processing subsystem that processes the second 2D image data of the first object based on the detection of the request.

18. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by an electronic device including a display, cause the electronic device to: Render a 3D object model of the first object to generate first 2D image data of the first object; The first 2D image data of the first object is displayed on the display according to a first display mode, wherein the first display mode includes displaying the first 2D image data of the first object within a 2D content area; as well as While displaying the first 2D image data of the first object according to the first display mode: Receive first manipulation input via one or more input devices; as well as Render the 3D object model of the first object based on the first manipulation input to generate a second 2D image data of the first object; The second 2D image data of the first object is displayed on the display according to the first display mode; as well as In response to a request to change from the first display mode to the second display mode, the second 2D image data of the first object is displayed on the display according to the second display mode, wherein the second display mode includes displaying the second 2D image data of the first object within a representation of the physical environment.

Images