Virtual paper

Abstract

The present disclosure relates to virtual paper. In one embodiment, a method of blending stereoscopic and conformal virtual content to a bounded surface is performed at a device comprising one or more processors, non-transitory memory, and one or more displays. The method includes displaying a bounded surface in a native user computer-generated reality (CGR) environment, where the bounded surface is displayed based on a first set of world coordinates characterizing the native user CGR environment. The method also includes displaying a first stereoscopic virtual object within a perimeter of a first side of the bounded surface, where the first stereoscopic virtual object is displayed according to a second set of world coordinates that are different from the first set of world coordinates characterizing the native user CGR environment.

Description

[0001] Cross-reference to related applications

[0002] This application is a divisional application of Chinese national application number 202010191667.X, filed on March 18, 2020, entitled "Virtual Paper".

[0003] This application claims the benefit of U.S. Provisional Patent Application 62 / 820,137, filed March 18, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates in general to computer-generated reality (CGR) applications on multifunctional devices. Background Technology

[0005] In previously available computer-generated reality (CGR) experiences, 2D and stereoscopic (or 3D) virtual content was typically displayed based on a set of world coordinates representing the user's CGR environment. Consequently, the display scope and user interaction possibilities associated with both 2D and stereoscopic virtual content were limited to rendering and displaying the virtual content based on the set of world coordinates representing the user's CGR environment. Attached Figure Description

[0006] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0007] Figure 1A and Figure 1B This is an illustration of an example of an operating environment based on some implementation schemes.

[0008] Figure 2 This is a block diagram of an exemplary controller according to some implementation schemes.

[0009] Figure 3 A block diagram of an exemplary computer-generated reality (CGR) device according to some implementation schemes.

[0010] Figures 4A to 4O An example of a virtual paper according to some implementation schemes is shown.

[0011] Figures 5A to 5M Examples of interaction with virtual content in an exemplary virtual paper are shown according to some implementation schemes.

[0012] Figure 6A and Figure 6B This is a flowchart illustrating a method for displaying a bounded surface according to some implementation schemes.

[0013] Figure 7A and Figure 7B This is a flowchart illustrating a method for interacting with content displayed within the perimeter of a bounded area, according to some implementation schemes.

[0014] Figure 8 A block diagram of a computing device according to some implementation schemes.

[0015] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Summary of the Invention

[0016] The various embodiments disclosed herein include devices, apparatuses, systems, and methods for blending stereoscopic virtual content and conformal virtual content into virtual paper. In various embodiments, the method is performed on a device including one or more processors, non-transitory memory, and one or more displays. The method includes displaying a bounded surface in a native user computer-generated reality (CGR) environment, wherein the bounded surface is displayed based on a first set of world coordinates characterizing the native user CGR environment. The method also includes displaying a first stereoscopic virtual object within the perimeter of a first side of the bounded surface, wherein the first stereoscopic virtual object is displayed according to a second set of world coordinates, different from the first set of world coordinates characterizing the native user CGR environment.

[0017] The various embodiments disclosed herein include devices, apparatuses, systems, and methods for interacting with virtual content in a virtual paper. In various embodiments, the method is performed on a device including one or more processors, non-transitory memory, and one or more displays. The method includes displaying a bounded region based on a first set of world coordinates, wherein content within the bounded region includes a stereoscopic virtual object displayed according to a second set of world coordinates. The method also includes receiving input for the content. The method further includes moving the content within the perimeter of the bounded region according to the input, wherein the movement includes moving the stereoscopic virtual object within the perimeter of the bounded region according to the input; and animate the stereoscopic virtual object according to the second set of world coordinates.

[0018] In some embodiments, the device includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. In some embodiments, instructions are stored in a non-transitory computer-readable storage medium that, when executed by one or more processors of the device, cause the device to perform or cause to perform any of the methods described herein. In some embodiments, the device includes one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Detailed Implementation

[0019] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details set forth herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0020] Compared to the aforementioned available Computer Generated Reality (CGR) systems, the various embodiments disclosed herein provide a multidimensional CGR environment. In some embodiments, the multidimensional CGR environment includes a display of virtual paper represented as a bounded surface or bounded region. In some embodiments, the bounded surface is displayed based on a first set of world coordinates characterizing the native user's CGR environment. Additionally, stereoscopic virtual content is displayed within the bounded surface based on a second set of world coordinates. In some embodiments, conformal virtual content (e.g., 2D virtual content in a subset of instances) is also displayed within the bounded surface. According to some embodiments, the conformal virtual content adapts to the contour of the bounded surface based on the first set of world coordinates.

[0021] For example, when displaying 3D virtual content such that the 3D virtual content is confined within the perimeter of a virtual paper, the 3D virtual content can protrude from a first side of the virtual paper and be displayed based on a second set of world coordinates. Conversely, 2D virtual content is displayed conformally to the surface of the virtual paper based on a first set of world coordinates. In some embodiments, a second side (e.g., the back side) of the virtual paper, opposite the first side, is displayed as a rasterized conformal representation of the 3D virtual content (e.g., a blurring effect or shadow of the 3D virtual content) and a rasterized conformal representation of the 2D virtual content.

[0022] The virtual paper according to the embodiments described herein not only improves the way virtual content is displayed in a CGR environment, but also improves the way virtual content is interacted with in a CGR environment. For example, when there is more 2D content than can be displayed within a window (or application) pane, previously available display methods typically display only a portion of the 2D content. In response to scroll input, the boundaries of the window or application pane clip the 2D content at the boundaries, so that another portion of the 2D content will be displayed within the boundaries. In a CGR environment for presenting bounded areas within stereoscopic content, previously available systems and methods do not provide an efficient and elegant way to clip stereoscopic virtual content designed to remain at least partially visible within the bounded areas.

[0023] In contrast, the various embodiments disclosed herein incorporate stereoscopic (or 3D) virtual content within the perimeter of a bounded area. Therefore, in some embodiments, in response to scroll input that moves a stereoscopic virtual object beyond the perimeter of the bounded area, the stereoscopic virtual object initially protruding from the surface of the bounded area retracts to the opposite side of the surface of the bounded area. Additionally, in some embodiments, when the stereoscopic virtual object is too large to fit within the native user's CGR environment (e.g., an ocean liner), the stereoscopic virtual object is placed within a bounded area represented by a second set of world coordinates. Placing the stereoscopic virtual object within the bounded area allows the user to subsequently view it through the surface of the bounded area, enabling full-size viewing of the stereoscopic virtual object from an appropriate distance.

[0024] Figure 1A This is a block diagram of an example of a computer-generated reality (CGR) environment 100A according to some embodiments. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the exemplary embodiments disclosed herein.

[0025] As described herein, a physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with physical environments through senses such as sight, touch, hearing, taste, and smell. In contrast, a CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's physical motion or its representation is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of characteristics of virtual objects in a CGR environment can be done in response to a representation of physical motion (e.g., a voice command).

[0026] Humans can use any of their senses to sense and / or interact with CGR objects, including sight, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, audio objects can enable audio transparency, selectively introducing ambient sound from the physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects. Examples of CGR include virtual reality and mixed reality.

[0027] As used herein, a virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. Examples of virtual objects include trees, buildings, and computer-generated images representing human avatars. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment, and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0028] Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments are simulated environments designed to incorporate sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between a purely physical environment as one end and a virtual reality environment as the other end, but not including either end.

[0029] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present the MR environment can track position and / or orientation relative to the physical environment, enabling virtual objects to interact with real objects (i.e., physical objects or representations of them from the physical environment). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground. Examples of mixed reality include augmented reality and augmented virtual reality.

[0030] As used herein, an augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceive the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto a physical environment, such as as a hologram or on a physical surface, so that a person can use the system to perceive the virtual objects superimposed on the physical environment.

[0031] As used herein, an augmented reality environment also refers to a simulated environment in which the representation of the physical environment is transformed through computer-generated sensory information. For example, in providing pass-through video, the system can transform one or more sensor images to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portion can be a representative but not realistic version of the original captured image. Furthermore, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0032] As used herein, an augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more features of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could adopt shadows that correspond to the sun's position within the physical environment.

[0033] Many different types of electronic systems enable people to sense and / or interact with a variety of CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0034] Therefore, as a non-limiting example, the CGR environment 100A includes a controller 102 and a CGR device (e.g., a head-mounted display (HMD)) 104. In the example of Figure 1, the CGR device 104 is worn by a user 10. In some embodiments, the CGR device 104 corresponds to a head-mounted display (HMD), tablet computer, mobile phone, wearable computing device, etc. In some embodiments, the CGR device 104 is configured to present a CGR experience to the user 10. In some embodiments, the CGR device 104 includes a suitable combination of software, firmware, and / or hardware.

[0035] According to some implementations, when user 10 is virtually and / or physically present within scene 106, CGR device 104 presents a CGR experience to user 10. In some implementations, when presenting a CGR experience, CGR device 104 is configured to present CGR content and enable video pass-through of scene 106 (e.g., CGR device 104 corresponds to an AR-enabled mobile phone or tablet). In some implementations, when presenting an AR experience, CGR device 104 is configured to present AR content and enable optical pass-through of scene 106 (e.g., CGR device 104 corresponds to AR-enabled glasses). In some implementations, when presenting a virtual reality (VR) experience, CGR device 104 is configured to present VR content and optionally enable video pass-through of scene 106 (e.g., CGR device 104 corresponds to a VR-enabled HMD).

[0036] In some embodiments, user 10 wears CGR device 104 on his / her head (e.g., as shown in Figure 1). Therefore, CGR device 104 includes one or more displays provided for displaying CGR content. For example, CGR device 104 surrounds user 10's field of view. In some embodiments, a CGR (e.g., AR / VR) room, enclosure, or chamber configured to present CGR content is used instead of CGR device 104, in which user 10 does not wear CGR device 104. In some embodiments, user 10 holds CGR device 104 in one or both of his / her hands.

[0037] In some implementations, controller 102 is configured to manage and coordinate the CGR experience of user 10. In some implementations, controller 102 includes a suitable combination of software, firmware, and / or hardware. In some implementations, controller 102 is a computing device located locally or remotely relative to scenario 106. For example, controller 102 is a local server located within scenario 106. In another example, controller 102 is a remote server (e.g., a cloud server, central server, etc.) located outside scenario 106. In some implementations, controller 102 is communicatively coupled to CGR device 104 via one or more wired or wireless communication channels (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the functionality of controller 102 is provided by and / or combined with CGR device 104.

[0038] like Figure 1A As shown, CGR device 104 presents scene 106. In some embodiments, scene 106 is generated by controller 102 and / or CGR device 104. In some embodiments, scene 106 includes a virtual scene that is a simulated replacement of a real-world scene. In other words, in some embodiments, scene 106 is simulated by controller 102 and / or CGR device 104. In such embodiments, scene 106 is different from the real-world scene in which CGR device 104 resides. In some embodiments, scene 106 includes an enhanced scene as a modified version of the real-world scene. For example, in some embodiments, controller 102 and / or CGR device 104 modify (e.g., enhance) the real-world scene in which CGR device 104 resides to generate scene 106. In some embodiments, controller 102 and / or CGR device 104 generates scene 106 by simulating a copy of the real-world scene in which CGR device 104 resides. In some implementations, controller 102 and / or CGR device 104 generate scene 106 by removing and / or adding items to a simulated copy of the real-world scene in which CGR device 104 resides.

[0039] See Figure 1B , Figure 1B This is an illustration of an exemplary operating environment 100B according to some embodiments. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the exemplary embodiments disclosed herein. Therefore, as a non-limiting example, operating environment 100B includes network 20, computing device 30, real-world scene 40, and device 104b.

[0040] exist Figure 1BIn the example, real-world scene 40 includes user 10. In various embodiments, device 104b captures a set of images of real-world scene 40 and transmits data representing scene 106 via network 20 to computing device 30. In some embodiments, device 104b includes controller 102 and camera 104a. In some embodiments, camera 104a captures the set of images, and controller 102 generates data representing scene 106 based on the set of images. In some embodiments, the data representing scene 106 includes body pose information 108 of user 10 within the field of view of camera 104a.

[0041] In various embodiments, body posture information 108 indicates the body posture of user 10 within the field of view of camera 104a. For example, in some embodiments, body posture information 108 indicates the joint positions and / or joint orientations of user 10 (e.g., the positions / or orientations of the shoulder, elbow, wrist, pelvic, knee, and ankle joints). In some embodiments, body posture information 108 indicates the positions / or orientations of various body parts of user 10 (e.g., the positions / or orientations of the head, torso, upper arm, lower arm, thigh, and calf).

[0042] In various implementations, the bandwidth consumed in transmitting body pose information 108 over network 20 is less than the bandwidth consumed in transmitting images captured by camera 104a. In some implementations, network resources are limited, and device 104b has access to a limited amount of available bandwidth. In such implementations, the bandwidth consumed in transmitting body pose information 108 is less than the available bandwidth, while the bandwidth consumed in transmitting images captured by camera 104a would be greater than the available bandwidth. In various implementations, transmitting body pose information 108 (e.g., instead of transmitting images) improves the operability of network 20 by using, for example, fewer network resources (e.g., by using less bandwidth).

[0043] In some implementations, computing device 30 uses body pose information 108 to render the avatar of user 10. For example, computing device 30 may provide body pose information 108 to a display engine (e.g., a rendering and display pipeline), which utilizes body pose information 108 to render the avatar in a virtual scene. Because computing device 30 uses body pose information 108 to render the avatar, the avatar's body pose is within a certain degree of similarity to the user 10's body pose in the real-world scene 40. Therefore, viewing the avatar in the virtual scene is within a certain degree of similarity to viewing the image in the real-world scene 40.

[0044] Figure 2This is a block diagram of an example controller 102 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, controller 102 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BlueTooth, ZigBee, etc. types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0045] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0046] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state storage devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices remotely located to one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures, or subsets thereof, including optional operating system 230 and CGR experience module 240.

[0047] Operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate single or multiple CGR experiences for one or more users (e.g., single CGR experiences for one or more users, or multiple CGR experiences for corresponding groups of one or more users). To this end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0048] In some implementations, the data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the CGR device 104. For this purpose, in various implementations, the data acquisition unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0049] In some implementations, the tracking unit 244 is configured to plot scene 106 and track at least the position / location of the CGR device 104 relative to scene 106. Figure 1A For this purpose, in various embodiments, the tracking unit 244 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0050] In some implementations, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by CGR device 104. For this purpose, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0051] In some implementations, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the CGR device 104. For this purpose, in various implementations, the data transmission unit 248 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0052] Although the data acquisition unit 242, tracking unit 244, coordination unit 246 and data transmission unit 248 are shown residing on a single device (e.g., controller 102), it should be understood that in other embodiments, any combination of the data acquisition unit 242, tracking unit 244, coordination unit 246 and data transmission unit 248 may reside in a separate computing device.

[0053] also, Figure 2This is used more as a functional description of various features that can exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 2 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented through one or more functional blocks in various implementations. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular implementation.

[0054] Figure 3 For CGR equipment 104 according to some implementation schemes ( Figure 1A A block diagram of an example. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the CGR device 104 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., interfaces of the type of USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, etc.), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more internal and / or external image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0055] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones 307A, one or more speakers 307B (e.g., headphones or loudspeakers), a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0056] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, the one or more CGR displays 312 correspond to display types such as holography, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), and microelectromechanical systems (MEMS). In some embodiments, the one or more CGR displays 312 correspond to diffraction displays, reflective displays, polarization displays, holographic waveguide displays, etc. For example, CGR device 104 includes a single CGR display. In another example, CGR device 104 includes CGR displays for each of the user's eyes. In some embodiments, the one or more CGR displays 312 are capable of displaying CGR content.

[0057] In some embodiments, one or more internal, external, inward-facing, outward-facing, front-facing, and / or back-facing image sensors 314 are configured to acquire image data corresponding to at least a portion of a user's face, wherein at least a portion of the user's face includes the user's eyes (and thus may be referred to as an eye-tracking camera). In some embodiments, one or more internal, external, inward-facing, outward-facing, front-facing, and / or back-facing image sensors 314 are configured to face forward (or outward) to acquire image data corresponding to the scene the user would see when the CGR device 104 is not present (and thus may be referred to as an outward-facing camera). One or more internal, external, inward-facing, outward-facing, front-facing, and / or back-facing image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0058] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 320 optionally includes one or more storage devices remotely located to one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and CGR rendering module 340.

[0059] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. For this purpose, in various embodiments, CGR presentation module 340 includes data acquisition unit 342, audio / CGR presentation unit 344, and data transmission unit 346.

[0060] In some implementations, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from controller 102 (e.g., via one or more communication interfaces 308), one or more I / O devices and sensors 306, or one or more internal, external, inward-facing, outward-facing, front-facing, and / or rear-facing image sensors 314. For this purpose, in various implementations, the data acquisition unit 342 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0061] In some embodiments, the audio / CGR presentation unit 344 is configured to present an audio / CGR experience via one or more CGR displays 312 (and, in various embodiments, speakers 307B and / or microphones 307A). For this purpose, in various embodiments, the audio / CGR presentation unit 344 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0062] In some implementations, the data transmission unit 346 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 102. For this purpose, in various implementations, the data transmission unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0063] Although the data acquisition unit 342, the audio / CGR rendering unit 344, and the data transmission unit 346 are shown residing on a single device (e.g., CGR device 104), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the audio / CGR rendering unit 344, and the data transmission unit 346 may reside in a separate computing device.

[0064] also, Figure 3 This is used more as a functional description of various features that may exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 3 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented through one or more functional blocks in various implementations. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular implementation.

[0065] Figure 4A An example of a CGR environment 400 according to some embodiments is shown, in which a first side (e.g., the front) of a virtual sheet of paper 405 is displayed. In some embodiments, the CGR environment 400 is a native user CGR environment. In some embodiments, the native user CGR environment is a CGR environment containing a user or an avatar representing a user. For example, CGR environment 400 includes user 401 or an avatar representing user 401. Therefore, Figure 4A The scene depicted can be a view of the CGR environment 400 from the perspective of user 401 or an observer.

[0066] In some implementations, within the native user CGR environment 400, the virtual paper 405 is displayed as a bounded surface, for example, a virtual surface with edges or boundaries surrounding its perimeter. Therefore, the virtual paper 405 is sometimes referred to hereinafter as a bounded surface 405. In some implementations, the bounded surface 405 is displayed according to a first set of world coordinates characterizing the native user CGR environment 400. In some implementations, the world coordinate system characterizing the native user CGR environment 400 is the primary or general coordinate system describing the scene of the native user CGR environment 400. For example, in a typical Cartesian coordinate system, the world has an origin and X, Y, and Z axes. In the native user CGR environment 400, axes X1, Y1, and Z1 roughly correspond to right / left, up / down, and front / back, respectively, as if user 401 or an avatar representing user 401 is located at and aligned with the origin. Figure 4AAs shown. Therefore, the position of an object in the native user CGR environment 400 can be described by a world coordinate set representing the native user CGR environment 400, such as (X1, Y1, Z1). In other words, the position of an object, including the virtual paper 405, is displayed in the native user CGR environment 400 as if viewed from the perspective of user 401 or the avatar representing user 401.

[0067] In some embodiments, at least one stereoscopic virtual object (e.g., a 3D chick 420, a 3D sphere object 430, and / or a 3D toolbox object 440) is displayed within the periphery of the front of the virtual paper 405. In some embodiments, at least one stereoscopic virtual object 420-440 is displayed according to a set of world coordinates, for example, the 3D chick 420 is displayed according to the world coordinate set (X2, Y2, Z2) or the 3D toolbox object 440 is displayed according to the world coordinate set (X3, Y3, Z3). Although not in Figure 4A As shown, each of the 3D toolbox objects 440 can be displayed according to its own world coordinate set.

[0068] In some implementations, these world coordinate sets (X2, Y2, Z2) and (X3, Y3, Z3) are different from the world coordinate set (X1, Y1, Z1) characterizing the native user's CGR environment 400. Therefore, as will be described in detail below, each of the stereoscopic virtual objects 420-440 is movable according to its corresponding world coordinate set, and the virtual paper 405 and the virtual objects displayed within it are animated according to their respective world coordinate sets. In some other implementations, the world coordinate set (X2, Y2, Z2) or (X3, Y3, Z3) is the same as the world coordinate set (X1, Y1, Z1) characterizing the native user's CGR environment 400. In other words, these world coordinate sets (X1, Y1, Z1), (X2, Y2, Z2), and / or (X3, Y3, Z3) are either bridgeable or non-intersecting.

[0069] In some implementations, conformal virtual objects, such as 2D text "matrix" 410 and energy representation "<" 412, are also displayed around the perimeter of the virtual paper 405. The 2D text "matrix" 410 and energy representation 412 are displayed conformally to the outline and / or surface of the virtual paper 405. Because the conformal virtual objects 410 and 412 conform to the outline and / or surface of the virtual paper 405 represented by a first set of world coordinates (X1, Y1, Z1), and the 3D chick 420 is displayed according to a second set of world coordinates (X2, Y2, Z2), these virtual objects 410, 412, and 420 can occlude each other. Figure 4AAs shown, the 2D text "matrix" 410 appears to float on the surface of the front of the virtual paper 405, and the 3D chick 420 appears to be inside the virtual paper 405. Therefore, from the user 401's perspective, the 2D text "matrix" 410 appears to be in front of (e.g., covering) the 3D chick 420. Thus, the 2D text "matrix" 410 obscures part of the comb of the 3D chick 420, which is inside the virtual paper 405.

[0070] The virtual paper 405 displayed in the CGR environment 400 differs from the virtual paper in previously existing systems. When constrained to conventional 3D space, 3D virtual objects typically appear in front of (or on top of) the paper. Therefore, in conventional 3D space, 3D virtual objects would obscure other content (e.g., text) on the virtual paper. In contrast, as will be described in detail below, the virtual paper 405 serves as an entry point from the native user CGR environment 400, represented by the world coordinate set (X1, Y1, Z1), to another 3D world (e.g., a 3D world represented by the world coordinate set (X2, Y2, Z2)). Therefore, stereoscopic virtual objects 420-440 are inserted into the virtual paper 405 and displayed as if inside the virtual paper 405. This allows the stereoscopic virtual objects 420-440 to coexist with associated text (e.g., 2D text "matrix" 410 or energy representation 412) for easy reading, thus providing a visually appealing page layout.

[0071] In some implementations, different world lighting illuminates different virtual objects. In some implementations, a single light source illuminates a group of virtual objects within and on the inner side of the virtual paper 405 (e.g., within a threshold distance from the surface of the virtual paper), while different light sources illuminate virtual objects close to or on the surface of the virtual paper 405. Figure 4A In this context, a light source above the 3D chick 420, a stereoscopic virtual object displayed within and on the inner side of the virtual paper 405 according to the second set of world coordinates (X2, Y2, Z2), causes a shadow 422 below the 3D chick 420. In contrast, no shadow is cast below the stereoscopic virtual object 430 located close to the surface of the virtual paper 405 (e.g., within a threshold distance from the surface of the virtual paper 405). Furthermore, as shown by a portion of the stereoscopic virtual object 430 (e.g., a luminous surface protruding from the surface of the virtual paper 405), the stereoscopic virtual object 430 can be illuminated by a light source in the native user CGR environment 400 characterized by the first set of world coordinates (X1, Y1, Z1). Specifically, a luminous surface with a small but intense specular highlight is displayed, the highlight being caused by diffuse reflection from the light source in the native user CGR environment 400.

[0072] It should be pointed out that, Figure 4AAn example of world lighting is shown. In some other embodiments, virtual objects displayed within virtual paper 405 share the same world lighting. Additionally, in some embodiments, the same light source illuminates both virtual paper 405 and the native user CGR environment 400. In such embodiments, the same light source (e.g., point light, spotlight, directional light, area light, volumetric light, ambient light, etc.) can provide lighting effects based on multiple world coordinates. For example, user 401, 2D text "matrix" 410, energy display 412, and stereoscopic virtual objects 420-440 can share a single world lighting, and the effects of such lighting can vary according to various world coordinates.

[0073] In some implementation schemes, Figure 4A The virtual paper 405 shown has a second side, such as the back side. Figure 4B The back of a virtual paper 405 according to some embodiments is shown. In some embodiments, the back of the virtual paper 405 includes a rasterized conformal representation of a stereoscopic virtual object. As is known in the art, rasterization is the process of converting a graphic component into a bitmap image made of pixels. If the graphic component is a 3D object, the 3D object is converted into a 2D image of the object. For example, a 3D chick 420 is rasterized on the front of the virtual paper 405. Figure 4A ), and generate and display the 2D representation of the 3D chick 420 on the back, such as 420-B. Figure 4B As shown. Similarly, 3D sphere objects 430 and 3D toolbox objects 440-1, 440-2, 440-3, 440-4 and 440-5 are also rasterized on the front of the virtual paper 405. Figure 4A ), and generate and display 2D representations of 3D content on the back side as 430-B, 440-1-B, 440-2-B, 440-3-B, 440-4-B, and 440-5-B, such as Figure 4B As shown.

[0074] In some implementations, the virtual paper 405 is semi-transparent. In some implementations, rasterized conformal representations 420-B, 430-B, 440-1-B, 440-2-B, 440-3-B, 440-4-B, and 440-5-B on the back of the virtual paper 405 are blurred. In addition to blurring the rasterized conformal representations of 3D content, the back of the virtual paper 405 also includes blurred 2D content, such as blurred 2D text “matrix” 410-B and blurred 2D navigation indications 412-B. The blurring of the content from the front of the virtual paper 405 creates the effect of a semi-transparent virtual paper 405 existing in the native user CGR environment of user 401, while still prompting the content on the front of the virtual paper 405.

[0075] In some implementations, the display of the virtual paper 405 may change in response to input to the virtual paper 405. For example, the input may be a user placing, twisting, swinging, tossing, rotating, flipping, spinning, bending, curling, and / or folding the virtual paper 405. In some implementations, the outline of the virtual paper 405 changes in response to received input. For example, as... Figures 4C to 4E As shown, the virtual paper 405 wrinkles when turned from the front to the back; for example, the outline of the virtual paper 405 has a curve. Also, as... Figure 4F and Figure 4G As shown, the virtual paper 405 is twisted or warped when turned from front to back. Therefore, the display of the virtual paper 405 changes from a flat surface to a curved surface, where some portions of the curved surface are obscured by other portions of the curved surface. For example, in Figure 4F In the image, a portion of the 2D text is obscured by a portion of the curved surface of the virtual paper 405.

[0076] In some implementations, input to the virtual paper 405 is not limited to movement of the virtual paper 405, such that input from user 401 to the virtual paper 405 includes changes in user 401's body posture. For example, changes in user 401's body posture may include user 401 walking to the back of the virtual paper 405 and / or tilting the CGR device (e.g., to view the virtual paper 405 from different angles). Figure 1A The device 104 or mobile device in the CGR device 104). In some embodiments, the I / O devices and sensors 306 and / or one or more communication interfaces 308 of the CGR device 104 are used. Figure 3 ) Acquire input, such as when a user clicks a button or indicator, when a user issues a voice command, or when input is received from a remote device through one or more communication interfaces 308.

[0077] like Figures 4C to 4G As shown, during the display transformation of the virtual paper 405, in response to the transformation of the outline of the virtual paper 405, the display of the virtual content associated with the virtual paper 405 also transforms. For example, a portion of the 3D chick protrudes from one side of the virtual paper 405. Figures 4C to 4E ), and the 3D dog portion protrudes from one side of the virtual paper 405 ( Figure 4F and Figure 4G This is similar to how a 3D chick or 3D dog is tossed around in response to the swinging of the virtual paper 405. Furthermore, the text on the virtual paper 405 also changes, for example, in... Figure 4C The Chinese character for "chicken" is tilted, and... Figure 4F The text paragraph below the 3D dog appears to be curved. See below for reference. Figures 4H to 4M These transformations will be described in further detail.

[0078] Go to Figures 4H to 4K , Figures 4H to 4K The transformation of the display of the virtual paper 405 in response to input of a rotating virtual paper 405, according to some embodiments, is shown. Figure 4H In response to input as indicated by the dashed arrow, the display of the virtual paper 405 changes and the outline of the virtual paper alters. For example, the right vertical edge of the virtual paper 405 appears shorter, and the top and bottom edges of the virtual paper 405 are slanted, as if the virtual paper 405 has been rotated or swung away from the user 401 (not shown). Figures 4I to 4K As shown, when the virtual paper 405 rotates further around the vertical axis 450, the right vertical edge of the virtual paper 405 becomes shorter and shorter, and the top and bottom edges of the virtual paper 405 become more slanted.

[0079] In addition to changing the display of the virtual paper 405, the display of the conformal virtual objects surrounding the virtual paper 405 also changes simultaneously. Although the display of the conformal virtual objects changes, the display of the conformal virtual objects still conforms to the display of the virtual paper 405. For example, in Figure 4A In this configuration, a conformal virtual object (e.g., a 2D text "matrix" 410) is displayed parallel to the top edge of a virtual sheet 405. As the virtual sheet 405 rotates about a vertical axis 450, in... Figures 4H to 4K The 2D text “matrix” 410 appears tilted, consistent with the display of the virtual paper 405.

[0080] As referenced above Figure 4A The virtual paper 405 is displayed according to a set of world coordinates (X1, Y1, Z1), while the three-dimensional virtual objects 420-440 are displayed according to different sets of world coordinates. Therefore, when the virtual paper 405 rotates along the vertical axis 450 within the world coordinate set (X1, Y1, Z1), the display of the virtual paper 405 changes based on the world coordinate set (X1, Y1, Z1). In contrast, the display of the three-dimensional virtual objects changes based on different sets of world coordinates.

[0081] For example, such as Figures 4H to 4K As shown, while the virtual paper 405 rotates, in addition to being pulled away from the user (not shown) by the virtual paper 405, the 3D chick 420 also rotates around different axes 452 according to different world coordinate sets. Furthermore, as... Figure 4KAs shown, in some embodiments, the transformation of the 3D virtual objects 420-440 includes protruding at least a portion of the 3D virtual objects 420-440 from the front of the virtual paper 405 based on different sets of world coordinates. For example, when the 3D chick 420 rotates around a different axis 452 based on a second set of world coordinates (X2, Y2, Z2), the beak, part of the head, and part of the wing covered by the jacket of the 3D chick 420 protrude from the front of the virtual paper 405, as... Figure 4K As shown. Similarly, a portion of the 3D sphere object 430 protrudes from the front of the virtual paper 405, as... Figure 4J and Figure 4K As shown.

[0082] In some implementations, virtual objects displayed within the perimeter of the first side of the virtual paper 405 obscure each other. For example, in Figure 4A In this scenario, user 401 is shown standing in front of virtual paper 405. Therefore, virtual paper 405 is displayed from a near-vertical viewpoint associated with user 401's posture. From this near-vertical viewpoint, the 2D text "matrix" 410 appears to be in front of the 3D chick 420 and partially obscures it. Figure 4H and Figure 4I In this context, when the virtual paper 405 is displayed as if tilted from a near-vertical viewpoint associated with the user's posture 401, the 2D text "matrix" 410 appears to float on top of the 3D chick 420. Furthermore, in Figure 4H and Figure 4I In the image, although the comb of the 3D chick 420 is still obscured by the 2D text "matrix" 410, the obscuration decreases when the virtual paper 405 rotates around the vertical axis 450. Figure 4J In this case, as the virtual paper 405 rotates further, a perspective view of the volume area is displayed, so that the 2D text "matrix" 410 no longer obscures the comb of the 3D chick 420. Instead, the 2D text "matrix" 410 is displayed on one side of the comb of the 3D chick 420, revealing the virtual content within the virtual paper 405. For example... Figure 4J As shown, the rest of the 3D chick 420 inside the virtual paper 405 disappears behind the boundary of the virtual paper 405 due to occlusion.

[0083] It should be noted that in some implementations, the action performed in response to receiving input is... Figures 4C to 4K The display of the transformation of the 3D object shown in the figure, for example, in response to the user's rotation ( Figures 4C to 4E ),distortion( Figure 4F and Figure 4G ) or rotate ( Figures 4H to 4KThe virtual paper 405 highlights the 3D content from its front side. In some implementations, the 3D object can be transformed without user input to the virtual paper 405. Figure 4L and Figure 4M A stereoscopic virtual object 460, which is displayed around a virtual sheet 405 without user input, is shown according to some embodiments.

[0084] exist Figure 4L and Figure 4M In addition to the 3D virtual object 460 (e.g., a 3D chick 460 displayed on a webpage), conformal virtual objects such as the 2D text "chicken" 462 and the 2D text description 464 are also displayed around the virtual paper 405. Figure 4L and Figure 4M In the absence of user input for the virtual paper 405, there is no transformation of the virtual paper 405 and / or the conformal virtual objects 462 and 464. When the virtual paper 405 remains stationary, the 3D chick 460 is displayed in an animated manner, for example, Figure 4L The head position of the 3D chick in 460 and Figure 4M The head position of the 3D Chicken 460 differs in different worlds. In other worlds, the display of the 3D Chicken 460 is based on its own world coordinate system (e.g., reference). Figure 4A The second set of world coordinates (X2, Y2, Z2) transformations described are different from the first set of world coordinates on which the virtual paper 405 and the conformal virtual objects 462 and 464 are based.

[0085] Go to Figure 4N and Figure 4O , Figure 4N and Figure 4O A virtual paper 405, serving as an entry point in the native user CGR environment 400, is shown according to some implementation schemes. As described above, in Figure 4A In the middle, a 3D object 420 is displayed within the perimeter of the first side of the virtual paper 405 based on the second set of world coordinates (X2, Y2, Z2). Further reference. Figure 4A As described above, the virtual paper 405 is displayed based on the first set of world coordinates (X1, Y1, Z1) representing the local user's CGR environment 400. In some embodiments, the virtual paper 405 is the entry point.

[0086] For example, in Figure 4NIn this context, the 3D chick 420 can be moved out of the perimeter of the virtual paper 405 and placed into the local user CGR environment 400. Once placed in the local user CGR environment, the 3D chick 420 is displayed according to the first set of world coordinates (X1, Y1, Z1). Furthermore, upon entering the local user CGR environment, the user 401 or an avatar representing the user 401 in the local user CGR environment can interact with the 3D chick 420, for example, by painting its comb and / or beak.

[0087] For example, in Figure 4O In this configuration, user 401, or an image representing user 401, can be moved to the periphery of the bounded surface 405, for example, by user input such as a command to enter an entrance represented by virtual paper 405 or a change in user posture instructing user 405 to enter the periphery of the front of virtual paper 405. In some embodiments, once such user input indicating a change in observation coordinates from a first set of world coordinates (X1, Y1, Z1) to a second set of world coordinates (X2, Y2, Z2) is received, the 3D virtual object in virtual paper 405 is adjusted. For example, the size of the 3D chick 420 or the distance between the 3D chick 420 and user 401, or an image representing user 401, can be adjusted based on the second set of world coordinates (X2, Y2, Z2) to suit the observation of the 3D chick 420.

[0088] Figures 5A to 5M Interaction with virtual content within virtual paper 505 is illustrated according to some embodiments. In some embodiments, virtual paper 505 is a bounded region marked by edges or boundaries surrounding the perimeter of an area, which may be flat or volumetric (e.g., an area with a flat or curved surface). Therefore, virtual paper 505 is sometimes also referred to hereinafter as bounded region 505. In some embodiments, bounded region 505 is displayed based on a first set of world coordinates, such as a set of world coordinates (X1, Y1, Z1) characterizing the native user's CGR environment. In some embodiments, bounded region 505 includes conformal virtual objects (e.g., 2D text "dinosaur cutter" 522) within the perimeter of a first side (e.g., the front) of bounded region 505. In some embodiments, the virtual content (hereinafter also referred to as "content") within bounded region 505 also includes one or more stereoscopic virtual objects. For example, in Figure 5AIn the bounded area 505, the content includes a first stereoscopic virtual object 512-1 (e.g., a large 3D dinosaur cutter 512-1) and a second stereoscopic virtual object 512-2 (e.g., a small 3D dinosaur cutter 512-2). In some embodiments, a conformal virtual object 522 is displayed at a relative position to one or more stereoscopic virtual objects 512, for example, the 2D text "dinosaur cutter" 522 is positioned above the 3D dinosaur cutter 512 at a distance measured according to a first set of world coordinates (X1, Y1, Z1). As will be described below, when the conformal virtual object (2D text "dinosaur cutter" 522) moves in response to user input, the relative position between the conformal virtual object (e.g., the 2D text "dinosaur cutter" 522) and the stereoscopic virtual object 512 is maintained during the movement.

[0089] In some implementations, as described above, the stereoscopic virtual object 512 can be displayed according to different world coordinate groups (e.g., world coordinate groups different from each other and / or different from those representing the native user's CGR environment). Therefore, the display of the stereoscopic virtual object 512 can be transformed based on the corresponding world coordinate group. Figure 5A In the design, a large 3D dinosaur cutter 512-1 is displayed according to the second set of world coordinates (X2, Y2, Z2), and a small 3D dinosaur cutter 512-2 is displayed according to the third set of world coordinates (X3, Y3, Z3). Therefore, in some embodiments, the large 3D dinosaur cutter 512-1 and the small 3D dinosaur cutter 512-2 are displayed in an animated manner according to different world coordinate sets, with different directions, dimensions, etc.

[0090] like Figure 5A As indicated by the dashed arrow, input is received for the content within the bounded region 505, wherein the input is in a direction based on a first set of world coordinates (X1, Y1, Z1), for example, the content scrolls upwards along the Z1 axis within the bounded region 505 in the native user's CGR environment. In response to receiving the upward scrolling input, the content within the perimeter of the bounded region 505 moves upwards based on the first set of world coordinates (X1, Y1, Z1). For example, in the example showing the result of the upward scrolling... Figure 5B In the middle, the 2D text "Dinosaur Cutter" 522 moves upward and disappears from the bounded area 505, and the stereoscopic virtual object 512 also moves upward closer to the top edge of the bounded area 505 according to the first set of world coordinates (X1,Y1,Z1).

[0091] In addition to moving the content upwards, the 3D virtual object 512 is displayed animated according to the corresponding world coordinate set. For example, Figure 5AThis illustrates that when the small 3D dinosaur cutter 512-2 is within a threshold distance from the periphery of the bounded region 505, the small 3D dinosaur cutter 512-2 appears to flatten out and / or retract to the opposite side of the front of the bounded region 505. Furthermore, the flattened small 3D dinosaur cutter 512-2 is occluded by the bottom edge of the bounded region 505. As another example, Figure 5B This illustrates that in response to scrolling upwards, as the large 3D dinosaur cutter 512-1 moves closer to the top edge of the bounded region 505, the large 3D dinosaur cutter 512-1 rotates along the Y2 axis, making it appear flattened into the bounded region in preparation for being occluded by the top edge of the bounded region 505. Further as... Figure 5B As shown, once the small 3D dinosaur cutter 512-2 moves upward and exceeds a threshold distance from the periphery of the front of the bounded region 505—for example, the small 3D dinosaur cutter 512-2 moves upward and is fully displayed within the bounded region 505—the small 3D dinosaur cutter 512-2 rotates along the Y3 axis to provide different angles for the small 3D dinosaur cutter 512-2. Therefore, a portion of the small 3D dinosaur cutter 512-2 protrudes from the front of the bounded region 505.

[0092] Figure 5C and Figure 5D The diagram illustrates, according to some embodiments, interaction between a bounded region 505 and content displayed within the periphery of the front of the bounded region 505 when the bounded region 505 is laid flat. In some embodiments, when the display of the bounded region 505 transforms in response to receiving user input, for example, by laying it flat or rotating the bounded region 505 about the X1 axis (not shown) of a first set of world coordinates (X1, Y1, Z1), the conformal virtual object 522 thus transforms to maintain alignment with the contour transformation of the bounded region 505. For example, in Figure 5C In this context, once the bounded region 505 is laid flat, the 2D text "Dinosaur Cutter" 522 is also laid flat to fit the front of the bounded region 505. Figure 5C In response to the scroll input indicated by the dashed arrow, the 2D text "Dinosaur Cutter" 522 moves along the direction of the scroll input and maintains its relative position to the stereoscopic virtual object 512, as shown below. Figure 5D As shown.

[0093] Unlike the transformation of conformal virtual objects (e.g., the 2D text "dinosaur cutter" 522), the stereoscopic virtual object 512 is animated in response to input from a flat, bounded region 505 according to different world coordinate sets. Figure 5C In the middle, the 3D dinosaur cutter 512 protrudes from the front of the bounded region 505, for example, upright or rotated based on the corresponding world coordinate set. Therefore, in Figure 5C and Figure 5DIn the middle, 512 3D dinosaur cutters are displayed in rows, or like one in front of another, rather than like... Figure 5A and 5B As shown in the diagram, it is displayed as one above or on top of another. Therefore, in response to the scroll input indicated by the dashed arrow, the 3D dinosaur cutter 512 scrolls from the front row to the back row.

[0094] Figure 5D It is also shown that in some embodiments, when the small 3D dinosaur cutter 512-2 approaches the edge of the bounded region 505, for example, within a threshold distance from the periphery of the bounded region 505, the small 3D dinosaur cutter 512-2 tilts, collapses, or rotates about the X3 axis (not shown) based on a third set of world coordinates (X3, Y3, Z3). Therefore, the small 3D dinosaur cutter 512-2 retracts towards the opposite side of the surface of the bounded region 505, and a portion of the small 3D dinosaur cutter 512-2 is occluded by the edge of the bounded region 505.

[0095] Figures 5E to 5H A second side (e.g., the rear side) of the bounded region 505 is shown according to some embodiments. Similar to the reference above. Figures 4A to 4O In some embodiments, the virtual paper 405 has a bounded region 505 with a back side that includes a rasterized conformal representation of a stereoscopic virtual object from the front side. For example, in Figure 5E In the text, the content from the front of the bounded area includes 3D objects 520, such as a 3D chick 520. Figure 5F In the bounded region 505, the corresponding back side includes a blurred image of chick 520-B, which corresponds to a rasterized conformal representation of the 3D chick 520 displayed on the front side.

[0096] exist Figure 5G In the middle, as indicated by the dashed arrow, the upward scrolling input targets the content within the perimeter of the first side of the bounded region 505. In response to receiving the upward scrolling input, as... Figure 5F As shown, the 3D chick 520 moves upwards, and a portion of another stereoscopic virtual object (e.g., the 3D dog 524 on the webpage) appears within the bounded area 505. Although not in Figure 5G As shown, when scrolling upwards, the 3D dog 524 will be displayed at the center of the bounded area 505. In response to the input of scrolling upwards, such as... Figure 5H As shown, the corresponding back side of the bounded region 505 includes a blurred image of dog 524-B, which corresponds to a rasterized conformal representation of the 3D dog 524 displayed on the front side.

[0097] In some implementations, the content on the back side of the bounded region 505 (e.g., a blurred image of chick 520-B) Figure 5F ) or blurry image of dog 524-B ( Figure 5H The blurred image (or 2D text) can also move in response to received input (e.g., scrolling the blurred image up and / or down). In response to received input regarding content on the back side, one or more corresponding stereoscopic virtual objects on the front side of the bounded area 505 can be moved accordingly. Furthermore, as described above, when the one or more corresponding stereoscopic virtual objects are moved, they can also be animated according to a corresponding world coordinate set.

[0098] Figure 5I The illustration shows the display of a large stereoscopic virtual object within a bounded area 505 according to some embodiments. In some embodiments, when a stereoscopic virtual object is too large to fit within the native user's CGR environment, the stereoscopic virtual object is placed within the bounded area 505. For example, when an ocean liner is too large to fit within the native user's CGR environment, the ocean liner may be placed within the bounded area 505. Figure 5I In this context, an ocean cruise ship 530 is displayed within a bounded region 505 based on world coordinates (X2, Y2, Z2), and the ocean cruise ship is scaled proportionally to the size of the bounded region 505. After placing the ocean cruise ship 530 within the bounded region 505, the user 401 can view it through the bounded region 505 to observe the ocean cruise ship 530 at full size from an appropriate distance. Similar to other stereoscopic virtual objects described above, the ocean cruise ship 530 displayed within the bounded region 505 can be animated according to its own world coordinate set (X2, Y2, Z2). For example, in... Figure 5I In this configuration, the ocean liner 530 can float or set sail, causing its bow (or front end) to protrude from the frontal surface of the bounded area 505 and become closer to the user 401 due to its three-dimensional presentation. Therefore, the user 401 can closely inspect the bow of the ocean liner 530.

[0099] In some implementations, characteristics of the ocean liner 530 can be further adjusted in response to inputs, such as distance, direction, orientation, scale, and speed. For example, in response to an input to a bounded area 505, the ocean liner 530 can turn within the bounded area 505. Therefore, the stern (or rear) of the ocean liner 530 will be closer to the user 401, and the user can inspect the stern of the ocean liner 530 at close range. Once the user 401 has completed the inspection of the ocean liner 530, an input from the user 401 corresponding to the portion of the ocean liner 530 protruding from the surface of the bounded area 505 (e.g., a gesture indicating a push signal) can send the ocean liner 530 into the sea.

[0100] In addition, as mentioned above Figure 4AIn some embodiments, different world lighting illuminates different virtual objects and / or different parts of the environment. For example, based on world coordinates (X2, Y2, Z2), a three-dimensional virtual object, such as an ocean cruise ship 530, is displayed within a bounded area 505 and on the side of that bounded area. Sunlight 541 above the ocean cruise ship 530 casts a shadow on it. Therefore, the shadow and / or reflection of the ocean cruise ship 530 may appear on the water below it. Figure 5I From the user 401's perspective, world lighting (e.g., sunlight 541) above and behind the ocean cruise ship 530 results in shadows and / or reflections directed towards the user 401. In contrast, the user 401's shadow 402 indicates a different light source 543 in the native user CGR environment, such as a light source 543 projecting light from behind the user 401 according to world coordinates (X1, Y1, Z1) used in the native user CGR environment. Therefore, from the user 401's perspective, shadow 402 is located in front of the user 401 and away from the user, for example, in the opposite direction to the shadow and / or reflection 531 of the ocean cruise ship 530. Additionally, conformal virtual objects (e.g., the text "ocean cruise ship" 532) displayed within a threshold distance from the surface of the bounded area 505 (e.g., on the surface of the bounded area 505) and close to the native user CGR environment can be illuminated by world lighting different from that of the ocean cruise ship 530. Therefore, there is no shadow of the text “ocean cruise ship” 532 directed at user 401.

[0101] It should be pointed out that, Figure 5I An example of world lighting is shown. In some other embodiments, stereoscopic virtual objects and conformal virtual objects displayed within bounded region 505 share the same world lighting. Furthermore, in some embodiments, the same light source illuminates both inside and outside bounded region 505. In such embodiments, the same light source (e.g., point light, spotlight, directional light, area light, volumetric light, ambient light, etc.) can provide illumination based on multiple world coordinates. For example, user 401, the text "ocean cruise ship" 532, and ocean cruise ship 530 can share a single world lighting, and the effect of such lighting can vary depending on the various world coordinates.

[0102] Figure 5J This paper illustrates the display and interaction of stereoscopic virtual objects for visualizing audio signals according to some embodiments. Audio signals are difficult to visualize in conventional 3D environments. For example, an audio signal representing a piece of music may not conform to familiar physical dimensions or lighting metaphors. Therefore, existing audio systems or electromagnetic wave visualization systems and methods are generally limited to 2D representations. According to some embodiments, audio signals can be visualized by stereoscopic virtual objects using a bounded region 505.

[0103] For example, in Figure 5JIn this configuration, stereoscopic virtual objects 540-1 and 540-2 are displayed within a bounded area 505. In some embodiments, the stereoscopic virtual objects correspond to a visual representation of an audio signal. For example, the bounded area 505 includes stereoscopic virtual object 540 to visualize music. Similar to the other stereoscopic virtual objects described above, the stereoscopic virtual object 540 displayed within the bounded area 505 can be animated according to a corresponding world coordinate set. As music plays, the stereoscopic virtual object 540 is animated to display the attributes of the music; for example, the stereoscopic object 540 may enlarge or brighten to correspond to a loud brass instrument performance, or shrink or darken to correspond to a quiet and mellow string instrument performance. Furthermore, as the attributes of the music change, the stereoscopic virtual object 540 moves around, for example, protruding from the front of the bounded area 505. Therefore, the music visualization through the stereoscopic virtual object 540 provides a multi-dimensional experience for the user 401.

[0104] Additionally, in some implementations, the attributes of the stereoscopic virtual object 540 are adjusted in response to input, such as size, color, animation speed, etc. For example, user 401 can touch the stereoscopic virtual object 540-1, move it around, make it fly, or pull it out of the bounded area 505. In some implementations, the stereoscopic virtual object 540 is associated with an audio signal represented by the stereoscopic virtual object. Therefore, changes to the attributes of the stereoscopic virtual object 540 will also change the corresponding audio signal. Thus, interaction with the stereoscopic virtual object 540 can change the music represented by the stereoscopic virtual object 540.

[0105] Figures 5K to 5M The diagram illustrates interaction with a map 550 displayed around a bounded area 505, according to some implementation schemes. In existing systems, the map typically has either a 2D or 3D representation of the area. Figure 5K In this system, map 550 is displayed within a bounded area 505. Similar to maps in existing systems, map 550 is interactive, allowing for panning and / or zooming. However, unlike maps in existing systems, map 550 may include both stereoscopic virtual objects and non-stereoscopic virtual objects (e.g., 2D text or 2D map images).

[0106] For example, such as Figure 5K As shown, a sub-region 552 is designated within the perimeter of the bounded region 505. In response to the designated sub-region 552, as follows... Figure 5L As shown, the 3D virtual object 560 is displayed around the perimeter of sub-region 552. Figure 5LIn this context, the stereoscopic virtual objects 560 include, for example, 3D high-rise buildings 560-1 and 560-3 and a 3D pyramid 560-2. These stereoscopic virtual objects 560 represent a 3D view of a portion of a map 550 within the perimeter of a sub-region 552, such as the area near Market Street marked on map 550. Although not in Figure 5L As shown, in some embodiments, the stereoscopic virtual object 560 may also include 3D street signs, 3D traffic lights, and / or real-time traffic conditions (e.g., moving cars and / or buses). In some embodiments, conformal virtual objects, such as a 2D map displaying streets / roads and road signs, are displayed outside the perimeter of sub-region 552. Therefore, the stereoscopic virtual object 560 and non-stereoscopic virtual objects (e.g., the remainder of the map within bounded region 505) coexist within the perimeter of bounded region 505.

[0107] In some implementations, the position of sub-region 552 within bounded region 505 is fixed according to a world coordinate set characterizing the native user's CGR environment. In other words, as map 550 moves, the position of sub-region 552 within bounded region 505 remains unchanged, as if sub-region 552 had opened a window pinned to its position within bounded region 505. Therefore, the distance from sub-region 552 to the edge of bounded region 505 is fixed. In such implementations, map content slides in and out of the window as map 550 moves. In some implementations, content sliding into the window is converted into corresponding stereoscopic virtual content, and content surrounding sub-region 552 sliding out is converted into corresponding conformal virtual content.

[0108] For example, in Figure 5L As indicated by the dashed arrow, a translation input to map 550 is detected within bounded region 505. In response to receiving a translation input toward the right edge of bounded region 505, map 550 moves to the right. Therefore, at least some of the stereoscopic virtual objects 560 are moved out of the periphery of sub-region 552. In some embodiments, once moved out of the periphery of sub-region 552, at least some of the stereoscopic virtual objects 560 are converted into one or more conformal virtual objects, for example, displaying a 2D map of streets / roads and road signs near the Market Street area. In some embodiments, in response to receiving a translation input toward the right edge of bounded region 505, another location / area on map 550 is moved to a location associated with sub-region 552, for example, an area near Jackson Street. Therefore, as... Figure 5L As shown, 2D streets / roads and / or road signs near Jackson Street are converted into corresponding 3D objects and displayed around sub-area 554, for example, as Figure 5M As shown, a small 3D building complex near Jackson Street.

[0109] Although not shown in the figures, in some implementations, the location of a sub-region 552 based on a world coordinate set representing the native user's CGR environment is associated with (or attached to) content on map 550. For example, in Figure 5L In this implementation, the position of sub-region 552 can be associated with an area near the market street. When content displayed within bounded area 505 is moved (e.g., panned or dragged), the position of sub-region 552 moves along with the associated content on map 550. Therefore, the 3D virtual object 560 displayed around sub-region 522 moves along with the rest of the content on map 550. In other words, when map 550 is panned, the 3D virtual object 560 within the specified sub-region moves along with the rest of map 550.

[0110] Figure 6A and Figure 6B A flowchart of a method 600 for displaying stereoscopic virtual content within the perimeter of a bounded surface, according to some embodiments, is shown. In some embodiments, method 600 is performed by a device having one or more processors, non-transitory memory, and one or more displays. In some embodiments, method 600 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 600 is performed by a processor and / or controller (e.g., a processor, a controller, a controller) that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory). Figure 1A , Figure 1B and Figure 2 The controller 102 in the process executes the method. In short, in some cases, method 600 includes: displaying a bounded surface in a native user computer-generated reality (CGR) environment, wherein the bounded surface is displayed based on a first set of world coordinates characterizing the native user CGR environment; and displaying a first stereoscopic virtual object within the perimeter of a first side of the bounded surface, wherein the first stereoscopic virtual object is displayed according to a second set of world coordinates, the second set of world coordinates being different from the first set of world coordinates characterizing the native user CGR environment.

[0111] In box 602, method 600 begins, wherein the device displays a bounded surface in a native user computer-generated reality (CGR) environment, wherein the bounded surface is displayed based on a first set of world coordinates characterizing the native user CGR environment. For example, Figure 4A The bounded surface 405 is shown based on the first set of world coordinates ((X1,Y1,Z1)). Figure 4A The first set of world coordinates (X1, Y1, Z1) of the native user CGR environment 400 representing the avatar of user 401 is also shown.

[0112] In box 604, method 600 continues, wherein the device displays a first stereoscopic virtual object within the periphery of a first side of a bounded surface, wherein the first stereoscopic virtual object is displayed according to a second set of world coordinates, the second set of world coordinates being different from the first set of world coordinates characterizing the native user's CGR environment. For example, in Figure 4A In this context, a 3D virtual object 420 (which is a 3D chick) is displayed within the perimeter of the front face of the bounded surface 405. Additionally, as... Figure 4A As shown, when the bounded surface 405 is displayed based on the first set of world coordinates (X1,Y1,Z1), the 3D chick 420 is displayed based on the second set of world coordinates (X2,Y2,Z2), which is different from the first set of world coordinates (X1,Y1,Z1). Therefore, from the perspective of user 401, the 3D chick 420 appears to be displayed on the inside front or behind the bounded surface 405.

[0113] In some embodiments, as shown in box 610, method 600 further includes displaying a conformal virtual object within the perimeter of a bounded surface, wherein the conformal virtual object conforms to the contour of the bounded surface represented by a first set of world coordinates. In such embodiments, as shown in box 612, method 600 further includes transforming the display of the conformal virtual object in response to transforming the contour of the bounded surface.

[0114] For example, in Figure 4A In the text, the 2D text "matrix" 410 is a conformal virtual object displayed within the perimeter of the first side of the bounded surface 405 based on the first set of world coordinates (X1, Y1, Z1). Figure 4A In the text, the 2D text "matrix" 410 is parallel to the top edge of the bounded surface 405 and conforms to the flat front surface of the bounded surface 405, as if it were floating on the flat front surface of the bounded surface 405. Figures 4H to 4K In this context, the contour of the bounded surface 405 changes in response to rotation input; for example, the contour of the bounded surface 405 changes from a rectangular shape to a trapezoidal shape. Therefore, the shape of the 2D text "matrix" 410 simultaneously changes to maintain alignment with the contour transformation of the bounded surface 405; for example, the viewpoint and shape of the 2D text "matrix" 410 change to conform to the trapezoidal bounded surface 405.

[0115] For example, in Figure 5G In the text, the 2D text "dog" is a conformal virtual object displayed within the perimeter of the front face of the bounded surface 505 and conforming to the outline of the flat bounded surface 505. Figure 4F In this context, the profile of bounded surface 405 changes in response to rotational input; for example, bounded surface 405 is distorted and its profile is curved. In response to the transformation of bounded surface 405, such as... Figure 4F As shown, the 2D text “dog” is displayed as a curve and appears wrinkled to conform to the distorted bounded surface 405.

[0116] See again Figure 6A In some embodiments, as shown in block 620, method 600 further includes displaying a second side of a bounded surface, wherein the back side of the bounded surface includes a rasterized conformal representation of the first stereoscopic virtual object. In such embodiments, as shown in block 622, displaying the second side of the bounded surface includes, according to some embodiments, displaying a blurred representation of the rasterized conformal representation of the first stereoscopic virtual object on the second side of the bounded surface.

[0117] For example, Figure 4B , Figure 4E and Figure 4G The back side of the bounded surface 405 is shown. (See diagram.) Figure 4B As shown, the back surface of the bounded surface 405 includes a conformal representation 420-B of a 3D chick 420 from the front surface of the bounded surface 405. Therefore, the bounded surface 405 appears translucent, causing a blurred chick image 420-B to appear on the back surface of the bounded surface 405 within the periphery of the back surface of the bounded surface 405. Similarly, in Figure 4E and Figure 4G In this context, the back face of bounded surface 405 includes a conformal (e.g., 2D) representation of the corresponding stereoscopic virtual object from the front face of bounded surface 405. This is because when bounded surface 405 is transformed (e.g., in response to an input that causes bounded surface 405 to rotate), the conformal representation of the corresponding stereoscopic virtual object adapts to bounded surface 405, such as... Figure 4E and Figure 4G As shown, the rasterization conformal representation on the back side is simultaneously transformed to maintain alignment with the curved bounded surface 405.

[0118] In some embodiments, as shown in box 630, method 600 further includes transforming the display of the bounded surface in response to input to the bounded surface to change the contour of the bounded surface. In such embodiments, as shown in box 632, method 600 further includes receiving input from a user, including detecting changes in the user's body posture. Additionally, in such embodiments, as shown in box 634, transforming the display of the bounded surface includes transforming the display of the first stereoscopic virtual object based on a first set of world coordinates and a second set of world coordinates.

[0119] For example, as mentioned above Figures 4C to 4KAs explained, the input could be a user turning the bounded surface 405 to its back. When the bounded surface 405 changes in response to the input, the 3D chick's head moves according to a second set of world coordinates (X2, Y2, Z2), simultaneously moving in tandem with the movement of the bounded surface 405. Alternatively, although not shown in the figure, the input could be a user walking to the back of the bounded surface 405 in the CGR environment 400 to inspect the back of the entrance. Yet another example is a user moving a portable multi-functional device to view the bounded surface 405 from different angles to obtain, for instance... Figure 4J The diagram shows a side view of the bounded surface 405. Therefore, changes in the user's body posture detected by the device can be inputs that trigger a transformation of the bounded surface 405. For example, changes in body posture may include: the user clicking a button to turn the bounded surface 405; the user twisting or flicking the bounded surface 405; the user folding the bounded surface 405; the user pushing the bounded surface 405 downwards; the user selecting a 3D toolbox object 440; and so on. Those skilled in the art will appreciate that this input is not limited to the changes in body posture described above. Other forms of input, such as voice input, can also be used to trigger transformations of the bounded surface. Additionally, in some embodiments, transformations of the bounded surface are performed without user input.

[0120] Go to Figure 6B In some embodiments, as shown in box 640, method 600 further includes transforming the display of the first stereoscopic virtual object based on a second set of world coordinates. In such embodiments, as shown in box 642, transforming the display of the first stereoscopic virtual object based on the second set of world coordinates includes displaying at least a portion of the first stereoscopic virtual object as protruding from a first side of a bounded surface. In other words, in some embodiments, the display of the stereoscopic virtual object does not transform in response to user input. In such embodiments, the transformation of the stereoscopic virtual object may be disproportionate to the transformation of the bounded surface and the transformation of the conformal virtual object.

[0121] For example, in Figure 4L and Figure 4M In the example, the head of the 3D chick 460 moves even when the bounded surface 405 remains stationary. For example, ... Figures 4H to 4K As shown, the 3D chick 420 rotates around axis 452 based on the second set of world coordinates (X2, Y2, Z2), while the bounded surface 405 rotates around the vertical axis 450 based on the first set of world coordinates (X1, Y1, Z1). For example, ... Figure 5C As shown, when the bounded surface 505 is laid flat, the 3D dinosaur cutter 512 stands upright and protrudes from the front of the bounded surface 505.

[0122] See still Figure 6BIn some embodiments, as shown in box 650, method 600 further includes displaying a virtual object within the periphery of a bounded surface, including at least a portion of the first stereoscopic virtual object. In such embodiments, as shown in box 652, method 600 further includes displaying a first side of the bounded surface that is tilted from a vertical line of sight associated with the user's posture, and stopping the display of the virtual object as if it were obscuring at least a portion of the first stereoscopic virtual object. For example, in Figure 4A In the front view of the bounded surface 405, a near-vertical line of sight is shown in relation to a user 401 who is standing almost in front of the bounded surface 405. From the perspective of the near-vertical line of sight, in Figure 4A In the image, a 3D chick 420 within a bounded surface 405 is behind a 2D text "matrix" 410 floating on the front of the bounded surface 405. Therefore, the 2D text "matrix" 410 obscures the comb of the 3D chick 420. In contrast, in... Figure 4J In the oblique view shown, when the bounded surface 405 is turned at a certain angle and the front of the bounded surface 405 is tilted from the vertical viewing angle, the 2D text “matrix” 410 no longer obscures the comb of the 3D chick 420.

[0123] In some implementations, as shown in block 660, method 600 further includes moving a first stereoscopic virtual object out of the periphery of a bounded surface and into the native user's CGR environment, and, in response to the first stereoscopic virtual object entering the native user's CGR environment, displaying the first stereoscopic virtual object according to a first set of world coordinates. For example, in Figure 4N In this scenario, the 3D chick 420 can be pulled outside the perimeter of the front of the bounded surface 405 and placed in front of the user 401, for example, as if the 3D chick 420 were pulled out of an entrance in preparation for painting. By pulling the 3D chick 420 outside the perimeter of the bounded surface 405, the 3D chick 420 is no longer displayed according to the second set of world coordinates (X2, Y2, Z2). Instead, the 3D chick 420 is displayed and / or animated along with the user 401 according to the first set of world coordinates (X1, Y1, Z1).

[0124] In some implementations, as shown in box 670, method 600 further includes receiving user input indicating a change in the observation coordinates from a first set of world coordinates in the native user's CGR environment to a second set of world coordinates within a bounded surface. In response to receiving the user input indicating the change in the observation coordinates, as also shown in box 670, method 600 further includes adjusting the display of the first stereoscopic virtual object according to the second set of world coordinates. For example, in Figure 4OIn this example, user 401 can enter the periphery of bounded surface 405 or view through it, for example, as if passing through an entrance to inspect 3D chick 420. In response to detecting user input (e.g., a change in body posture) indicating a change in the observation coordinates from a first set of world coordinates (X1, Y1, Z1) to a second set of world coordinates (X2, Y2, Z2), the display of 3D chick 420 changes; for example, 3D chick 420 moves according to the second set of world coordinates (X2, Y2, Z2). In another example, although not shown in the figure, after the user pulls the 3D virtual object out of the bounded surface, the user can interact with the 3D virtual object (e.g., change or add color, add a tie, etc.), and then place the 3D virtual object back into the periphery of the front of the bounded surface to view the effects of the modifications on the bounded surface.

[0125] In some implementations, as shown in box 680, method 600 further includes displaying a second stereoscopic virtual object within the perimeter of a bounded surface, wherein the second stereoscopic virtual object is displayed according to a third set of world coordinates different from the first and second sets of world coordinates. For example, in Figures 4H to 4K In this system, each 3D toolbox object 440 can rotate according to its own set of world coordinates. The corresponding set of world coordinates for each 3D toolbox object 440 can differ from the second set of world coordinates (X2, Y2, Z2) upon which the 3D chick 420 is based. Therefore, the 3D chick 420 and the 3D toolbox object 440 can be displayed in different animation styles, for example, rotating around different axes in different world coordinate systems.

[0126] Figure 7A and Figure 7B This is a flowchart illustrating a method 700 for displaying stereoscopic virtual content within the perimeter of a bounded surface according to some embodiments. In some embodiments, method 700 is performed by a device having one or more processors, non-transitory memory, and one or more displays. In some embodiments, method 700 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 700 is performed by a processor and / or controller (e.g., a processor, a controller, a controller) that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory). Figure 1A , Figure 1B and Figure 2 The controller 102 in the middle executes the method. In short, in some cases, method 700 includes: displaying a bounded region based on a first set of world coordinates, wherein content within the bounded region includes a stereoscopic virtual object displayed according to a second set of world coordinates; receiving input for the content; and moving the content within the perimeter of a first side of the bounded region according to the input, including moving the stereoscopic virtual object within the perimeter of the first side of the bounded region according to the input; and animate display of the stereoscopic virtual object according to the second set of world coordinates.

[0127] In box 710, method 700 begins, where the device displays a bounded region based on a first set of world coordinates, where the content within the bounded region includes stereoscopic virtual objects displayed according to a second set of world coordinates. For example, in Figure 5A In the diagram, the bounded region 505 is displayed based on the first set of world coordinates (X1, Y1, Z1). Further details are as follows... Figure 5A As shown, the content displayed within the bounded area 505 includes a 3D dinosaur cutter 512, which is displayed according to the second set of world coordinates (X2,Y2,Z2).

[0128] In some implementations, as shown in box 712, the method further includes scaling the stereoscopic virtual object to fit within the perimeter of a first side of the bounded region. For example, as... Figure 5I The ocean liner 530 shown is displayed at full size and at an appropriate distance within the perimeter of the front of the bounded area 505 for easy observation.

[0129] In box 720, method 700 continues, where the device receives input for the content. For example, the input could be a user scrolling the content up and / or down, such as... Figure 5A , Figure 5G and Figure 5L As shown by the dashed arrow in the image. For example, the input could be a user-scrolled 3D virtual object, such as... Figure 5C As shown by the dashed arrow in the image. For example, in... Figure 5I In the ocean liner example shown, the input could be a user moving the ocean liner 530 within the bounded area 505 to view the front or side of the ocean liner 530. Alternatively, the input could be a user 401 interacting with a 3D virtual object 540-1, such as... Figure 5J As shown.

[0130] See still Figure 7A In box 730, method 700 continues, wherein the device moves content within the perimeter of a first side of the bounded region based on input. For example, in Figure 5A and Figure 5B In response to the upward scroll input, the 3D dinosaur cutter 512 scrolls upward according to the direction of the upward scroll input. For example, in... Figure 5C and Figure 5D In response to line-by-line scrolling input from back to front, the 3D dinosaur cutter 512 is moved from back to front.

[0131] In some implementations, moving content within the perimeter of a first side of a bounded region based on input includes: (a) moving a stereoscopic virtual object within the perimeter of a first side of the bounded region based on input, as shown in box 732; and (b) animating the stereoscopic virtual object based on a second set of world coordinates, as shown in box 734. For example, in Figure 5B In the middle, the small 3D dinosaur cutter 512-2 rotates according to the second set of world coordinates (X2,Y2,Z2).

[0132] In some implementations, as shown in box 736, animating a stereoscopic virtual object based on a second set of world coordinates includes: determining the distance of the stereoscopic virtual object from the periphery of a first side of the bounded region based on a first set of world coordinates; and, based on determining that the distance is within a threshold, shrinking the stereoscopic virtual object back into the bounded region and occluding a portion of the stereoscopic virtual object by the periphery of the first side of the bounded region. For example, when scrolling near the edge of the bounded region 505, Figure 5B The large 3D dinosaur cutter 512-1 and Figure 5D The small 3D dinosaur cutter 512-2 is folded inside the bounded area 505. Furthermore, depending on the nature of the occlusion, once... Figure 5B The large 3D dinosaur cutter 512-1 and Figure 5D The small 3D dinosaur cutter 512-2 retracts into the bounded area 505, and a portion of the corresponding stereoscopic virtual object 512 is occluded by the perimeter of the front of the bounded area 505. For example, Figure 5B The large 3D dinosaur cutter 512-1 in the middle is obscured by the top edge of the bounded area 505, while Figure 5D The small 3D dinosaur cutter 512-2 in the middle is obscured by the bottom edge of the bounded area 505.

[0133] In some implementations, as shown in box 738, animating a stereoscopic virtual object based on a second set of world coordinates includes: determining the distance of the stereoscopic virtual object from the periphery of a first side of a bounded region based on a first set of world coordinates; and, based on determining that the distance is greater than a threshold, displaying at least a portion of the stereoscopic virtual object as protruding from the first side of the bounded region. For example, in Figure 5B In the middle, as the content within the bounded area 505 is scrolled upwards, the small 3D dinosaur cutter 512-2 rotates and protrudes from the front of the bounded area 505.

[0134] Go to Figure 7BIn some embodiments, method 700 further includes, as shown in box 740, displaying a second side of a bounded region, wherein the second side of the bounded region includes a rasterized conformal representation of the stereoscopic virtual object. In such embodiments, as shown in box 742, method 700 further includes updating the display of the second side of the bounded region in response to receiving input. In some embodiments, updating the display of the second side of the bounded region further includes: moving the rasterized conformal representation of the stereoscopic virtual object based on a first set of world coordinates; and modifying the rasterized conformal representation of the stereoscopic virtual object based on an animated display of the stereoscopic virtual object.

[0135] For example, Figure 5E The front side of the bounded region 505 is shown, while Figure 5F The corresponding back side of bounded region 505 is shown. Figure 5F In the middle, the back side of the bounded surface 505 includes a rasterized conformal representation 520-B of the 3D chick 520 from the front side of the bounded region 505. Figure 5G In the process, as the content of bounded region 505 is scrolled upwards, 3D dog 524 moves into bounded region 505. When 3D dog 524 is moved further upwards, as... Figure 5H As shown, the content on the back side of the bounded region 505 has been modified to show the rasterized conformal representation 524-B of the 3D dog 524.

[0136] In some embodiments, method 700 further includes, as shown in box 750, specifying a sub-region within a bounded region, wherein the sub-region includes a stereoscopic virtual object. In such embodiments, as shown in box 752, moving content within the perimeter of the bounded region based on input includes: moving the stereoscopic virtual object outside the perimeter of the sub-region; and, according to some embodiments, converting the stereoscopic virtual object into a conformal virtual object based on a first set of world coordinates for display outside the perimeter of the sub-region. Additionally, in such embodiments, as shown in box 754, moving content within the perimeter of the bounded region based on input includes: moving a conformal virtual object displayed outside the perimeter of the sub-region inside the perimeter of the sub-region; and converting the conformal virtual object into at least one stereoscopic virtual object.

[0137] For example, Figure 5K A map 550 is shown displayed within a bounded area 505 and a designated sub-area 552 within the perimeter of the front of the bounded area 505. In response to the designated sub-area 552, as... Figure 5L As shown, a 3D virtual object 560 is displayed within the perimeter of sub-region 552, such as a 3D view of the area near the market street. Outside the perimeter of sub-region 552, a 2D map is displayed within the remaining portion of bounded area 505. As the map 550 is panned or dragged, as... Figure 5L As shown, map 550 has been updated. For example, in Figure 5MIn this view, the 3D view of the area near Market Street has been moved out of sub-region 522 and replaced by the 3D view of the area near Jackson Street. In other words, in response to panning or dragging input, Figure 5L 3D content within sub-region 522 (e.g., a 3D view of the area near Market Street) has been moved out of the perimeter of sub-region 522; and once moved out of sub-region 522, such content is converted to 2D content on the map. Further, in response to panning or dragging input, Figure 5L Some 2D content outside sub-region 522 (e.g., a 2D view of the area near Jackson Street) has been moved to the periphery of sub-region 522; and once inside sub-region 522, such content is converted to 3D content.

[0138] In some implementations, as shown in box 760, the content includes a conformal virtual object that adapts to the contour of a bounded region. Method 700 also includes displaying the conformal virtual object at a relative position to the stereoscopic virtual object based on a first set of world coordinates. For example, the content within bounded region 505 also includes... Figure 5A The 2D text "dinosaur cutter" 522 in the middle, 2D text between stereoscopic virtual objects (e.g., Figure 5G The 3D chick 520 and the 2D text "Dinosaur Cutter" 522) Figure 5I The 2D text "ocean liner" Figure 5J 2D text “music” and / or Figures 5K to 5M The 2D map outside sub-region 552. In such implementations, moving content within the perimeter of a first side of the bounded region based on input includes moving conformal virtual objects along the input direction based on a first set of world coordinates, while maintaining the relative position of the stereoscopic virtual objects according to some implementations.

[0139] For example, in Figure 5A In the middle, the 2D text "Dinosaur Cutter" 522 is displayed above the large 3D dinosaur cutter 512-1, and closer to the top edge of the bounded area 522. As the content scrolls upwards, the 2D text "Dinosaur Cutter" 522 moves upwards along with the 3D stereoscopic virtual object 512. (As...) Figure 5B As shown, the 2D text has been moved upwards so much that it is clipped by the edge of the bounded region 505. Similarly, in... Figure 5CIn the context of the bounded area 505 being laid flat, the 2D text "Dinosaur Cutter" 522 is positioned at a certain distance behind the 3D virtual object 512. In response to scrolling from back to front, the 2D text "Dinosaur Cutter" 522 moves together with the 3D virtual object 512 in the same direction closer to the bottom edge of the bounded area 505, and the position of the 2D text "Dinosaur Cutter" 522 relative to the 3D virtual object 512 is maintained; for example, the 2D text "Dinosaur Cutter" 522 remains at the same relative distance behind the larger 3D dinosaur cutter 512-1.

[0140] Figure 8 This is a block diagram of a computing device 800 according to some embodiments. In some embodiments, the computing device 800 corresponds to... Figure 1A The device 800 comprises at least a portion of the device 104 and performs one or more of the functions described above. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the computing device 800 includes one or more processing units (CPUs) 802 (e.g., processors), one or more input / output (I / O) interfaces 803 (e.g., network interfaces, input devices, output devices, and / or sensor interfaces), a memory 810, a programming interface 805, and one or more communication buses 804 for interconnecting these components and various other components.

[0141] In some embodiments, one or more communication buses 804 include circuitry for interconnecting and communicating between system components. Memory 810 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and in some embodiments, includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 810 optionally includes one or more storage devices remotely located to one or more CPUs 802. Memory 810 includes a non-transitory computer-readable storage medium. Furthermore, in some embodiments, memory 810 or its non-transitory computer-readable storage medium stores programs, modules, and data structures or subsets thereof, including optionally an operating system 820, an image capture control module 850, an image processing module 852, a body pose processing module 854, an input processing module 856, and a CGR content presentation module 858. In some embodiments, one or more instructions are included in a combination of logic and non-transitory memory. Operating system 820 includes procedures for handling various basic system services and for performing hardware-related tasks.

[0142] In some implementations, the image capture control module 850 is configured to control the function of an image sensor or camera component to capture images or acquire image data, such as image data for generating CGR content. To this end, the image capture control module 850 includes a set of instructions 851a, as well as heuristics and metadata 851b.

[0143] In some implementations, the image processing module 852 is configured to preprocess raw image data from an image sensor or camera assembly (e.g., converting RAW image data to RGB or YCbCr image data and deriving pose information, etc.). To this end, the image processing module 852 includes a set of instructions 853a, as well as heuristics and metadata 853b.

[0144] In some implementations, the body posture processing module 854 is configured to process the user's body posture (e.g., process IMU data), for example, to generate user input based on the user's body posture or body movement. To this end, the body posture processing module 854 includes a set of instructions 855a, as well as heuristics and metadata 855b.

[0145] In some implementations, the input processing module 856 is configured to process user input, such as scrolling or dragging content on a virtual sheet of paper, moving the virtual sheet of paper, or walking to the other side of the virtual sheet of paper. To this end, the input processing module 856 includes a set of instructions 857a, as well as heuristics and metadata 857b.

[0146] In some implementations, the CGR content rendering module 858 is configured to composite, render, and / or display CGR content items together with other post-processed image data. To this end, the CGR content rendering module 858 includes a set of instructions 859a, as well as heuristics and metadata 859b.

[0147] Although the image capture control module 850, image processing module 852, body pose processing module 854, input processing module 856, and CGR content presentation module 858 are shown residing on a single computing device, it should be understood that in other embodiments, any combination of the image capture control module 850, image processing module 852, body pose processing module 854, input processing module 856, and CGR content presentation module 858 may reside on separate computing devices in various embodiments. For example, in some embodiments, each of the image capture control module 850, image processing module 852, body pose processing module 854, input processing module 856, and CGR content presentation module 858 may reside on a separate computing device or in the cloud.

[0148] also, Figure 8This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 8 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented through one or more functional blocks in various implementations. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0149] While various aspects of embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above embodiments can be embodied in a wide variety of forms, and any particular structure and / or function described above are merely illustrative. Based on this disclosure, those skilled in the art will understand that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Furthermore, other structures and / or functions, other than or different from one or more aspects set forth herein, can be used to implement such an apparatus and / or practice such a method.

[0150] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0151] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “an” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprises” and / or “comprising” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0152] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method comprising: In a device that includes one or more processors, non-transitory memory, and one or more displays: The first side of the bounded region is shown according to the first set of world coordinates; as well as Displaying a three-dimensional virtual object within the perimeter of the first side of the bounded region using a second set of world coordinates different from the first set of world coordinates, including: When the distance between the three-dimensional virtual object and the periphery of the first side of the bounded region is greater than a threshold, a portion of the three-dimensional virtual object is displayed as protruding from the first side of the bounded region; as well as When the distance between the 3D virtual object and the periphery of the first side of the bounded region is not greater than the threshold, the 3D virtual object is shrunk back into the first side of the bounded region.

2. The method of claim 1, wherein displaying the portion of the stereoscopic virtual object as protruding from the first side of the bounded area comprises: The three-dimensional virtual object is displayed in an animated manner based on the second set of world coordinates without user input.

3. The method of claim 1, wherein in response to user input for the bounded region, the portion of the stereoscopic virtual object is displayed as protruding from the first side of the bounded region.

4. The method of claim 1, wherein in response to user input interacting with content displayed within the first side of the bounded region, the stereoscopic virtual object is shrunk back into the first side of the bounded region when the distance between the stereoscopic virtual object and the periphery of the first side of the bounded region is not greater than the threshold.

5. The method according to claim 1, further comprising: At least a portion of the three-dimensional virtual object is retracted by the periphery occlusion of the first side of the bounded region.

6. The method of claim 1, wherein the first side of the bounded region is displayed in a real CGR environment generated on the native user computer, and the method further comprises: In the native user CGR environment characterized by the first set of world coordinates, the portion of the stereoscopic virtual object protruding from the first side of the bounded region is illuminated by a light source.

7. The method of claim 1, wherein the light source illuminates the three-dimensional virtual object on the first side of the retracted bounded region according to the second set of world coordinates.

8. The method according to claim 1, further comprising: Based on the first set of world coordinates, a conformal virtual object is displayed within the perimeter of the first side of the bounded region at a position relative to the three-dimensional virtual object, wherein the conformal virtual object conforms to the outline of the bounded region.

9. The method according to claim 1, further comprising: The second side of the bounded region is displayed, wherein the second side of the bounded region includes the rasterized conformal representation of the stereoscopic virtual object.

10. An apparatus comprising: One or more processors; Non-transitory memory; One or more displays; as well as One or more programs stored in the non-transitory memory, when executed by the one or more processors, cause the device to: The first side of the bounded region is shown according to the first set of world coordinates; as well as Displaying a three-dimensional virtual object within the perimeter of the first side of the bounded region using a second set of world coordinates different from the first set of world coordinates, including: When the distance between the three-dimensional virtual object and the periphery of the first side of the bounded region is greater than a threshold, a portion of the three-dimensional virtual object is displayed as protruding from the first side of the bounded region; as well as When the distance between the 3D virtual object and the periphery of the first side of the bounded region is not greater than the threshold, the 3D virtual object is shrunk back into the first side of the bounded region.

11. The device of claim 10, wherein displaying the portion of the stereoscopic virtual object as protruding from the first side of the bounded area comprises: The three-dimensional virtual object is displayed in an animated manner based on the second set of world coordinates without user input.

12. The device of claim 10, wherein in response to user input for the bounded region, the portion of the stereoscopic virtual object is displayed as protruding from the first side of the bounded region.

13. The device of claim 10, wherein in response to user input interacting with content displayed within the first side of the bounded region, the device retracts the stereoscopic virtual object back into the first side of the bounded region when the distance between the stereoscopic virtual object and the periphery of the first side of the bounded region is not greater than the threshold.

14. The device of claim 10, wherein the one or more procedures further cause the device to: At least a portion of the three-dimensional virtual object is retracted by the periphery occlusion of the first side of the bounded region.

15. The device of claim 10, wherein the first side of the bounded region is displayed in a real CGR environment generated by a native user computer, wherein the one or more programs further cause the device to: In the native user CGR environment characterized by the first set of world coordinates, the portion of the stereoscopic virtual object protruding from the first side of the bounded region is illuminated by a light source.

16. A non-transitory memory storing one or more programs, said one or more programs causing the device to: The first side of the bounded region is shown according to the first set of world coordinates; as well as Displaying a three-dimensional virtual object within the perimeter of the first side of the bounded region using a second set of world coordinates different from the first set of world coordinates, including: When the distance between the three-dimensional virtual object and the periphery of the first side of the bounded region is greater than a threshold, a portion of the three-dimensional virtual object is displayed as protruding from the first side of the bounded region; as well as When the distance between the 3D virtual object and the periphery of the first side of the bounded region is not greater than the threshold, the 3D virtual object is shrunk back into the first side of the bounded region.

17. The non-transitory memory of claim 16, wherein displaying the portion of the stereoscopic virtual object as protruding from the first side of the bounded region comprises: The three-dimensional virtual object is displayed in an animated manner based on the second set of world coordinates without user input.

18. The non-transitory memory of claim 16, wherein in response to user input to the bounded region, the portion of the stereoscopic virtual object is displayed as protruding from the first side of the bounded region.

19. The non-transitory memory of claim 16, wherein in response to user input interacting with content displayed within the first side of the bounded region, the stereoscopic virtual object is retracted back into the first side of the bounded region when the distance between the stereoscopic virtual object and the periphery of the first side of the bounded region is not greater than the threshold.

20. The non-transitory memory of claim 16, wherein the one or more programs further cause the device to: At least a portion of the three-dimensional virtual object is retracted by the periphery occlusion of the first side of the bounded region.

Images