Inserting images from real environment into virtual environment

Abstract

The invention relates to inserting images from a real environment into a virtual environment. The present disclosure relates to techniques for inserting images from a real environment into a virtual environment. While presenting (e.g., displaying) a virtual environment at an electronic device, a proximity of the electronic device to a physical object located in the real environment is detected. In response to detecting that the proximity of the electronic device to the physical object is less than the first threshold distance, the image of the physical object is isolated from other images of the real environment. The isolated image of the physical object is inserted in the virtual environment at a location corresponding to the location of the physical object in the real environment. The image of the physical object has a first visibility value associated with a proximity of the electronic device to the physical object.

Description

[0001] This application is a divisional application of the invention patent application with application date of September 6, 2019, application number 201980056533.2, and invention name “Inserting images from a real environment into a virtual environment”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 728,610, filed on September 7, 2018, entitled “TRANSITIONING BETWEEN IMAGERY AND SOUNDS OF A VIRTUAL ENVIRONMENT AND A REAL ENVIRONMENT,” U.S. Provisional Patent Application No. 62 / 729,154, filed on September 10, 2018, entitled “INSERTING IMAGERY FROM A REALENVIRONMENT INTO A VIRTUAL ENVIRONMENT,” and U.S. Provisional Patent Application No. 62 / 892,870, filed on August 28, 2019, entitled “TRANSITIONING BETWEEN IMAGERY AND SOUNDS OF AVIRTUAL ENVIRONMENT AND A REAL ENVIRONMENT,” which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present patent application relates generally to virtual environments, and more particularly to techniques for inserting imagery from a real environment into a virtual environment. Background Art

[0005] The computer may project or partially superimpose the computer-generated images on the user's field of view to provide a virtual environment that can be experienced by the user. The virtual environment can be based on different types of reality. The electronic device optionally detects the user's real movements and projects and simulates those movements within a series of visual images or videos of the virtual environment. By projecting or simulating these movements within the virtual environment, the user may appear to move to other locations within the virtual environment. Summary of the invention

[0006] The present disclosure describes techniques for inserting images from a real environment into a virtual environment. In response to an electronic device detecting that a user (and / or the device) is approaching a physical object in the real environment, an image from the real environment is inserted. The image of the physical object is then isolated and inserted into the virtual environment. As described herein, these techniques provide an enhanced degree of safety to a user by providing an image of a physical object before the user comes into contact with the physical object in some exemplary embodiments.

[0007] According to some embodiments, a method is described. The method includes: at an electronic device having one or more displays: presenting a virtual environment using the one or more displays; detecting the proximity of the electronic device to a physical object located in a real environment; in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: isolating the image of the physical object from other images of the real environment; presenting the virtual environment, wherein the isolated image of the physical object is inserted into the virtual environment at a position corresponding to the position of the physical object in the real environment, and the image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object; and in response to detecting that the proximity of the electronic device to the physical object is greater than the first threshold distance: presenting the virtual environment without the isolated image of the physical object.

[0008] In some embodiments, the method further includes: in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: displaying the virtual environment with a second visibility value associated with the proximity of the electronic device to the physical object. In some embodiments, the method further includes: in response to detecting that the proximity of the electronic device to the physical object is less than a second threshold distance: modifying the first visibility value of the image of the physical object. In some embodiments, the method further includes: in response to detecting that the proximity of the electronic device to the physical object is less than a third threshold distance: stopping presenting the virtual environment; and providing a view of the real environment.

[0009] In some embodiments, presenting the virtual environment with the isolated image of the physical object includes compositing the image of the virtual environment with the isolated image of the physical object. In some embodiments, the compositing uses an alpha channel associated with the image of the virtual environment and the image of the physical object. In some embodiments, presenting the virtual environment with the isolated image of the physical object includes aligning a user's perspective of the physical object with a user's perspective of the virtual environment.

[0010] In some embodiments, the method also includes providing virtual environment audio associated with the virtual environment using one or more speakers. And in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: providing a combined mix of the virtual environment audio and the real environment audio, wherein the amount of the virtual environment audio in the combined mix is ​​associated with the proximity of the electronic device to the physical object. In some embodiments, the virtual environment audio includes multiple audio objects, and wherein providing the combined mix includes cross-fading the multiple audio objects with the real environment audio. In some embodiments, the amount of cross-fading applied to one or more first audio objects in the multiple audio objects is associated with the prominence of the one or more first audio objects in the virtual environment.

[0011] In some embodiments, presenting a virtual environment with isolated images of physical objects includes synthesizing the image of the virtual environment with the isolated image of the physical object using an alpha channel associated with the image of the virtual environment and the isolated image of the physical object, and an amount of cross-fade applied to one or more second audio objects in the plurality of audio objects is associated with the value of the corresponding alpha channel.

[0012] According to some embodiments, a non-transitory computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for performing the following operations: presenting a virtual environment using one or more displays; detecting the proximity of the electronic device to a physical object located in a real environment; in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: isolating the image of the physical object from other images of the real environment; and presenting the virtual environment, wherein the isolated image of the physical object is inserted into the virtual environment at a position corresponding to the position of the physical object in the real environment, and the image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object; and in response to detecting that the proximity of the electronic device to the physical object is greater than the first threshold distance: presenting the virtual environment without the isolated image of the physical object.

[0013] According to some embodiments, a transient computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for performing the following operations: presenting a virtual environment using one or more displays; detecting the proximity of the electronic device to a physical object located in a real environment; in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: isolating the image of the physical object from other images of the real environment; presenting the virtual environment, wherein the isolated image of the physical object is inserted into the virtual environment at a position corresponding to the position of the physical object in the real environment, and the image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object; and in response to detecting that the proximity of the electronic device to the physical object is greater than the first threshold distance: presenting the virtual environment without the isolated image of the physical object.

[0014] According to some embodiments, an electronic device is described, which includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing the following operations: presenting a virtual environment using one or more displays; detecting the proximity of the electronic device to a physical object located in a real environment; in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: isolating the image of the physical object from other images of the real environment; presenting the virtual environment, wherein the isolated image of the physical object is inserted into the virtual environment at a position corresponding to the position of the physical object in the real environment, and the image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object; and in response to detecting that the proximity of the electronic device to the physical object is greater than the first threshold distance: presenting the virtual environment without the isolated image of the physical object.

[0015] According to some embodiments, an electronic device is described. The electronic device includes a device for presenting a virtual environment; a device for detecting the proximity of the electronic device to a physical object located in a real environment; a device for performing the following operations in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance: isolating the image of the physical object from other images of the real environment; presenting the virtual environment, wherein the isolated image of the physical object is inserted into the virtual environment at a position corresponding to the position of the physical object in the real environment, and the image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object; and a device for performing the following operations in response to detecting that the proximity of the electronic device to the physical object is greater than the first threshold distance: presenting the virtual environment without the isolated image of the physical object. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A to Figure 1B Depicted are exemplary systems for use in various augmented reality techniques.

[0017] Figure 2 An example of a real environment in which a virtual environment is to be viewed according to some embodiments is shown.

[0018] Figure 3 An example of a virtual environment according to some embodiments is shown.

[0019] FIG. 4A to FIG. 4E Exemplary techniques for transitioning between a virtual environment and a real environment are shown according to some embodiments.

[0020] FIG. 5A to FIG. 5C Exemplary techniques for inserting images from a real environment into a virtual environment are shown according to some embodiments.

[0021] Figure 6 is a flowchart illustrating an exemplary process performed by an electronic device according to some embodiments.

[0022] Figure 7 is a flowchart illustrating an exemplary process performed by an electronic device according to some embodiments.

[0023] Figure 8 is a flowchart illustrating an exemplary process performed by an electronic device according to some embodiments.

[0024] The embodiments shown in the drawings are exemplary only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0025] Various examples of electronic systems and techniques for using such systems in connection with various augmented reality techniques are described.

[0026] The physical set (also referred to as the real environment) refers to the world that individuals can sense and / or interact with without the use of electronic systems. A physical set, such as a physical park, includes physical elements (also referred to as physical objects), such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with the physical set, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0027] In contrast to a physical set, an augmented reality (ER) set (also referred to as a virtual environment) refers to a fully (or partially) computer-generated set that a person can sense and / or otherwise interact with using an electronic system. In an ER, the movement of a person is locally monitored, and in response thereto, at least one property corresponding to at least one virtual object in the ER set is changed in a manner consistent with one or more laws of physics. For example, in response to the ER system detecting that the person is looking up, the ER system can adjust various audio and graphics presented to the person in a manner consistent with how such sounds and appearances would change in a physical set. Adjustments to properties of virtual objects in the ER set can also be made in response to representations of movement (e.g., voice commands).

[0028] A person may sense and / or interact with ER objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER scenery, a person may sense and / or interact with only sound objects.

[0029] Virtual reality (VR) is an example of ER. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0030] Mixed reality (MR) is another example of ER. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0031] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0032] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0033] AR scenery may also refer to an augmented scenery in which a representation of a physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery can be graphically modified (e.g., enlarged) so that the modified portion can still represent the originally captured image (but not an exact copy). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0034] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0035] Various systems allow people to sense and / or interact with the ER scenery. For example, a head-mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into the head-mounted system. The head-mounted system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or translucent display may also be included in the head-mounted system. A translucent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eyes. The display may also include an LED, an OLED, a silicon-based liquid crystal, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. A transparent or translucent display may, for example, selectively transition between a transparent / translucent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0036] Figure 1A and Figure 1B An exemplary system 100 for use in various augmented reality techniques is shown.

[0037] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0038] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop), and other elements of system 100 are implemented in a second device (e.g., a head mounted device). In some examples, device 100a is implemented in the base station device or the second device.

[0039] like Figure 1BAs shown, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate through a communication bus 150 of device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate through a communication bus 150 of device 100c.

[0040] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the following techniques.

[0041] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0042] System 100 includes display 120. One or more displays 120 may have an opaque display. One or more displays 120 may have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to the individual's eyes. One or more displays 120 may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one example, a transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. Other examples of one or more displays 120 include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, a tablet computer, a smart phone, and a desktop or laptop computer. Alternatively, system 100 may be designed to receive an external display (e.g., a smart phone). In some examples, system 100 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical set (e.g., onto a physical surface or as a hologram).

[0043] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0044] The system 100 includes an image sensor 108. The one or more image sensors 108 optionally include one or more visible light image sensors operable to acquire images of physical elements from the physical scenery, such as a charge coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor. The one or more image sensors also optionally include one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the physical scenery. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scenery. The one or more image sensors 108 also optionally include one or more event cameras configured to capture movement of physical elements in the physical scenery. The one or more image sensors 108 also optionally include one or more depth sensors configured to detect the distance of the physical elements from the system 100. In some examples, the system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scenery around the system 100. In some examples, the image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical elements in the physical environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses one or more image sensors 108 to detect the position and orientation of the system 100 and / or one or more displays 120 in the physical set. For example, the system 100 uses one or more image sensors 108 to track the position and orientation of one or more displays 120 relative to one or more fixed elements in the physical set.

[0045] In some examples, system 100 includes microphone 112. System 100 detects sounds from a user and / or a user's physical set using one or more microphones 112. In some examples, one or more microphones 112 include a microphone array (including a plurality of microphones) that optionally operate in concert to identify ambient noise or localize a sound source in the space of the physical set.

[0046] The system 100 includes an orientation sensor 110 for detecting the orientation and / or movement of the system 100 and / or the display 120. For example, the system 100 uses one or more orientation sensors 110 to track changes in the position and / or orientation of the system 100 and / or one or more displays 120, such as relative to physical elements in a physical scene. The orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0047] Reference now Figure 2 , Figure 3 , FIG. 4A to FIG. 4E , FIG. 5A to FIG. 5C , Figure 6 , Figure 7 and Figure 8 , describes exemplary techniques for converting between images and sounds of a virtual environment and images and sounds of a real environment, and techniques for inserting images from a real environment into a virtual environment. In some examples, the conversion and / or insertion occurs in response to an electronic device detecting an event such as a signal from an input device, proximity to a physical object (also referred to as a physical element), and / or a triggering sound. In some exemplary embodiments, the technology enhances user convenience and provides enhanced awareness to the user by providing images and sounds of the real environment at the user's request and / or in response to obstacles or warnings in the real environment.

[0048] Figure 2 An example of a real environment 200 (also referred to as a physical set) is shown in which a virtual environment (also referred to as an ER set) is to be viewed according to some embodiments. The real environment 200 includes physical objects 202 (also referred to as physical elements), such as a table 202-a, a computer 202-b, and a wall 202-c. Although the real environment 200 is in Figure 2 2 is shown as a room with physical objects 202, but it should be understood that real environment 200 can be any real-world location at which a virtual environment is to be viewed.

[0049] The real environment 200 is visible to the user of the device 100a, as shown in FIG. Figure 1A to Figure 1B In some embodiments, the real environment 200 is displayed to the user via a video pass-through mode of the device 100a. In other embodiments, a substantially direct view of the real environment 200 is provided to the user, such as using a heads-up display.

[0050] Figure 3 An example of a virtual environment 300 (also referred to as an ER set) according to some embodiments is shown. The virtual environment 300 includes virtual objects 302, such as a virtual robot 302-a, a virtual planet 302-b, and a virtual spaceship 302-c. Although the virtual environment 300 is Figure 3 Although shown as the deck of a spaceship, it should be understood that virtual environment 300 may be any environment that displays one or more virtual objects.

[0051] In some examples, during operation, device 100a (eg, Figure 1A to Figure 1BThe device 100a also uses the speaker 118 to provide the user with audio associated with the virtual environment 300. The virtual environment audio includes one or more audio objects associated with various components of the virtual environment 300. For example, a first audio object may be associated with a virtual robot 302-a (e.g., a sound, a beep, etc.), a second audio object may be associated with a virtual spaceship 302-c (e.g., an engine sound, an alarm, etc.), and a third audio object may be associated with the ambient noise of the virtual environment 300 (e.g., an engine sound, a beep, etc.). Each of the audio objects is mixed together to form the virtual environment audio. In some embodiments, mixing the audio objects includes adjusting the volume level, spatial placement, and / or spectrum of each audio object so that each audio object is integrated into the virtual environment audio. In some embodiments, the volume level, spatial placement, and / or spectrum of each audio object is adjusted based on the position of the associated virtual object 302 in the virtual environment 300 and the position and orientation of the user's head relative to the virtual object 302. In this way, when the user hears the virtual environment audio, the sound corresponding to each audio object appears to be emanating from the virtual position of the associated virtual object 302 in the virtual environment.

[0052] When displaying the virtual environment 300 and providing the virtual environment audio, the device 100a detects the user's movement, including detecting the user's position within the real environment 200 and the orientation of the user's head (e.g., where the user is looking). As the user moves, the view of the virtual environment 300 changes to correspond to the user's current position and line of sight. In addition, the mix of provided audio objects changes to correspond to the current position and orientation of the user's ears. For example, the volume level, spatial placement, and / or spectrum of each audio object changes so that the sound appears to be emitted from a consistent virtual location as the user's position and / or orientation changes.

[0053] FIG. 4A to FIG. 4E 300 (see reference 2) according to some embodiments. Figure 3 ) and the real environment 200 (as shown in reference Figure 2 An exemplary technique for converting between the above) is provided. Figure 4A As shown, the view of virtual environment 300 begins to transition to a view of real environment 200. When the transition begins, the view of real environment 200 is at least partially visible (e.g., real environment 200 and virtual environment 300 are superimposed on each other) while the view of virtual environment 300 is provided. Additionally, in some embodiments, audio from the real environment is at least partially audible while the view of real environment 200 becomes at least partially visible.

[0054] When switching from a view of virtual environment 300 to a view of real environment 200, the image of virtual environment 300 is composited (e.g., fused) with the image of real environment 200. The compositing is performed by using visibility values ​​associated with the image of each environment to combine the environments with each other. In some embodiments, the visibility values ​​correspond to alpha channel information in the image of each environment. The visibility values ​​are used to adjust the transparency of each environment. Prior to the transition, virtual environment 300 has no transparency (e.g., alpha = 1.0). As the transition begins (such as when the image of the virtual environment 300 is switched to the real environment 200), the image of the virtual environment 300 is composited (e.g., fused) with the image of the real environment 200. The compositing is performed by using visibility values ​​associated with the image of each environment to combine the environments with each other. In some embodiments, the visibility values ​​correspond to alpha channel information in the image of each environment. The visibility values ​​are used to adjust the transparency of each environment. Prior to the transition, virtual environment 300 has no transparency (e.g., alpha = 1.0). As the transition begins (such as when the image of the real environment 200 is switched to the real environment 200), the image of the virtual environment 300 is composited (e.g., fused) with the image of the real environment 200. Figure 4A ), the transparency of the virtual environment 300 is increased (eg, α=0.9), and an image of the real environment 200 is added to the partially transparent image of the virtual environment 300, wherein the real environment 200 has a complementary visibility value (eg, α=0.1).

[0055] In some embodiments, compositing is performed using different visibility values ​​for different virtual objects in the virtual environment 300. For example, as the transition begins, a first virtual object (e.g., virtual robot 302-a) has a first visibility value (e.g., α=0.8), while a second virtual object (e.g., virtual spaceship 302-c) has a different second visibility value (e.g., α=0.9). These virtual objects are added to the imagery of the real environment 200, where overlapping portions of the imagery of the real environment 200 have complementary visibility values ​​(e.g., α=0.2 and α=0.1, respectively). This allows different portions of the virtual environment 300 to fade in or out faster or slower than other portions of the virtual environment 300. In some embodiments, the visibility values ​​correspond to alpha channel information for individual pixels of the virtual environment 300 and the real environment 200.

[0056] In some embodiments, compositing is performed using different visibility values ​​for different objects in the real environment 200. For example, objects that define the outline of the real environment 200 (e.g., floor, wall, ceiling, window, table) may fade in faster than other objects in the real environment 200 (e.g., objects on a table, artwork on a wall).

[0057] In some embodiments, when the views of the virtual environment 300 and the real environment 200 are combined, the user's perspective of seeing one or both of the environments changes. For example, when a view of the real environment 200 is provided using a camera that transmits an image of the real environment, the perspective provided by the camera may be different from the perspective from which the user sees the virtual environment 300 (e.g., the camera's perspective is two inches in front of the user's eyes). Therefore, the device 100a changes the perspective of the real environment 200 or the virtual environment 300 so that the two environments are aligned. For example, the device 100a modifies the image of the real environment 200 so that the apparent perspective of the user viewing the real environment 200 is aligned with the perspective of the virtual environment 300 (e.g., the perspective of the real environment 200 moves back two inches). Alternatively, in some examples, the device 100a changes the perspective of the virtual environment 300 so that the perspective of the virtual environment 300 is aligned with the perspective of the camera that captures the real environment 200 (e.g., the perspective of the virtual environment 300 moves forward two inches).

[0058] In some embodiments, the volume levels of audio objects associated with virtual objects 302 in the virtual environment 300 change as the view of the virtual environment 300 transitions to the view of the real environment 200. In some embodiments, the volume levels of certain audio objects change at a different rate than other audio objects. For example, the volume level of a prominent audio object (e.g., audio associated with a main character, such as the virtual robot 302-a) may decrease faster than the volume level of an audio object for ambient background noise, or vice versa.

[0059] In some embodiments, the volume level of the audio object is associated with the visibility value of the virtual environment and the real environment. Therefore, when the virtual environment 300 becomes less visible, the volume level of the audio object associated with the virtual environment 300 is reduced by a corresponding amount. For example, as the visibility value of the virtual environment 300 decreases (e.g., α changes from 1.0 to 0.9), the volume level of the audio object associated with the virtual environment 300 is also reduced (e.g., -6dB). Various algorithms can be used to associate the volume level of the audio object with the visibility value of the virtual environment and the real environment. In the above example, a 10% change in the visibility value corresponds to a 6dB reduction in volume. However, other ratios, functions, or algorithms can be used together with the visibility value to adjust the volume level of the audio object. In addition, as described above, in some embodiments, the volume level of some audio objects changes at a rate different from that of other audio objects. In these embodiments, different ratios, functions, or algorithms can be used to adjust the volume level of each audio object based on the corresponding visibility value. For example, when the visibility value of the virtual environment decreases by 10% (e.g., α changes from 1.0 to 0.9), a prominent audio object (e.g., audio associated with a primary character, such as virtual robot 302-a) decreases by a first amount (e.g., -9dB), while the volume level of a less prominent audio object (e.g., ambient background noise) decreases by a different second amount (e.g., -6dB).

[0060] In some embodiments, audio objects associated with the virtual environment 300 are mixed with audio from the real environment 200. In some embodiments, mixing is performed using a loudness transfer function (e.g., crossfade) applied to the audio objects associated with the virtual environment 300 and the audio from the real environment 200. The loudness transfer function can use various techniques for mixing audio, such as the "sine and cosine theorem", where the crossfade center point is -3dB (i.e., 0.707 linear). In some embodiments, the audio from the real environment 200 is detected using the microphone 112 of the device 100a, and then mixed with the audio objects associated with the virtual environment based on the volume level of each corresponding audio object. In some embodiments, the volume levels of the audio objects and the audio from the real environment 200 are associated with the visibility values ​​of the virtual environment and the real environment. Therefore, in some embodiments, the value of the loudness transfer function is associated with the visibility value. For example, as the visibility of the virtual environment 300 decreases (e.g., α changes from 1.0 to 0.9) and the visibility of the real environment 200 increases (e.g., α changes from 0.0 to 0.1), the audio objects associated with the virtual environment 300 cross-fade with the audio from the real environment 200 using loudness transfer function values ​​corresponding to the change in visibility (e.g., the volume level of the audio objects decreases and is mixed with the audio from the real environment 200 at less than its full volume level). As described above, in some embodiments, the volume level of certain audio objects (e.g., audio objects associated with prominent characters in the virtual environment 300) changes at a different rate than other audio objects (e.g., audio objects associated with the ambient background noise of the virtual environment 300). In some embodiments, these different rates of change in volume level are applied when the audio objects are mixed with the audio from the real environment 200.

[0061] In some embodiments, other forms of audio transitions are used instead of crossfades when the image changes. These other audio transitions include straight cuts (e.g., the audio changes immediately without a gradual crossfade), poke sound clips (e.g., the audio changes before the image of the environment changes), drag sound clips (e.g., the current audio continues for a period of time after the image of the environment changes), or using the same audio during the transition (e.g., music or ambient noise from the virtual environment 300 continues after the image changes to the real environment 200).

[0062] In some embodiments, the audio conversion includes conversion between acoustic characteristics (e.g., reverberation, decay time (T60), frequency response, time energy frequency (TEF), energy time curve (ETC), etc.) of the real environment 200 and the virtual environment 300. In some embodiments, the acoustic characteristics of audio objects associated with the virtual environment 300 are modified to correspond to the acoustic characteristics of the real environment 200. For example, when in the virtual environment 300, the audio objects associated with the virtual avatar (e.g., the virtual robot 302-a) include acoustic characteristics corresponding to the virtual environment 300 (e.g., the sound emitted by the virtual robot 302-a is processed to have acoustic characteristics corresponding to the interior of the spaceship). When the audio is converted to the real environment 200, the acoustic characteristics of the audio objects associated with the virtual avatar are modified to correspond to the acoustic characteristics of the real environment 200 (e.g., the sound emitted by the virtual robot 302-a is processed to have acoustic characteristics corresponding to the user's room). In this way, when transitioning from the virtual environment 300 to the real environment 200, the audio objects associated with the virtual avatar are more realistically integrated with the audio in the real environment 200.

[0063] In some embodiments, the acoustic characteristics of the audio in the real environment 200 are modified to correspond to the acoustic characteristics of the virtual environment 300. For example, when in the real environment 200, the voice of a person speaking in the real environment 200 (e.g., the user's own voice) has acoustic characteristics corresponding to the real environment (e.g., the voice of the person speaking is unprocessed and has acoustic characteristics corresponding to the natural acoustics of the room in which the person is speaking). When the audio is converted to the virtual environment 300, the acoustic characteristics of the audio in the real environment (e.g., the person's speaking voice) are modified to correspond to the acoustic characteristics of the virtual environment 300 (e.g., the person's voice is processed to have acoustic characteristics corresponding to the interior of a spaceship). In this way, when converting from the real environment 200 to the virtual environment 300, the audio in the real environment 200 is more realistically integrated with the audio in the virtual environment 300.

[0064] In some embodiments, the acoustic characteristics of the real environment 200 are determined by the device 100a. In some embodiments, the acoustic characteristics of the real environment 200 are predetermined acoustic characteristics assigned to an environment (e.g., a large room, a small room, a concert hall, a cathedral). In some embodiments, the acoustic characteristics of the real environment 200 are determined by analyzing the acoustic characteristics and / or physical characteristics of the real environment 200. For example, the acoustic characteristics of the real environment 200 can be determined by emitting a predetermined test sound (e.g., a chirp or a frequency sweep) and analyzing the changes in the test sound caused by the real environment 200 (e.g., the test sound is reflected to the physical characteristics of the room, and the reflection is analyzed). For another example, the physical characteristics of the room (e.g., the layout and material composition of the walls, floors, ceilings, and objects in the room) can be detected. The acoustic model of the real environment 200 can then be estimated based on the physical characteristics of the real environment 200.

[0065] In some embodiments, before applying the acoustic characteristics of the virtual environment 300, the audio in the real environment 200 is filtered to remove the acoustic characteristics of the real environment 200. In some embodiments, the acoustic characteristics of the real environment 200 are gradually filtered out (e.g., gradually removed) as the acoustic characteristics of the virtual environment 300 are applied to the audio in the real environment 200. In this way, as the view switches to the virtual environment 300, the audio in the real environment 200 is gradually modified to correspond to the acoustic characteristics of the audio objects of the virtual environment 300.

[0066] In some embodiments, the acoustic characteristics applied to the audio in the real environment 200 are based in part on the audio output device (e.g., closed-back headphones, open-back headphones, in-ear headphones, external earphones, or speakers). When using an audio output device that allows the audio in the real environment 200 to be heard without significant distortion or loudness reduction (e.g., open-back headphones, external earphones, speakers), the audio in the real environment 200 is not processed (e.g., amplified or filtered) while providing a view of the real environment 200. When using an audio output device that distorts or reduces the loudness of the audio in the real environment 200 (e.g., closed-back headphones, in-ear headphones), the audio in the real environment 200 is processed (e.g., amplified or filtered) to apply the acoustic characteristics of the real environment 200 while providing a view of the real environment 200. In this way, distortion and / or loudness reduction are mitigated (e.g., the user hears the audio in the real environment 200 in a manner similar to a user who does not use closed-back headphones or in-ear headphones).

[0067] In some examples, the conversion of the view to the real environment 200 is in response to the electronic device 100a detecting a conversion event. In some embodiments, detecting a conversion event includes receiving a signal from an input device such as a rotary input mechanism (e.g., a knob) of the electronic device 100a. When displaying the virtual environment 300, if the rotary input mechanism is detected to rotate along a first direction, the real environment 200 becomes at least partially visible, and the virtual environment 300 becomes less visible (e.g., as described above, the two environments merge together). The visibility of each environment is associated with the rotation amount (and / or rotation speed) of the rotary input mechanism. Compared to the relatively small rotation amount (and / or slow rotation) of the rotary input mechanism, a larger rotation amount (and / or faster rotation) causes the real environment 200 to become more visible and the virtual environment 300 to become less visible. In other words, the visibility value for fusing the two environments is associated with the rotation amount (and / or rotation speed) of the rotary input mechanism. This allows the user to control the visibility of each environment (real and virtual). In some embodiments, the rotation of the rotary input mechanism has a nonlinear relationship with the visibility value. For example, quickly rotating the rotary input mechanism half a turn results in different visibility values ​​for the environment compared to slowly rotating the rotary input mechanism half a turn.

[0068] In some embodiments, detecting a transition event includes detecting that the proximity of the electronic device 100a (and / or the user) to a physical object (also referred to as a physical element) in the real environment 200 is less than a threshold distance (e.g., 1 foot). For example, when displaying the virtual environment 300, if the electronic device 100a (and / or the user) approaches the table 202-a of the real environment 200, the real environment 200 becomes at least partially visible and the virtual environment 300 becomes less visible (e.g., as described above, the two environments merge together). In some examples, the visibility of each environment is associated with the proximity of the device 100a (and / or the user) to the physical object (e.g., the table 202-a). For example, when the device 100a (and / or the user) moves closer to the physical object, the real environment 200 becomes more visible and the virtual environment 300 becomes less visible. This provides an indication or alert to the user before the user contacts the physical object.

[0069] In some embodiments, device 100a detects proximity of device 100a and / or user to a physical object via one or more internal and / or external image sensors. In some embodiments, the threshold distance is predefined (e.g., predefined by an operating system of device 100a or set by a user of device 100a).

[0070] In some embodiments, detecting a transition event includes detecting a triggering sound, such as someone speaking or an alerting sound (phone call, message, alarm, etc.). For example, when displaying the virtual environment 300, if a triggering sound is detected, the real environment 200 becomes at least partially visible and the virtual environment 300 becomes less visible (e.g., as described above, the two environments merge together). By providing a view of the real environment 200 that is at least partially visible, the user can identify the source of the triggering sound. In some embodiments, the representation of the source of the triggering sound (e.g., a person speaking, a phone, a computer, etc.) is more visible than other elements of the real environment 200.

[0071] like Figure 4B As shown, the transition from the view of the virtual environment 300 to the view of the real environment 200 continues by the real environment 200 becoming more visible. In some embodiments, the transition continues in response to an input device such as an additional input on a rotary input mechanism. In some embodiments, the transition continues in response to detecting that the proximity of the device 100a to the physical object in the real environment 200 has decreased. Therefore, when the user approaches the physical object, the physical object (and the real environment 200) becomes more visible. In addition, in some embodiments, when providing audio associated with the real environment 200, the volume level of the audio object associated with the virtual environment 300 continues to decrease. In some embodiments, the volume level of some audio objects (e.g., audio objects associated with prominent virtual objects, such as virtual robots 302-a) decreases faster than other audio objects (e.g., audio objects associated with the ambient background noise of the virtual environment 300), or vice versa.

[0072] like Figure 4C As shown, the view of the virtual environment 300 has been approximately half-converted to the view of the real environment, making the two environments roughly equally visible. In addition, in some embodiments, the audio objects associated with the virtual environment 300 are provided at a volume level roughly the same as the audio associated with the real environment 200.

[0073] like Figure 4D As shown, the transition from the view of virtual environment 300 to the view of real environment 200 continues by virtual environment 300 becoming less visible than real environment 200 .

[0074] like Figure 4E As shown, the transition from the view of virtual environment 300 to the view of real environment 200 continues by virtual environment 300 continuing to become less visible than real environment 200 .

[0075] When the conversion is complete, device 100a provides Figure 2 A view of the real environment 200 is shown, while a view of the virtual environment 300 is not provided.

[0076] When providing a view of the real environment 200, a substantially reverse process may be performed to transition from a view of the real environment 200 to a view of the virtual environment 300. The process is performed in response to detecting another transition event. In some embodiments, the transition event includes receiving a signal from an input device (e.g., a rotational input mechanism in the opposite direction), or detecting that the proximity of the device 100a to a physical object is greater than a threshold distance (e.g., 1 foot).

[0077] In addition, during the transition process, the transition may be reversed so that the view of the virtual environment 300 becomes more visible while the view of the real environment 200 becomes less visible (or vice versa). The reversal of the transition process is in response to detecting a second transition event. In some embodiments, detecting the second transition event includes receiving a second signal from an input device (e.g., a rotational input mechanism). For example, if a user begins to rotate a rotary input mechanism in a first direction that causes a transition from a view of the virtual environment 300 to a view of the real environment 200, rotation of the rotary input mechanism in an opposite second direction causes the transition to be reversed. In some embodiments, rotation of the rotary input mechanism in the opposite second direction has a non-linear relationship with the visibility value of the environment. For example, quickly rotating the rotary input mechanism half a circle results in a different visibility value for the environment than slowly rotating the rotary input mechanism half a circle.

[0078] The reversal of the transformation may occur at different times during the transformation process. For example, if the transformation is partially completed, such as Figure 4B As shown, and a second transition event is detected, the transition process is reversed, and the view of the real environment becomes less visible, as shown in FIG. Figure 4A shown.

[0079] In some embodiments, detecting the second transition event includes detecting that the proximity of the device 100a to the physical object has increased. In response to the increase in proximity, the transition process is reversed as described above.

[0080] This process of switching back and forth between the view of the virtual environment 300 and the view of the real environment 200 provides the user with visual (and in some embodiments, auditory) cues that they are approaching an obstacle without completely interrupting the user's experience in the virtual environment 300. By gradually fading in the view of the real environment 200, the user may be less easily disoriented than if the virtual environment 300 were suddenly replaced with a view of the real environment 200. Then, when the user is no longer close to the obstacle, the view of the real environment 200 fades out, and the user's experience in the virtual environment 300 proceeds more naturally than if the virtual environment 300 suddenly reappeared.

[0081] FIG. 5A to FIG. 5C200 (eg, reference Figure 2 The image of the virtual environment 300 (as shown in reference Figure 3 As described in the exemplary techniques of Figure 5A As shown, an image of a physical object 202-a (also referred to as a physical element) (e.g., a table) is inserted into the virtual environment 300. When the image of the physical object 202-a is initially inserted into the virtual environment 300, the image of the physical object 202-a is at least partially visible.

[0082] In response to detecting that the electronic device 100a (and / or the user) is within a proximity of less than a threshold distance (e.g., 1 foot) to a physical object 202-a in the real environment 200, an image of the physical object 202-a is inserted. For example, when displaying the virtual environment 300, if the electronic device 100a approaches the physical object 202-a in the real environment 200, the image of the physical object 202-a becomes at least partially visible within the virtual environment 300 at a location corresponding to the real-world location of the physical object 202-a in the real environment 200. In some embodiments, when the image of the physical object 202-a is inserted, the image of the virtual environment 300 (e.g., at a location corresponding to the image of the physical object 202-a) also becomes less visible.

[0083] In some embodiments, the device 100a detects the proximity of the device 100a and / or a user (e.g., an arm, leg, or head of a user's body) to the physical object 202-a via one or more internal and / or external image sensors. In some embodiments, the threshold distance is predefined (e.g., predefined by an operating system of the device 100a or set by a user of the device 100a).

[0084] In some embodiments, the image of the physical object 202-a is isolated from other images of the real environment 200. For example, when the device 100a detects that the proximity to the physical object 202-a is less than a threshold distance (e.g., 1 foot), the image of the physical object 202-a is isolated from the surrounding images of the real environment 200. The isolated image of the physical object 202-a is then inserted into the image of the virtual environment 300. In some embodiments, for example, when the physical object 202a is in the field of view of one or more sensors (e.g., image sensors) of the device 100a, the image of the physical object 202a is detected via the one or more sensors of the device 100a. In some embodiments, in addition or alternatively, the physical object 202-a is detected via one or more sensors (e.g., image sensors, depth sensors, motion sensors, accelerometers, gyroscopes) external to the device 100a (e.g., sensors installed in the real environment 200).

[0085] In some embodiments, a generic object shape (e.g., a generic table, cube, sphere, etc.) is used to depict the image of the physical object 202-a within the virtual environment 300. In some embodiments, the generic object shape is visually similar to the corresponding physical object 202-a in the real environment 200 (e.g., similar size, similar color, etc.).

[0086] In some embodiments, the image of the physical object 202-a is synthesized (e.g., fused) with the image of the virtual environment 300. Synthesis is performed by using visibility values ​​associated with the image of the physical object 202-a and the image of the virtual environment 300 to combine the images with each other. In some embodiments, the visibility value corresponds to alpha channel information in the image. The visibility value is used to adjust the transparency of the image of the physical object 202-a and the image of the virtual environment 300. Prior to insertion, the virtual environment 300 has no transparency (e.g., alpha = 1.0). When the image of the physical object is initially inserted (such as Figure 5A ), wherein the transparency of the virtual environment 300 into which the physical object 202-a is inserted is slightly increased (e.g., α=0.9), and the image of the physical object 202-a is added to the partially transparent image of the virtual environment 300, wherein the image of the physical object 202-a has a complementary visibility value (e.g., α=0.1). In some embodiments, the visibility values ​​correspond to alpha channel information of the individual pixels of the virtual environment 300 and the image of the physical object 202-a. Thus, for example, before, during, and / or after a transition between a view of the virtual environment 300 and a view of the real environment 200, one or more portions of the virtual environment 300 can be displayed with a different transparency than one or more other portions of the virtual environment 300.

[0087] The visibility of the image of the physical object 202-a is associated with the proximity of the device 100a (and / or the user) to the physical object 202-a. As the device 100a moves closer to the physical object 202-a in the real environment 200, the image of the physical object 202-a inserted in the virtual environment 300 becomes more visible. In some embodiments, the image of the virtual environment 300 also becomes less visible. This provides an indication or alert to the user before the user contacts the physical object.

[0088] like Figure 5B As shown, as the proximity of the device 100a (and / or the user) decreases, the image of the physical object 202-a becomes more visible in the virtual environment 300. Additionally, in some embodiments, the image of the virtual environment 300 at the location where the image of the physical object 202-a is inserted becomes less visible. In some embodiments, as the image of the physical object 202-a becomes more visible, all of the imagery of the virtual environment 300 becomes less visible.

[0089] like Figure 5C As shown, the image of the physical object 202-a is fully visible in the virtual environment 300 (e.g., α=1.0). In response to the proximity of the device 100a (and / or the user) decreasing to less than a second threshold distance (e.g., 3 inches), the image of the physical object 202-a becomes fully visible. In addition, in some embodiments, the image of the virtual environment 300 at the location where the image of the physical object 202-a was inserted is no longer visible (e.g., α=0.0). In some embodiments, all images of the virtual environment 300 are no longer visible, such that when the proximity to the physical object 202-a decreases to less than the second threshold, only the image of the physical object 202-a is visible.

[0090] After inserting the image of the physical object 202-a into the virtual environment 300, a substantially reverse process may be performed to remove the image of the physical object 202-a from the virtual environment 300. The reversal process is responsive to detecting that the proximity of the device 100a (and / or the user) to the physical object 202-a has increased beyond a threshold distance. This reversal may occur at different times after the image of the physical object 202-a becomes visible in the virtual environment 300. For example, if the image of the physical object 202-a is partially visible, such as Figure 5B As shown, and as the proximity to the physical object 202-a increases, the image of the physical object 202-a becomes less visible, such as Figure 5A If the proximity to the physical object 202-a continues to increase to a distance greater than the threshold, the image of the physical object 202-a is removed from the virtual environment 300 and the image is displayed as shown. Figure 3 The virtual environment shown.

[0091] This process of inserting the image of the physical object 202-a in the virtual environment 300 provides the user with a visual (and, in some embodiments, an auditory) cue that they are close to an obstacle without completely interrupting the user's experience of the virtual environment 300. By inserting the image of the physical object 202-a in the virtual environment 300 while leaving other aspects of the virtual environment 300 unchanged, the user may be less easily disoriented than if all aspects of the virtual environment 300 were suddenly replaced. Then, when the user is no longer close to the obstacle, the image of the physical object 202-a disappears, and the user's experience of the virtual environment 300 proceeds more naturally than if all aspects of the virtual environment 300 were suddenly reappeared.

[0092] Figure 66 is a flowchart showing an exemplary process 600 performed by an electronic device (e.g., device 100a) according to some embodiments. In some embodiments, the electronic device has one or more displays (e.g., display 120). In some embodiments, the display is a (at least partially) transparent display. In some embodiments, the electronic device is connected to and communicates with a display separate from the device. In some embodiments, the electronic device has one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116). In some embodiments, the electronic device is connected to and communicates with one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116) separate from the device. In some embodiments, the device is a head-mounted device. In some embodiments, the electronic device is separated from the head-mounted device, but is fixed on (or configured to be fixed to) the head-mounted device. In some embodiments, the electronic device includes one or more speakers (e.g., speaker 118) for outputting audio. In some embodiments, the electronic device is connected (or configured to be connected) to (e.g., via a wireless connection, via a wired connection) and communicates with (or is configured to communicate with) one or more speakers (e.g., speaker 118) for outputting audio.

[0093] At block 602, a view of a first environment of a first type is provided. In some embodiments, the first environment of the first type is a virtual environment (e.g., virtual environment 300). In some embodiments, the first environment of the first type is a representation of a real environment (e.g., real environment 200). When initially providing the view of the first environment, the view is provided at a first visibility value (e.g., α=1.0, such as Figure 3 ) provides the environment.

[0094] At block 604, first ambient audio associated with the first environment is provided. In some embodiments, the first ambient audio includes audio associated with a virtual environment (such as virtual environment 300, such as Figure 3 In some embodiments, the audio objects are mixed together to form the first ambient audio. In some embodiments, the mixing of the audio objects includes adjusting the volume level, spatial placement, and / or spectrum of each audio object so that each audio object is integrated into the first ambient audio. In some embodiments, the volume level, spatial placement, and / or spectrum of each audio object are adjusted based on the position of the associated virtual object (e.g., virtual object 302) in the virtual environment (e.g., virtual environment 300) and the position and orientation of the user's head relative to the virtual object.

[0095] At block 606, a transition event is detected. In some embodiments, detecting a transition event includes receiving a signal from an input device. In some embodiments, the input device includes a rotary input mechanism (e.g., a knob). In some embodiments, rotation of the rotary input mechanism has a non-linear relationship with the received signal (e.g., rotating the knob quickly produces a different signal than rotating the knob slowly).

[0096] In some embodiments, detecting a transition event includes detecting a user's contact with a physical object in the real environment (e.g., table 202-a, such as Figure 2 In some embodiments, detecting a conversion event includes detecting a triggering sound.

[0097] At block 608, in response to detecting the transition event, providing a combined view of the first environment and a second environment of a second type (such as FIG. 4A to FIG. 4E ). The combined view includes an image of a first environment at a first visibility value and an image of a second environment at a second visibility value. In some embodiments, the visibility values ​​correspond to alpha channel information in the image of each environment. The visibility values ​​are used to adjust the transparency of each environment. In some embodiments, the first visibility value and the second visibility value are based on a received signal from an input device. In some embodiments, when detecting a conversion event includes detecting that the proximity of a user to a physical object in a real environment is less than a threshold distance, the first visibility value and the second visibility value are based on the proximity of the user to the physical object. In some embodiments, the image of the second environment includes a representation of a physical object (e.g., an image of table 202-a). In some embodiments, when detecting a conversion event includes detecting a triggering sound, the image of the second environment includes a representation of the source of the triggering sound.

[0098] In some embodiments, the second environment of the second type is a representation of a real environment (e.g., real environment 200). In some embodiments, the second environment of the second type is a virtual environment (e.g., virtual environment 300). In some embodiments, providing a combined view includes compositing an image of the first environment with an image of the second environment. In some embodiments, the compositing uses an alpha channel associated with the first environment and the second environment.

[0099] At box 610, in response to detecting the conversion event, a combined mix of the first environment audio and the second environment audio associated with the second environment is provided. At least one of the first environment audio or the second environment audio includes a plurality of audio objects, and the volume level of the plurality of audio objects is associated with at least one of the first visibility value or the second visibility value. In some embodiments, the first environment audio includes a plurality of audio objects, and providing a combined mix includes making the plurality of audio objects cross-fade with the second environment audio according to the corresponding volume level of the plurality of audio objects. In some embodiments, the volume level of one or more first audio objects in the plurality of audio objects is also associated with the prominence of one or more first audio objects (e.g., audio objects associated with the main character of the virtual environment 300, such as the virtual robot 302-a) in the first environment audio. In some embodiments, providing a combined view includes aligning the user perspective of the first environment with the user perspective of the second environment.

[0100] In some embodiments, after providing the combined view of the first environment and the second environment and the combined mix of the first environment audio and the second environment audio, the combined view of the first environment and the second environment and the combined mix of the first environment audio and the second environment audio are stopped. Then the view of the second environment and the second environment audio are provided (e.g., Figure 2 As shown, providing a view of the real environment 200 and audio from the real environment 200).

[0101] In some embodiments, a second transition event is detected (e.g., further rotation of the input mechanism, closer proximity to a physical object). In response to detecting the second transition event, providing a combined view of the first environment and the second environment and a combined mix of the first environment audio and the second environment audio is stopped. The view of the second environment and the second environment audio are then provided (e.g., Figure 2 As shown, providing a view of the real environment 200 and audio from the real environment 200).

[0102] In some embodiments, in response to detecting the second transition event, the first visibility value and the second visibility value and the combined mix of the first ambient audio and the second ambient audio are modified (e.g., the transition from the view of the virtual environment 300 to the view of the real environment 200 continues, such as FIG. 4B to FIG. 4E shown).

[0103] In some embodiments, in response to detecting the second transition event, providing the combined view of the first environment and the second environment and the combined mix of the first environment audio and the second environment audio is stopped. The view of the first environment and the first environment audio are then provided (e.g., the view returns to the view of the virtual environment 300, such as Figure 3 As shown, where the audio object is associated with the virtual environment 300).

[0104] Figure 7 700 is a flowchart showing an exemplary process 700 performed by an electronic device (e.g., device 100a) according to some embodiments. In some embodiments, the electronic device has one or more displays (e.g., display 120). In some embodiments, the display is a (at least partially) transparent display. In some embodiments, the electronic device is connected to and communicates with a display separate from the device. In some embodiments, the electronic device has one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116). In some embodiments, the electronic device is connected to and communicates with one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116) separate from the device. In some embodiments, the device is a head-mounted device. In some embodiments, the electronic device is separated from the head-mounted device, but is fixed on (or configured to be fixed to) the head-mounted device. In some embodiments, the electronic device includes one or more speakers (e.g., speaker 118) for outputting audio. In some embodiments, the electronic device is connected (or configured to be connected) to (e.g., via a wireless connection, via a wired connection) and communicates with (or is configured to communicate with) one or more speakers (e.g., speaker 118) for outputting audio.

[0105] At block 702, a virtual environment (e.g., virtual environment 300) is presented (e.g., displayed) using one or more displays. When the virtual environment is initially presented, the first visibility value (e.g., α=1.0, such as Figure 3 The environment is presented as shown.

[0106] At block 704, the electronic device is detected to be in contact with a physical object (e.g., physical object 202-a, such as Figure 2 shown).

[0107] At block 706 , in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance, the image of the physical object is isolated from other images of the real environment.

[0108] At block 708, a virtual environment is presented in which an isolated image of the physical object is inserted into the virtual environment at a location corresponding to the location of the physical object in the real environment (such as FIG. 5A to FIG. 5C200 ). The image of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object (e.g., as the device moves closer to the physical object 202-a in the real environment 200, the image of the physical object 202-a inserted in the virtual environment 300 becomes more visible). In some embodiments, presenting the virtual environment with the isolated image of the physical object includes compositing the image of the virtual environment with the isolated image of the physical object. In some embodiments, the compositing uses an alpha channel associated with the image of the virtual environment and the image of the physical object. In some embodiments, presenting the virtual environment with the isolated image of the physical object includes aligning a user's perspective of the physical object with a user's perspective of the virtual environment.

[0109] In some embodiments, in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance, the virtual environment is presented with a second visibility value associated with the proximity of the electronic device to the physical object (e.g., when the image of the physical object 202-a is inserted, the image of the virtual environment 300 becomes less visible). In some embodiments, in response to detecting that the proximity of the electronic device to the physical object is less than a second threshold distance, the first visibility value of the image of the physical object is modified (e.g., when the device moves closer to the physical object 202-a, the image of the object becomes more visible). In some embodiments, in response to detecting that the proximity of the electronic device to the physical object is less than a third threshold distance, the presentation of the virtual environment is stopped and a view of the real environment is provided (e.g., a view of the real environment 200 is provided, such as Figure 2 shown).

[0110] At block 710, in response to detecting that the proximity of the electronic device to the physical object is greater than a first threshold distance, presenting the virtual environment without an isolated image of the physical object (e.g., removing the image of the physical object 202-a and presenting the virtual environment 300, such as Figure 3 shown).

[0111] In some embodiments, process 700 also includes providing virtual environment audio associated with the virtual environment using one or more speakers. In some embodiments, in response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance, providing a combined mix of the virtual environment audio and the real environment audio, wherein the amount of the virtual environment audio in the combined mix is ​​associated with the proximity of the electronic device to the physical object.

[0112] In some embodiments, the virtual environment audio includes a plurality of audio objects, and providing the combined mix includes cross-fading the plurality of audio objects with the real environment audio. In some embodiments, an amount of cross-fading applied to one or more first audio objects in the plurality of audio objects is associated with a prominence of the one or more first audio objects in the virtual environment (e.g., audio objects associated with a primary character of the virtual environment 300, such as the virtual robot 302-a). In some embodiments, an amount of cross-fading applied to one or more second audio objects in the plurality of audio objects is associated with a value of an alpha channel of the image.

[0113] Figure 8 8 is a flowchart showing an exemplary process 800 performed by an electronic device (e.g., device 100a) according to some embodiments. In some embodiments, the electronic device has one or more displays (e.g., display 120). In some embodiments, the display is a (at least partially) transparent display. In some embodiments, the electronic device is connected to and communicates with a display separate from the device. In some embodiments, the electronic device has one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116). In some embodiments, the electronic device is connected to and communicates with one or more sensor devices (e.g., image sensor 108, orientation sensor 110, position sensor 116) separate from the device. In some embodiments, the device is a head-mounted device. In some embodiments, the electronic device is separated from the head-mounted device, but is fixed on (or configured to be fixed to) the head-mounted device. In some embodiments, the electronic device includes one or more speakers (e.g., speaker 118) for outputting audio. In some embodiments, the electronic device is connected (or configured to be connected) to (e.g., via a wireless connection, via a wired connection) and communicates with (or is configured to communicate with) one or more speakers (e.g., speaker 118) for outputting audio.

[0114] At block 802, a view of a first environment of a first type is provided (e.g., displayed). In some embodiments, the first environment of the first type is a real environment (e.g., real environment 200). In some embodiments, the first environment of the first type is a virtual environment (e.g., virtual environment 300).

[0115] At box 804, first ambient audio associated with the first environment is provided, wherein the first ambient audio has a first acoustic characteristic corresponding to the provided view of the first environment. In some embodiments, providing the first ambient audio includes modifying the first ambient audio to have a first acoustic characteristic corresponding to the provided view of the first environment. In some embodiments, the first ambient audio includes the user's voice. In some embodiments, the first ambient audio includes an audio object associated with a component of the first environment of the first type.

[0116] At block 806, a transition event is detected. In some embodiments, detecting a transition event comprises receiving a signal from an input device. In some embodiments, the input device comprises a rotary input mechanism (e.g., a knob). In some embodiments, detecting a transition event comprises detecting that a user's proximity to a physical object in the real environment (e.g., table 202-a) is less than a threshold distance. In some embodiments, detecting a transition event comprises detecting a trigger sound.

[0117] At block 808, in response to detecting the transition event, a view of a second environment of a second type is provided. In some embodiments, the second environment of the second type is a virtual environment (e.g., virtual environment 300). In some embodiments, the second environment of the second type is a real environment (e.g., real environment 200).

[0118] At block 810, the first environment audio is modified to have a second acoustic characteristic corresponding to the provided view of the second environment. In some embodiments, the acoustic characteristic includes one or more of reverberation, decay time, frequency response, time energy frequency, and energy time curve.

[0119] At block 812, the modified first ambient audio is provided. In some embodiments, the modified first ambient audio is provided in a combined mix with second ambient audio associated with the second environment. In some embodiments, the second ambient audio includes audio modified to have a second acoustic characteristic corresponding to the provided view of the second environment.

[0120] In some embodiments, a second transition event is detected. In response to detecting the second transition event, providing a view of the second environment of the second type is stopped, modifying the first environment audio to have the second acoustic characteristic is stopped, providing a view of the first environment of the first type, and providing the first environment audio, wherein the first environment audio has the first acoustic characteristic corresponding to the provided view of the first environment.

[0121] Executable instructions for performing features of processes 600, 700, and / or 800 described above are optionally included in a transient or non-transitory computer-readable storage medium (e.g., memory 106) or other computer program product configured to be executed by one or more processors (e.g., processor 102).

[0122] The various processes defined herein contemplate the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized to provide images and sounds of a real environment in a virtual environment. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. Users should understand and control the use of their personal information as described herein.

[0123] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive, user accessible, and recognized to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0124] However, users may limit the extent to which parties may access or otherwise obtain personal information. For example, settings or other preferences may be adjusted so that users may determine whether their personal information may be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, aspects of these features may be implemented without the use of such information. For example, if user preferences, account names, and / or location history are collected, that information may be obfuscated or otherwise generalized so that the information does not identify the respective user.

[0125] The foregoing descriptions of specific embodiments have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed, and it is to be understood that many modifications and variations are possible in light of the above teachings.

Claims

1. A method comprising: At an electronic device having one or more displays: presenting a virtual environment via the one or more displays; detecting physical objects located in a real environment via one or more sensors; presenting the virtual environment with the virtual object corresponding to the physical object based on determining that a set of one or more criteria for inserting the virtual object corresponding to the physical object into the virtual environment is satisfied; as well as Based on determining that the set of one or more criteria for inserting the virtual object corresponding to the physical object into the virtual environment is not satisfied, presenting the virtual environment without the virtual object corresponding to the physical object.

2. The method according to claim 1, wherein: The set of one or more criteria for inserting the virtual object corresponding to the physical object into the virtual environment includes a criterion that is satisfied when a proximity of the electronic device to the physical object is less than a threshold distance.

3. The method according to claim 2, wherein: When the proximity of the electronic device to the physical object is greater than a first threshold distance, the set of one or more criteria for inserting the virtual object corresponding to the physical object into the virtual environment is not satisfied.

4. The method according to any one of claims 1 to 3, wherein: The virtual object corresponding to the physical object is a generic object shape.

5. The method according to claim 4, wherein: The generic object shape is visually similar to the physical object.

6. The method according to any one of claims 1 to 3, wherein: Rendering the virtual environment with the virtual object corresponding to the physical object includes rendering the virtual object corresponding to the physical object at a location in the virtual environment that corresponds to a location of the physical object in the real environment.

7. The method according to any one of claims 1 to 3, wherein: Presenting the virtual environment with the virtual object corresponding to the physical object includes presenting the virtual object corresponding to the physical object with a visibility value associated with a proximity of the electronic device to the physical object.

8. The method according to claim 7, wherein: Presenting the virtual environment having the virtual objects corresponding to the physical objects comprises: presenting a first portion of the virtual environment at a first transparency associated with the proximity of the electronic device to the physical object, wherein presenting the first portion of the virtual environment at the first transparency comprises combining the first portion of the virtual environment with a first image of the real environment; and Presenting a second portion of the virtual environment with a second transparency different from the first transparency associated with the proximity of the electronic device to the physical object, wherein presenting the second portion of the virtual environment with the second transparency includes combining the second portion of the virtual environment with a second image of the real environment.

9. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more displays, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 8.

10. An electronic device comprising: one or more displays; one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 8.

11. An electronic device having one or more displays, comprising: Device for carrying out the method according to any one of claims 1 to 8.

12. A computer program product comprising one or more programs configured to be executed by one or more processors of an electronic device having one or more displays, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 8.

Images