Location synchronization of virtual and physical cameras

Abstract

The invention provides position synchronization of virtual and physical cameras. A device for position synchronization of a virtual camera and a physical camera may include a processor configured to determine a first position of the physical camera relative to another electronic device in a physical environment. The processor may be configured to initiate positioning the virtual camera at a second location within the computer-generated environment, wherein the second location relative to the representation of the person in the computer-generated environment coincides with the first location. The processor may be configured to receive image frames captured by a physical camera and virtual image frames generated by a virtual camera. The processor may be configured to generate a computer-generated real image frame including a synthesis of at least a portion of an image frame with at least a portion of a virtual image frame.

Description

[0001] This application is a divisional application of the invention patent application with the application date of September 9, 2020, application number 202080054608.6, and invention name "Positional Synchronization of Virtual and Physical Cameras".

[0002] Cross - Reference to Related Applications

[0003] This patent application claims the priority benefit of U.S. Provisional Patent Application No. 62 / 897,910, filed on September 9, 2019, with the title "Positional Synchronization of Virtual and Physical Cameras", the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0004] This specification generally relates to the recording of computer-generated reality, including the positional synchronization or alignment of virtual and physical cameras to generate a synthetic computer-generated reality recording. Background Art

[0005] Augmented reality technology aims to bridge the gap between computer-generated environments and physical environments by providing an augmented physical environment enhanced with electronic information. Thus, the electronic information appears to be part of the physical environment perceived by the user. In an example, augmented reality technology further provides a user interface to interact with the electronic information overlaid in the augmented physical environment. Brief Description of the Drawings

[0006] Some features of the subject technology are shown in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0007] Figure 1 An exemplary system architecture including various electronic devices that can implement the subject system is shown in accordance with one or more specific implementations.

[0008] Figure 2 An exemplary electronic device that can be used for the positional synchronization of virtual and physical cameras is shown in accordance with one or more specific implementations.

[0009] Figure 3 A flowchart of an example process for the automatic positional synchronization of a virtual camera relative to a physical camera is shown in accordance with one or more specific implementations.

[0010] Figure 4 A flowchart of an example process for providing guidance for the positional synchronization of a physical camera relative to a virtual camera is shown in accordance with one or more specific implementations.

[0011] Figure 5A flowchart showing an example process for providing a viewport into a computer-generated reality environment according to one or more specific implementations.

[0012] Figure 6 An example environment is shown according to one or more specific implementations, in which the positions of virtual and physical cameras can be synchronized for computer-generated reality recording.

[0013] Figure 7 An example environment is shown according to one or more specific implementations, in which the positions of virtual and physical cameras can be synchronized to provide a computer-generated reality viewport.

[0014] Figure 8 An exemplary electronic system is shown according to one or more specific implementations that can be used to implement various aspects of the present subject matter technology. Detailed Description

[0015] The detailed description shown below is intended as a description of various configurations of the present subject matter technology and is not intended to represent the only configuration in which the present subject matter technology can be practiced. The drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the present subject matter technology. However, the present subject matter technology is not limited to the specific details set forth herein, but may be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the present subject matter technology.

[0016] Computer-generated reality (CGR) systems enable physical and virtual environments to be combined to varying degrees to facilitate real-time interaction of users. Thus, as described herein, such CGR systems can include various possible combinations of physical and virtual environments, including augmented reality, which mainly includes physical elements and is closer to the physical environment compared to virtual environments (e.g., without physical elements). In this way, the physical environment can be connected to the virtual environment through the CGR system. A user immersed in a CGR environment can navigate through such an environment, and the CGR system can track the user's viewpoint to provide visualization based on how the user is located in the environment.

[0017] The physical environment refers to the physical world that people can sense and / or interact with without the help of an electronic system. Physical environments such as a physical park include physical items such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0018] Conversely, a computer-generated reality (CGR) environment is an environment that is a complete or partial simulation that a person senses and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements or representations thereof are tracked and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system can detect a person's body and / or head turning and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects in a CGR environment can be made in response to a representation of physical movement (e.g., a voice command).

[0019] A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of point audio sources in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects.

[0020] Examples of CGR include virtual reality and mixed reality.

[0021] A virtual reality (VR) environment is an environment that is a simulation designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with the virtual objects in a VR environment by way of a simulation of the person's presence within the computer-generated environment and / or by way of a simulation of a subgroup of the person's physical movements within the computer-generated environment.

[0022] In contrast to a VR environment that is designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment is an environment that is a simulation designed to incorporate sensory input or representations thereof from the physical environment in addition to including computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any condition between a fully physical environment as one end and a virtual reality environment as the other end, but excluding these two ends.

[0023] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems for presenting an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system can cause movement such that a virtual tree appears stationary relative to the physical ground.

[0024] An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on the physical environment or its representation. For example, an electronic system for presenting an AR environment can have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or translucent display such that the person perceives the virtual objects superimposed on a portion of the physical environment using the system. Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system combines the images or video with the virtual objects and presents the combination on the opaque display. The person indirectly views the physical environment using the system via the images or video of the physical environment and perceives the virtual objects superimposed on and / or behind a portion of the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as a "passthrough video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system can have a projection system that projects virtual objects into the physical environment, such as as a hologram or on a physical surface, such that the person perceives the virtual objects superimposed on the physical environment using the system.

[0025] An augmented reality environment is also a simulated environment in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing a passthrough video, the system can transform one or more sensor images to impose an alternative perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions can be a representative but not true version of the originally captured image. As yet another example, the representation of the physical environment can be transformed by graphically removing portions thereof or blurring portions thereof.

[0026] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but the face of a person is a realistic reproduction of an image taken of the physical person. As another example, a virtual object can adopt the shape or color of a physical item imaged by one or more imaging sensors. As yet another example, a virtual object can adopt a shadow that conforms to the positioning of the sun in the physical environment.

[0027] There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include mobile devices, tablet devices, projection-based systems, head-up displays (HUDs), head-mounted systems, vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headsets / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablet computers or tablet devices, and desktop / laptop computers. For example, a head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system can be configured to receive an external opaque display (e.g., a smart phone). The head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system can have a transparent or semi-transparent display instead of an opaque display. The transparent or semi-transparent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or semi-transparent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphical image onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, such as as a hologram or on a physical surface.

[0028] The CGR system enables physical and computer-generated environments to be combined in different degrees to facilitate real-time interaction of users. Thus, as described herein, such CGR systems can include various possible combinations of physical and computer-generated environments. In this way, the physical environment can be connected to the computer-generated environment through the CGR system. For example, a user immersed in a computer-generated environment via an electronic device can navigate through such an environment, and the system can track the user's viewpoint to provide visualization based on how the user is located in the environment. The user can be represented in the computer-generated environment, for example, by an avatar.

[0029] Virtual cameras can be positioned throughout the computer-generated environment to capture virtual images and / or videos of the user's avatar moving throughout the computer-generated environment, while physical cameras (e.g., image capture devices) can be positioned throughout the physical environment around the user to capture images and / or videos of the surrounding physical environment. The subject system facilitates the synchronization of the virtual cameras and the physical cameras with respect to the position of the user's avatar in the computer-generated environment and the user in the physical environment, such that the virtual images captured by the virtual cameras are position-aligned and / or perspective-aligned with the physical images captured by the physical cameras. In this way, position-aligned and / or perspective-aligned virtual images and physical images can be synthesized to generate computer-generated reality images and / or recordings.

[0030] For example, the subject system can facilitate the user to synchronize the position of the physical camera that captures images / videos of the user's movement in the physical environment with the position of the virtual camera that simultaneously captures virtual images / videos of the user's avatar's movement in the computer-generated environment. In this way, the user's images / videos can be segmented from the images / videos captured from the physical environment and can be synthesized with the virtual images / videos to generate computer-generated reality images / recordings, where the user's avatar is replaced by the user's images in the physical environment.

[0031] The subject system can also be used to enable another user's device (such as a mobile device or a tablet) to be used as a viewport into the CGR environment that the user is experiencing while using and / or wearing an electronic device. For example, the subject system can place a virtual camera in the CGR environment at a position synchronized with the position of the other user's device in the physical environment. Then, virtual objects can be segmented from the virtual images and synthesized with the physical images simultaneously captured by the physical camera of the other user's device to provide the other user with a viewport into the CGR environment that the user is experiencing.

[0032] Figure 1Exemplary system architecture 100 is shown that includes various electronic devices that can implement the systems of the present subject matter according to one or more specific implementations. However, not all of the depicted components may be used in all specific implementations, and one or more specific implementations may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components can be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0033] System architecture 100 includes electronic device 105, handheld electronic device 104, electronic device 110, electronic device 115, and server 120. For explanatory purposes, system architecture 100 is shown in Figure 1 as including electronic device 105, handheld electronic device 104, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0034] Electronic device 105 may be implemented, for example, as a tablet device, a handheld and / or mobile device, or as a head-mounted portable system (e.g., worn by user 101). Electronic device 105 includes a display system capable of presenting a visualization of a computer-generated reality environment to the user. Electronic device 105 may be powered by a battery and / or another power source. In one example, the display system of electronic device 105 provides a stereoscopic presentation of a computer-generated reality environment to the user, enabling a three-dimensional visual display of a particular scene rendering. In one or more specific implementations, instead of using electronic device 105 to access a computer-generated reality environment or in addition thereto, the user may use handheld electronic device 104, such as a tablet computer, a watch, a mobile device, etc.

[0035] The electronic device 105 may include one or more cameras, such as camera 150 (e.g., visible light camera, infrared camera, etc.). In addition, the electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, depth sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight sensors, etc.), GPS sensors, Wi-Fi sensors, near field communication sensors, radio frequency sensors, etc. In addition, the electronic device 105 may include hardware elements that can receive user input, such as hardware buttons or switches. User input detected by such sensors and / or hardware elements corresponds to various input modalities for initiating a coexistence session within an application, for example. Such input modalities may include, but are not limited to, face tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, respiration, temperature, electroencephalogram, smell), recognizing speech or audio (e.g., specific hot words), and activating buttons or switches, etc.

[0036] In one or more specific embodiments, the electronic device 105 may be communicatively coupled to a base device, such as electronic device 110 and / or electronic device 115. Generally speaking, compared with the electronic device 105, such base devices may include more computing resources and / or available power. In one example, the electronic device 105 may operate in various modes. For example, the electronic device 105 may operate in an independent mode independent of any base device. When the electronic device 105 operates in the independent mode, the number of input modalities may be constrained by the power and / or processing limitations of the electronic device 105 (such as the available battery power of the device). In response to the power limitation, the electronic device 105 may deactivate certain sensors within the device itself to conserve battery power and / or release processing resources.

[0037] The electronic device 105 may also operate in a wirelessly connected mode (e.g., connected to a base device via a wireless connection) to work in combination with a given base device. The electronic device 105 may also operate in a connected mode where the electronic device 105 is physically connected to the base device (e.g., via a cable or some other physical connector), and may utilize the power resources provided by the base device (e.g., in the case where the base device charges the electronic device 105 when physically connected).

[0038] When the electronic device 105 operates in a wireless connection mode or a connected mode, at least a part of processing user input and / or rendering a computer-generated reality environment can be offloaded to the base device, thereby reducing the processing burden on the electronic device 105. For example, in a specific implementation, the electronic device 105 works in combination with the electronic device 110 or the electronic device 115 to generate a computer-generated reality environment, which includes physical objects and / or virtual objects that enable different forms of interaction (such as visual, auditory, and / or physical or tactile interaction) between the user and the generated computer-generated reality environment in real time. In an example, the electronic device 105 provides a rendering of a scene corresponding to the computer-generated reality environment, and the scene can be perceived by the user and interacted with in real time, such as a host environment for a co-presence conversation with another user. Additionally, as part of presenting the rendered scene, the electronic device 105 can provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and the current system workload.

[0039] The electronic device 105 can also detect events that have occurred within the scene of the computer-generated reality environment. Examples of such events include detecting the presence of a specific person, entity, or object in the scene. In response to the detected event, the electronic device 105 can provide an annotation (such as in the form of metadata) in the computer-generated reality environment corresponding to the detected event.

[0040] The network 106 can communicatively (directly or indirectly) couple, for example, the electronic device 104, the electronic device 105, the electronic device 110, and / or the electronic device 115 to each other and / or to the server 120. In one or more specific implementations, the network 106 can be an interconnected network that may include the Internet or devices communicatively coupled to the Internet.

[0041] The electronic device 110 can include a touch screen and can be, for example, a smart phone including a touch screen, a portable computing device such as a laptop computer including a touch screen, an accessory device including a touch screen (such as a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (such as a watch, a wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device having a touchpad. In one or more specific implementations, the electronic device 110 may not include a touch screen but can support touch screen-like gestures, such as in a computer-generated reality environment. In one or more specific implementations, the electronic device 110 can include a touchpad. In Figure 1In this case, by way of example, the electronic device 110 is depicted as a mobile smart phone device having a touch screen. In one or more specific embodiments, the electronic device 110, the handheld electronic device 104, and / or the electronic device 105 may be and / or may include all or part of the electronic devices discussed hereinafter with respect to the electronic system (hereinafter with respect to Figure 8 discussed). In one or more specific embodiments, the electronic device 110 may be another device, such as an Internet Protocol (IP) camera, a tablet computer, or an accompanying device such as an electronic stylus, etc.

[0042] The electronic device 115 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, an accompanying device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a wristband, etc. In Figure 1 this case, by way of example, the electronic device 115 is depicted as a desktop computer. The electronic device 115 may be and / or may include all or part of the electronic system described hereinafter with respect to Figure 8 described.

[0043] The server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software, and includes specific hardware (e.g., a processor, a graphics processor, and other dedicated or custom processors) for rendering and generating content for a computer-generated reality environment such as graphics, images, videos, audio, and multimedia files. In one specific embodiment, the server 120 may be used as a cloud storage server that stores any of the foregoing computer-generated reality content generated by the above devices and / or the server 120.

[0044] In one or more specific embodiments further discussed hereinafter with respect to Figure 6 a user who enters a computer-generated reality environment using the electronic device 105 and / or using the electronic device 104 may wish to generate a record that combines an image of their body in the physical environment with a virtual image frame, such as a virtual video, generated from the computer-generated environment provided by the electronic device 105 and / or the computer-generated reality environment. However, in order to composite an image of the user's body onto the virtual image frame, the position of the physical camera (e.g., the electronic device 110) that captures an image of the user's body in the physical environment may need to be synchronized (e.g., aligned) with the position of the virtual camera that generates the virtual image frame from the computer-generated environment.

[0045] The subject system facilitates a user in achieving position synchronization of the physical camera and the virtual camera by automatically positioning the virtual camera in the computer-generated environment at a position that coincides with the position of the physical camera in the physical environment (hereinafter with respect to Figure 3Example processes for further discussion) and / or provide guidance to the user to position a physical camera in the physical environment (e.g., electronic device 110) at a location that coincides with the location of a virtual camera in the computer-generated environment (discussed further below with respect to Figure 4 Example processes for further discussion). The subject system can then synthesize at least a portion of an image frame captured by the physical camera with a virtual image frame generated by the virtual camera to generate a computer-generated reality image frame.

[0046] In one or more specific embodiments discussed further below with respect to Figure 7 While a user is using and / or wearing electronic device 105 to access a computer-generated reality environment, another user may wish to view the computer-generated reality environment that the user is experiencing. The subject system enables other users to use their electronic devices (such as electronic device 110) as a viewport into the computer-generated reality environment provided by electronic device 105 by positioning a virtual camera in the computer-generated reality environment at a location that coincides with the location of the physical camera of the other user's electronic device in the physical environment (relative to electronic device 105). Example processes for providing a viewport into a computer-generated reality environment experienced by another user are discussed further below with respect to Figure 5 Example processes for providing a viewport into a computer-generated reality environment experienced by another user.

[0047] In one or more specific embodiments, a user using and / or wearing an electronic device to experience a computer-generated reality environment may wish to generate a computer-generated reality image of themselves in the computer-generated reality environment (e.g., a computer-generated reality "selfie"), the computer-generated reality image including an image of their physical body from the physical environment synthesized with the computer-generated reality environment being experienced. However, in order to align the image captured from the physical environment with the virtual image generated from the computer-generated environment, the subject system positions a virtual camera in the computer-generated environment at a location that is synchronized and / or aligned with the location of the physical camera in the physical environment (relative to electronic device 105), such that the image frame from the physical camera can be synthesized with the virtual image frame from the virtual camera.

[0048] For purposes of explanation, the subject system is described herein with respect to synchronizing the position of one virtual camera to one physical camera. However, the subject system can be used to synchronize the positions of one or more virtual cameras to one or more physical cameras. In one or more specific embodiments, a user can utilize the subject system to position a virtual camera in the computer-generated environment without using any physical cameras.

[0049] Figure 2An exemplary electronic device 110 is shown that can be used for position synchronization of virtual and physical cameras according to one or more specific embodiments. However, not all of the depicted components may be used in all specific embodiments, and one or more specific embodiments may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components can be made without departing from the essence or scope of the claims listed herein. Additional components, different components, or fewer components may be provided. In one or more specific embodiments, one or more components of electronic device 105, electronic device 112, electronic device 115, and / or server 120.

[0050] Electronic device 110 may include a host processor 202, a memory 204, a wireless interface 206, an image capture device 208, and one or more position sensors 210. In one or more specific embodiments, electronic device 110 may utilize wireless interface 206 as a position sensor and may or may not include any additional position sensors 210.

[0051] Wireless interface 206 may include one or more antennas and one or more transceivers for transmitting / receiving wireless communications. In one or more specific embodiments, wireless interface 206 may be configured to perform wireless ranging operations with another device (such as electronic device 105). Wireless ranging operations may include, for example, ranging operations performed by exchanging ultra-wideband signals (such as 500Mhz signals) that provide millimeter and / or sub-millimeter positioning accuracy, such as based on time of arrival and / or angle of arrival determined from the exchanged signals.

[0052] Image capture device 208 may be and / or may include, for example, one or more image sensors. Image capture device 208 may further include one or more lighting devices, such as infrared devices, light-emitting diode devices, or generally any lighting device. In one or more specific embodiments, image capture device 208 may be partially and / or wholly referred to as a physical camera. In one or more specific embodiments, image capture device 208 may be used to determine the position of electronic device 110 relative to another device (such as electronic device 105). For example, one or more image sensors of image capture device 208 may be used to generate a depth map and / or otherwise determine the depth of another device (such as electronic device 105).

[0053] In one or more specific embodiments, image capture device 208 may also be used to segment an image of a user's body from an image of the physical environment. For example, one or more depth maps may be generated from images captured by multiple image sensors of image capture device 208, and one or more depth maps may be used to identify / segment objects in the image, such as the user.

[0054] One or more position sensors 210 may include one or more sensors that may provide information that can be used to determine the position of the electronic device 110, such as relative to another device (e.g., electronic device 105). For example, one or more position sensors may include one or more gyroscopes, accelerometers, magnetometers, ultrasonic transceivers, radar / lidar transceivers, or any sensor that can generally assist in determining the position of the electronic device 110 relative to another device (such as electronic device 105). In one or more specific implementations, one or more position sensors 210 may also be used to determine the orientation of the electronic device 110, e.g., along the x-axis, y-axis, and / or z-axis.

[0055] The host processor 202 may include appropriate logic components, circuits, and / or code that enable processing of data and / or control of the operation of the electronic device 110. In this regard, the host processor 202 may be enabled to provide control signals to various other components of the electronic device 110. The host processor 202 may also control data transfer between various parts of the electronic device 110. Additionally, the host processor 202 may enable the implementation of an operating system or otherwise execute code to manage the operation of the electronic device 110. In one or more specific implementations, the host processor 202 may facilitate determining the position of the electronic device 110 relative to another device (such as electronic device 105) using computer vision. For example, the host processor (and / or one or more remote processing units, such as at the server 120) may analyze one or more images captured by the image capture device 208 to determine the position of the electronic device 110 relative to another device (such as electronic device 105) shown in the one or more images.

[0056] The memory 204 may include appropriate logic components, circuits, and / or code that enable storage of various types of information, such as received data, generated data, code, and / or configuration information. The memory 204 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, and / or magnetic storage devices.

[0057] In one or more specific implementations, one or more of the host processor 202, the memory 204, the wireless interface 206, the image capture device 208, one or more position sensors 210, and / or one or more of their parts may be implemented in software (e.g., subroutines and code), in hardware (e.g., application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic device (PLD), controller, state machine, gated logic components, discrete hardware components, or any other suitable device), and / or a combination of both.

[0058] Figure 3A flowchart of an example process 300 for automatic position synchronization of a virtual camera relative to a physical camera according to one or more specific implementations is shown. For purposes of explanation, process 300 is mainly described herein with reference to Figure 1 electronic device 105. However, electronic device 105 is presented as an exemplary device, and one or more of the operations described herein may be performed by any suitable device. For example, process 300 may be performed by electronic device 110 or generally any electronic device. Further for purposes of illustration, the operations of process 300 are described herein as occurring sequentially or linearly. However, multiple operations of process 300 may occur in parallel. In addition, the operations of process 300 need not be performed in the order shown, and / or one or more of the operations of process 300 need not be performed and / or may be replaced by other operations.

[0059] The example process 300 may be initiated by a user experiencing a computer-generated environment and / or a computer-generated reality environment using electronic device 105, who wishes to generate a record that combines an image of their body in a physical environment with a virtual image frame, such as a virtual video, generated from the computer-generated environment and / or the computer-generated reality environment provided by electronic device 105. For example, the user may position another electronic device (such as electronic device 110) that includes a physical camera such that the field of view of the physical camera includes the user using and / or wearing electronic device 105 and / or utilizing electronic device 104 in a physical environment, as Figure 6 shown. The user may then select configuration options via, for example, electronic device 105 and / or electronic device 110 to initiate the example process 300.

[0060] Once the example process 300 is initiated, electronic device 105 determines the position (e.g., location and orientation) of the physical camera (such as electronic device 110) relative to electronic device 105 in the physical environment (302). For example, when the physical camera includes a wireless interface that supports ranging operations, electronic device 105 may initiate a ranging operation, such as an ultra-wideband ranging operation, via the physical camera. In one or more specific implementations, electronic device 105 may utilize one or more image sensors to determine the position of the physical camera, such as by using computer vision, depth maps, etc. In one or more specific implementations, electronic device 105 may receive, for example, from the physical camera the position of the physical camera relative to electronic device 105. For example, the physical camera may include additional position sensors that may be used to determine its position relative to electronic device 105.

[0061] The electronic device 105 may initiate positioning a virtual camera within a computer-generated reality environment provided by the electronic device 105 to coincide with the position of a physical camera in a physical environment relative to the electronic device 105 (304). The virtual camera may be positioned to have an orientation and position that coincide with the orientation and position of the physical camera relative to the electronic device 105 such that an image frame captured by the physical camera is aligned with a virtual image frame generated by the virtual camera. For example, the position of the virtual camera relative to a virtual representation (such as a user's avatar) corresponding to the electronic device 105 in the computer-generated environment may coincide with the position and orientation of the physical camera in the physical environment relative to the electronic device 105.

[0062] The electronic device 105 may initiate recording of the physical environment by the physical camera while initiating generation of a virtual recording of the computer-generated environment by the virtual camera (306). For example, the electronic device 105 may transmit a message to the physical camera indicating the time at which the physical camera should initiate recording. In one or more particular implementations, the recordings may not be initiated simultaneously; however, the virtual and / or physical cameras may be synchronized in time such that the images can be time-aligned later, e.g., based on timestamps.

[0063] The electronic device 105 may receive an image frame captured by the physical camera and a virtual image frame captured by the virtual camera (308). For example, the physical camera may stream the image frame to the electronic device 105 via a wired and / or wireless connection. The electronic device 105 may segment at least one of the image frame or the virtual image frame (310). For example, the electronic device 105 may segment an image of a user from the physical image frame, such as by utilizing computer vision / object recognition and / or by utilizing additional data (such as a depth map or other information that can be used to identify and segment the user from the image frame) corresponding to the image frame provided by the electronic device 110. In one or more particular implementations, the user may stand in front of a uniformly colored screen (such as a green screen), which facilitates segmenting the image of the user from the image frame.

[0064] In one or more particular implementations, the electronic device 110 may include one or more sensors, such as multiple image sensors, which may be used to generate a depth map or otherwise generate information that can be used to segment an image of a user from an image frame. The electronic device 110 may provide the segmented image of the user to the electronic device 105 such that the electronic device 105 can bypass segmentation and composite the segmented user image onto the virtual image.

[0065] The electronic device 105 generates a computer-generated reality frame (312) from at least a portion of an image frame (e.g., a segmented user image) and at least a portion of a virtual image frame (such as an entire virtual image frame other than a virtual representation of the user, e.g., the user's avatar). In one or more specific implementations, the electronic device 105 may superimpose an image of the user onto the user's avatar in the corresponding virtual image frame to generate a computer-generated reality image frame.

[0066] The electronic device 105 provides the computer-generated reality image frame, such as for display to the user, for local and / or remote storage and / or for streaming (314) to one or more other users. Operations 308 - 314 may be repeated for each image frame and corresponding virtual image frame to generate a computer-generated reality video stream. Thus, for example, the computer-generated reality image frame and / or video stream may be displayed by the electronic device 105 to the user such that the user can preview the computer-generated reality video stream and make any physical and / or virtual camera adjustments that may be needed and / or desired.

[0067] In one or more specific implementations, the electronic device 105 may continuously monitor the position of the physical camera relative to the electronic device 105 (302), and may continuously adjust the position of the virtual camera to coincide with changes in the position of the physical camera. Thus, if another user is holding the physical camera and the other user is moving, shaking, or otherwise not keeping the physical camera in a fixed position, the position of the virtual camera will also move and / or shake such that the entire computer-generated reality image frame will reflect the movement / shaking rather than only the physical image frame portion of the computer-generated reality image frame.

[0068] In one or more specific implementations, the example process 300 may be performed in whole or in part by the electronic device 110. For example, the electronic device 110 may transmit instructions to the electronic device 105 and may receive a stream of virtual image frames generated by a virtual camera positioned by the electronic device 105. In one or more specific implementations, at least a portion of the example process 300 may be performed and / or facilitated by the server 120. For example, the server may receive the image frame captured by the physical camera and the virtual image frame generated by the virtual image camera, may generate a computer-generated reality image frame, and may transmit the computer-generated reality image frame to the electronic device 110, the electronic device 105, and / or one or more additional electronic devices, such as the electronic devices of other users authorized to view the user's computer-generated reality video stream.

[0069] In one or more embodiments, the electronic device 105 and a physical camera (such as the electronic device 110) may be associated with and / or registered to the same user account. The electronic device 105 and / or the electronic device 110 may confirm that other devices are associated with the same user account before initiating the example process 300. Alternatively and / or additionally, the electronic device 105 and the electronic device 110 may be associated with different user accounts and may participate in an initial pairing / authorization operation before initiating the example process 300.

[0070] Figure 4 FIG. 4 is a flow diagram of an example process for providing guidance for position synchronization of a physical camera relative to a virtual camera in accordance with one or more embodiments. For purposes of explanation, the process 400 is described herein primarily with reference to Figure 1 the electronic device 105. However, the electronic device 105 is presented as an exemplary device, and one or more of the operations described herein may be performed by any suitable device. For example, the process 400 may be performed by the electronic device 110 or any electronic device generally. Further for purposes of illustration, the operations of the process 400 are described herein as occurring sequentially or linearly. However, multiple operations of the process 400 may occur in parallel. Additionally, the operations of the process 400 need not be performed in the order shown, and / or one or more of the operations of the process 400 need not be performed and / or may be replaced by other operations.

[0071] The example process 400 may be initiated by a user using and / or wearing the electronic device 105 who is experiencing a computer-generated environment and / or a computer-generated reality environment and who desires to generate a record that combines an image of their body in the physical environment with virtual image frames, such as a virtual video, generated from the computer-generated environment and / or the computer-generated reality environment provided by the electronic device 105. For example, the user may position a virtual camera at a specific location within the computer-generated environment having a desired angle or field of view. The process 400 may then be initiated, such as by selecting a user interface element, to facilitate the user in positioning a physical camera, such as the electronic device 110, in the physical environment at a position that coincides with the position of the virtual camera in the computer-generated environment, as Figure 6 shown.

[0072] Once the example process 400 is initiated, the electronic device 105 determines the position of the virtual camera relative to the position in the computer-generated environment that corresponds to the position of the electronic device 105 in the computer-generated environment provided by the electronic device 105 (402). The position in the computer-generated environment that corresponds to the position of the electronic device 105 may be, for example, the position of the user's avatar in the computer-generated environment.

[0073] Then, the electronic device 105 provides guidance (404) to the user for positioning the physical camera in the physical environment at a position that coincides with the position of the virtual camera in the computer-generated environment relative to the electronic device 105. For example, the electronic device 105 may provide an augmented reality display that indicates the position in the physical environment where the physical camera should be positioned.

[0074] In one or more specific implementations, the process 400 may be performed by a physical camera, such as the electronic device 110. For example, the physical camera may receive position information from the electronic device 105 and then may facilitate the user in positioning the physical camera at an appropriate position in the physical environment. In one or more specific implementations, the physical camera may be included in an autonomous mobile device, such as a drone or a robotic device, which may automatically move the physical camera to the position indicated by the electronic device 105.

[0075] Figure 5 A flowchart showing an example process for providing a viewport into a computer-generated reality environment according to one or more specific implementations is shown. For purposes of explanation, the process 500 is mainly described herein with reference to Figure 1 the electronic device 110. However, the electronic device 110 is presented as an exemplary device, and the operations described herein may be performed by any suitable device. For example, the process 500 may be performed by the electronic device 112 or generally any electronic device. Further for purposes of explanation, the operations of the process 500 are described herein as occurring sequentially or linearly. However, multiple operations of the process 500 may occur in parallel. In addition, the operations of the process 500 need not be performed in the order shown, and / or one or more operations of the process 500 need not be performed and / or may be replaced by other operations.

[0076] The example process 500 may be initiated when a user is experiencing a computer-generated reality environment via the electronic device 105 and another user wants to view the computer-generated reality environment. For example, as Figure 7 shown, a user using and / or wearing the electronic device 105 may be playing a virtual game, and the virtual game board is positioned on (and / or adjacent to) a physical object, such as a table, in the physical environment. Although other users in the room can see the physical table, they cannot see the virtual game that the user is playing on the physical table using and / or wearing the electronic device 105. Therefore, other users may request to receive a viewport into the computer-generated reality environment provided by the electronic device 105 via the electronic device 110.

[0077] In one or more embodiments, if the electronic device 110 is registered to the same user account as the electronic device 105 and / or associated with the same user account, other users may be automatically granted access to the viewport. However, if the electronic device 110 is registered to a different user account than the electronic device 105 and / or associated with the different user account, the electronic device 110 may transmit a request for authorization to access the viewport to the electronic device 105. For example, the electronic device 105 may present the request to the user using and / or wearing the electronic device 105, and the user may accept or reject the request.

[0078] When the electronic device 110 is granted access to the viewport into the computer-generated reality environment generated by the electronic device 105, the electronic device 110 determines the position of the physical camera on the electronic device 110 relative to the electronic device 105 in the physical environment (502). For example, the position may be determined by performing ultra-wideband ranging operations using the electronic device 105 and / or any other suitable positioning technique described herein.

[0079] The electronic device 110 may transmit its position to the electronic device 105, and / or the electronic device 105 may use a suitable positioning technique to determine the position of the electronic device 110 (504). The electronic device 105 may position the virtual camera in the computer-generated reality environment at a position that coincides with the position of the physical camera of the electronic device 110 in the physical environment.

[0080] Then, the electronic device 110 may receive at least a portion of the virtual image frame generated by the virtual camera from the electronic device 105 (506). For example, the virtual camera may capture only virtual objects in the computer-generated reality environment, and / or the electronic device 105 may segment the virtual objects from the virtual image and may provide only the virtual objects to the electronic device 110. The electronic device 110 also receives the image frame captured by the physical camera of the electronic device 110 (508) and composites at least a portion of the virtual image onto the physical image frame, such as by overlaying at least the portion of the virtual image onto the physical image frame, to generate a computer-generated reality image frame (510).

[0081] Then, the electronic device 110 displays a computer-generated reality image frame on the display of the electronic device 110 to provide a viewport (512) into the computer-generated reality environment provided by the electronic device 105. Operations 506-512 may be repeated, such as for each image frame, to display a computer-generated reality video stream that provides a live viewport into the computer-generated reality environment. In one or more particular implementations, the electronic device 105 may display an indication (such as a red icon) to a user using and / or wearing the electronic device 105 that indicates that a viewport into the computer-generated reality environment is being provided to another user, such as via the electronic device 110.

[0082] Figure 6 An example environment is shown in which the positions of virtual and physical cameras may be synchronized for computer-generated reality recording in accordance with one or more particular implementations. However, not all of the depicted components may be used in all particular implementations, and one or more particular implementations may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0083] In the example environment 600, a user has positioned an electronic device 110 that includes a physical camera such that the field of view of the physical camera encompasses the location of a user using and / or wearing the electronic device 104 (or electronic device 105). Then, the electronic device 104 may perform, for example, the process 300 described above to position a virtual camera in the computer-generated reality environment provided by the electronic device 104. Alternatively and / or additionally, the user may have pre-configured the position of the virtual camera in the computer-generated environment, and the process 400 may have been used to determine an appropriate location to place the electronic device 110.

[0084] Once the position of the physical camera of the electronic device 110 and the position of the virtual camera in the computer-generated environment provided by the electronic device 104 are synchronized, the image frames captured by the physical camera may be combined with the virtual image frames generated by the virtual camera to generate one or more computer-generated reality image frames.

[0085] In one or more particular implementations, physical and virtual recording may not occur simultaneously. For example, recording with the physical camera may occur first, and then virtual recording may be added later, or vice versa. In such cases, the positions of the virtual and physical cameras will still be synchronized to allow subsequent recording combination; however, the recordings may occur asynchronously.

[0086] Figure 7An example environment 700 is shown in accordance with one or more particular implementations, where the positions of virtual and physical cameras may be synchronized to provide a computer-generated reality viewport. However, not all of the depicted components may be used in all particular implementations, and one or more particular implementations may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0087] In the example environment 700, a user is experiencing a computer-generated reality environment using and / or wearing an electronic device 105, which includes virtual objects positioned on and / or adjacent to a physical table 704. For example, an electronic device 110 may execute a process 500 to provide another user with a viewport 702 into the computer-generated reality environment being experienced by the user using and / or wearing the electronic device 105. In one or more particular implementations, an image of the electronic device 105 may be removed from a computer-generated reality image frame providing the viewport, and instead, an image of the user's actual face and / or avatar may be substituted.

[0088] In one or more particular implementations, a user using and / or wearing the electronic device 105 may be experiencing a computer-generated reality environment and / or may be participating in a co-presence session with another user. The user may desire to use a physical camera on their electronic device 110 to generate a self-capture image in the computer-generated environment, which may colloquially be referred to as a "selfie".

[0089] Accordingly, the electronic device 105 and / or the electronic device 110 may execute a process 300 to place a virtual camera in the computer-generated environment at a position that coincides with the position of the physical camera of the electronic device 110 in the physical environment relative to the electronic device 105. In this way, a virtual image frame generated by the virtual camera may be combined with an image frame generated by the physical camera of the electronic device 110 to generate a mixed reality self-capture image or a mixed reality selfie.

[0090] In one or more particular implementations, an image of the electronic device 105 may be removed from the mixed reality image and substituted with an image of the face and / or avatar of the user using and / or wearing the electronic device 105. If the user is participating in a co-presence session with one or more other users and is capturing a mixed reality selfie of one or more other users, the electronic devices being used and / or worn by the other users may also be substituted with the faces and / or avatars of the other users in the computer-generated reality image.

[0091] As described above, one aspect of the present technology is the collection and use of data obtained from various sources. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, social network identifiers, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0092] The present disclosure recognizes that the use of such personal information data in the technology of the present invention can be used to benefit users. The present disclosure also contemplates uses of personal information data that are beneficial to users. For example, health and fitness data can be used to provide insights into a user's overall health condition, or can be used as positive feedback for individuals using the technology to pursue health goals.

[0093] The present disclosure contemplates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be readily accessible to users and should be updated as the collection and / or use of the data changes. Personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. In addition, such collection / sharing should occur after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and secure access to such personal information data and to ensure that others with access to personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adjusted to account for the particular types of personal information data being collected and / or accessed and to apply applicable laws and standards that include specific considerations of the jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Thus, different privacy practices should be maintained for different types of personal data in each country.

[0094] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the technology may be configured to allow a user to select to participate in ("opt-in") or opt out of the collection of personal information data at any time during or after registering for the service. In addition to providing opt-in and opt-out options, the present disclosure contemplates providing notice relating to access or use of personal information. For example, a user may be notified upon download of an application that their personal information data will be accessed, and then again just prior to the personal information data being accessed by the application.

[0095] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. The risk can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data among users), and / or other methods.

[0096] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the inventive technology will not fail to operate properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user based on non-personal information data or a small amount of personal information, such as the content requested by a device associated with the user, other non-personal information, or publicly available information.

[0097] Figure 8 An exemplary electronic system 800 is shown that can be used to implement aspects of the present subject matter in accordance with one or more particular implementations. The electronic system 800 can be Figure 1One or more of the electronic devices 105, 104, 110, 112, 115 and / or the server 120 shown and / or can be a part thereof. The electronic system 800 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 800 includes a bus 808, one or more processing units 812, a system memory 804 (and / or cache), a ROM 810, a permanent storage device 802, an input device interface 814, an output device interface 806, and one or more network interfaces 816, or subsets and variations thereof.

[0098] The bus 808 generally represents all system buses, peripheral buses, and chipset buses that communicatively connect many internal devices of the electronic system 800. In one or more specific implementations, the bus 808 communicatively connects one or more processing units 812 with the ROM 810, the system memory 804, and the permanent storage device 802. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to execute the processes disclosed by the present subject matter. In different specific implementations, the one or more processing units 812 can be a single processor or a multi-core processor.

[0099] The ROM 810 stores static data and instructions required by the one or more processing units 812 and other modules of the electronic system 800. On the other hand, the permanent storage device 802 can be a read-write memory device. The permanent storage device 802 can be a non-volatile memory unit that stores instructions and data even when the electronic system 800 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or an optical disk and its corresponding disk drive) can be used as the permanent storage device 802.

[0100] In one or more specific implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) can be used as the permanent storage device 802. Like the permanent storage device 802, the system memory 804 can be a read-write memory device. However, different from the permanent storage device 802, the system memory 804 can be a volatile read-write memory, such as a random access memory. The system memory 804 can store any instructions and data among the instructions and data that the one or more processing units 812 may need during operation. In one or more specific implementations, the processes disclosed by the present subject matter are stored in the system memory 804, the permanent storage device 802, and / or the ROM 810. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to execute the processes of one or more specific implementations.

[0101] The bus 808 is also connected to an input device interface 814 and an output device interface 806. The input device interface 814 enables a user to convey information to and select commands for the electronic system 800. Input devices that may be used with the input device interface 814 can include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 806 can enable, for example, the display of images generated by the electronic system 800. Output devices that may be used with the output device interface 806 can include, for example, a printer and a display device such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information. One or more particular implementations can include a device that acts as both an input device and an output device, such as a touch screen. In these particular implementations, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input received from the user can be in any form, including acoustic, voice, or tactile input.

[0102] Finally, as Figure 8 shown, the bus 808 also couples the electronic system 800 to one or more networks and / or to one or more network nodes via one or more network interfaces 816. In this way, the electronic system 800 can be part of a computer network such as a local area network (LAN), a wide area network ("WAN"), or an intranet, or can be part of a network of networks such as the Internet. Any or all components of the electronic system 800 can be used with the present subject matter disclosure.

[0103] The foregoing functions can be implemented in computer software, firmware, or hardware. The technology can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as a mobile device. The processes and logical flows can be performed by one or more programmable processors and one or more programmable logic circuits. General and special purpose computing devices and storage devices can be interconnected by a communication network.

[0104] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact discs read-only (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), digital versatile discs read-only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state disk drives, read-only and recordable discs, ultra density optical discs, any other optical or magnetic medium, and floppy disks. The computer-readable medium can store a computer program that can be executed by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files that include higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0105] While the foregoing discussion has mainly related to microprocessors or multi-core processors that execute software, some specific implementations are performed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0106] As used in this specification and in any claims of this patent application, the terms "computer", "server", "processor", and "memory" all refer to electronic or other technological devices. These terms exclude a person or a group of persons. For the purposes of this specification, the term display or displaying means displaying on an electronic device. As used in this specification and in any claims of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, touchable objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0107] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user and a keyboard and a pointing device by which the user can provide input to the computer, where the display device is, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the pointing device is, for example, a mouse or a trackball. Other kinds of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input received from the user can be in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from the devices used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on a user client device.

[0108] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), the Internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network).

[0109] The computing system can include clients and servers. The clients and servers are generally remote from each other and can interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, the server transmits data (e.g., HTML pages) to the client device (e.g., for displaying data to a user interacting with the client device and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., results of user interaction) can be received at the server.

[0110] Those skilled in the art will recognize that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability of hardware and software, the various illustrative blocks, modules, elements, components, methods, and algorithms have been described generally in terms of functionality above. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality may be implemented differently for each particular application. The various components and blocks may be arranged differently (e.g., in a different order, or partitioned in a different manner) without departing from the scope of the claimed subject matter.

[0111] It should be understood that the specific order or hierarchical structure of the steps in the processes disclosed herein are examples of exemplary methods. Based on design preferences, it should be understood that the specific order or hierarchical structure of the steps in a process may be rearranged. Some of the steps in the process may be performed simultaneously. The appended method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchical structure presented.

[0112] The foregoing description has been provided to enable a person skilled in the art to practice the various aspects described herein. The foregoing description provides examples of the claimed subject matter, and the claimed subject matter is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims, where the singular forms of elements are not intended to mean "only one" but rather "one or more" unless specifically stated otherwise. Unless specifically stated otherwise, the term "some" means one or more. Pronouns in the masculine gender (e.g., his) include the feminine and neuter genders (e.g., her and its), and vice versa. Headings and subheadings (if any) are used only for convenience and do not limit the claimed invention described herein.

[0113] As used herein, the term website may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, etc. Thus, the term website may be used interchangeably with the terms web page and server. The predicate words "configured to", "operable to", and "programmed to" do not imply any particular tangible or intangible modification to a subject matter but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0114] As used herein, the term "automatically" may include performance by a computer or machine without user intervention; e.g., by instructions responsive to a predicate action of a computer or machine or other initiating mechanism. The word "example" is used herein to mean "serving as an example or illustration". Any aspect or design described herein as an "example" is not necessarily to be understood as preferred or advantageous over other aspects or designs.

[0115] Phrases such as "aspect" do not mean that such aspect is essential to the claimed technology or that such aspect applies to all configurations of the claimed technology. Disclosure related to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as "aspect" may refer to one or more aspects, and vice versa. Phrases such as "embodiment" do not mean that such embodiment is essential to the claimed technology or that such embodiment applies to all configurations of the claimed technology. Disclosure related to an embodiment may apply to all embodiments, or one or more embodiments. An embodiment may provide one or more examples. Phrases such as "embodiment" may refer to one or more embodiments, and vice versa. Phrases such as "configuration" do not mean that such configuration is essential to the claimed technology or that such configuration applies to all configurations of the claimed technology. Disclosure related to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

Claims

1. An electronic device, comprising: an image capture device; a memory; and at least one processor configured to: determine a first position of the electronic device relative to another electronic device in a physical environment; transmit the first position of the electronic device to the other electronic device; receive, from the other electronic device, at least a portion of a virtual object of a computer-generated reality environment included in at least one virtual image frame, the at least a portion of the at least one virtual image frame having been captured by a virtual camera placed at a second position within the computer-generated reality environment; receive, from the image capture device of the electronic device, at least one image frame captured from the physical environment; generate a computer-generated reality image frame based at least in part on the virtual object of the at least one virtual image frame received from the other electronic device and the at least one image frame; and display, by the electronic device, the computer-generated reality image frame.

2. The electronic device according to claim 1, wherein the second position of the virtual camera in the computer-generated reality environment relative to a third position associated with the other electronic device coincides with the first position of the image capture device in the physical environment relative to the other electronic device.

3. The electronic device according to claim 1, wherein the electronic device is associated with a first user account and the other electronic device is associated with a second user account different from the first user account.

4. The electronic device according to claim 1, wherein the at least one processor is further configured to: detect a change in the first position of the electronic device relative to the other electronic device in the physical environment; transmit an indication of the change in the first position to the other electronic device; and receive, from the other electronic device, another at least one virtual image frame including the virtual object of the computer-generated reality environment, the another at least one virtual image frame having been captured by the virtual camera after being repositioned to a third position reflecting the change in the first position of the electronic device.

5. A method of generating a synthetic computer-generated reality record, comprising: determining, by an electronic device including an image capture device, a first position of the electronic device relative to another electronic device in a physical environment; transmitting the first position of the electronic device to the other electronic device; receiving, by the electronic device, from the other electronic device, at least a portion of a virtual object of a computer-generated reality environment included in at least one virtual image frame, the at least a portion of the at least one virtual image frame having been captured by a virtual camera placed at a second position within the computer-generated reality environment; receiving, by the electronic device, from the image capture device of the electronic device, at least one image frame captured from the physical environment; The electronic device generates a computer-generated reality image frame at least in part based on the virtual object and the at least one image frame of the at least one virtual image frame received from the other electronic device; and The electronic device displays the computer-generated reality image frame.

6. The method according to claim 5, wherein the second position of the virtual camera relative to a third position associated with the other electronic device in the computer-generated reality environment coincides with the first position of the image capture device relative to the other electronic device in the physical environment.

7. The method according to claim 5, wherein the electronic device is associated with a first user account, and the other electronic device is associated with a second user account different from the first user account.

8. The method according to claim 5, further comprising: detecting a change in the first position of the electronic device relative to the other electronic device in the physical environment; transmitting an indication of the change in the first position to the other electronic device; and receiving from the other electronic device another at least one virtual image frame including the virtual object of the computer-generated reality environment, the another at least one virtual image frame having been captured by the virtual camera after being repositioned to a third position reflecting the change in the first position of the electronic device.

9. A non-transitory machine-readable medium, the non-transitory machine-readable medium including code that, when executed by one or more processors, causes the one or more processors to perform operations, the code including: code for an electronic device including an image capture device to determine a first position of the electronic device relative to another electronic device in a physical environment; code for transmitting the first position of the electronic device to the other electronic device; code for the electronic device to receive from the other electronic device at least a portion of at least one virtual image frame including a virtual object of a computer-generated reality environment, the at least a portion of the at least one virtual image frame having been captured by a virtual camera placed at a second position within the computer-generated reality environment; code for the electronic device to receive from the image capture device of the electronic device at least one image frame captured from the physical environment; code for the electronic device to generate a computer-generated reality image frame at least in part based on the virtual object of the at least one virtual image frame received from the other electronic device and the at least one image frame; and code for the electronic device to display the computer-generated reality image frame.

10. The machine-readable medium according to claim 9, wherein the second position of the virtual camera relative to a third position associated with the other electronic device in the computer-generated reality environment coincides with the first position of the image capture device relative to the other electronic device in the physical environment.

11. The machine-readable medium according to claim 9, wherein the electronic device is associated with a first user account, and the other electronic device is associated with a second user account different from the first user account.

12. The machine-readable medium according to claim 9, wherein the code further comprises: code for detecting a change in the first position of the electronic device relative to the other electronic device in the physical environment; code for transmitting an indication of the change in the first position to the other electronic device; and code for receiving from the other electronic device at least one additional virtual image frame of the virtual object of the computer-generated environment, the at least one additional virtual image frame having been captured by the virtual camera after being repositioned to a third position reflecting the change in the first position of the electronic device.

13. An electronic device, comprising: a memory; and at least one processor configured to: determine a first position of a first electronic device relative to a second electronic device in a physical environment; initiate positioning a virtual camera at a second position within a computer-generated environment generated by the second electronic device, wherein the second position of the virtual camera in the computer-generated environment coincides with the first position of the first electronic device in the physical environment; receive at least one image frame and at least one virtual image frame captured by the first electronic device; and generate a computer-generated reality image by the first electronic device or the second electronic device, the computer-generated reality image comprising at least a portion of the at least one image frame captured by the first electronic device and at least a portion of the at least one virtual image frame.

14. The electronic device according to claim 13, wherein the first electronic device comprises a physical camera.

15. The electronic device according to claim 14, wherein the second position of the virtual camera relative to a representation of a person in the computer-generated environment coincides with the first position of the physical camera relative to the second electronic device in the physical environment, the person being associated with the second electronic device.

16. The electronic device according to claim 14, wherein a first field of view of the computer-generated environment generated by the second electronic device is different from a second field of view of the virtual camera in the computer-generated environment.

17. The electronic device according to claim 13, wherein the at least one processor is further configured to: detect a change in the first position of the first electronic device; and initiate a change in the second position of the virtual camera to coincide with the changed first position of the first electronic device.

18. A method of generating a synthetic computer-generated reality recording, comprising: determining a first position of a first electronic device relative to a second electronic device in a physical environment; Initiate positioning a virtual camera at a second position within a computer-generated environment generated by the second electronic device, wherein the second position of the virtual camera within the computer-generated environment coincides with the first position of the first electronic device within the physical environment; Receive at least one image frame and at least one virtual image frame captured by the first electronic device; And Generate a computer-generated reality image by the first electronic device or the second electronic device, the computer-generated reality image including at least a portion of the at least one image frame captured by the first electronic device and at least a portion of the at least one virtual image frame.

19. The method according to claim 18, wherein the first electronic device includes a physical camera.

20. The method according to claim 19, wherein the second position of the virtual camera within the computer-generated environment relative to a representation of a person coincides with the first position of the physical camera within the physical environment relative to the second electronic device, the person being associated with the second electronic device.

21. The method according to claim 19, wherein a first field of view of the computer-generated environment generated by the second electronic device is different from a second field of view of the virtual camera within the computer-generated environment.

22. The method according to claim 18, further comprising: Detecting a change in the first position of the first electronic device; And Initiating a change in the second position of the virtual camera to coincide with the changed first position of the first electronic device.

23. A non-transitory machine-readable medium, the non-transitory machine-readable medium including code that, when executed by one or more processors, causes the one or more processors to perform operations, the code including: Code for determining a first position of a first electronic device within a physical environment relative to a second electronic device; Code for initiating positioning a virtual camera at a second position within a computer-generated environment generated by the second electronic device, wherein the second position of the virtual camera within the computer-generated environment coincides with the first position of the first electronic device within the physical environment; Code for receiving at least one image frame and at least one virtual image frame captured by the first electronic device; And Code for generating a computer-generated reality image by the first electronic device or the second electronic device, the computer-generated reality image including at least a portion of the at least one image frame captured by the first electronic device and at least a portion of the at least one virtual image frame.

24. The machine-readable medium according to claim 23, wherein the first electronic device includes a physical camera.

25. The machine-readable medium according to claim 24, wherein the second position of the virtual camera within the computer-generated environment relative to a representation of a person coincides with the first position of the physical camera within the physical environment relative to the second electronic device, the person being associated with the second electronic device.

26. The machine-readable medium according to claim 24, wherein a first field of view of the computer-generated environment generated by the second electronic device is different from a second field of view of the virtual camera in the computer-generated environment.

27. The machine-readable medium according to claim 23, wherein the code further comprises: code for detecting a change in the first position of the first electronic device; and code for initiating a change in the second position of the virtual camera to coincide with the changed first position of the first electronic device.

Images