Synchronize virtual and physical camera positions

Abstract

A device for synchronizing the position of a virtual camera and a physical camera may include a processor configured to determine a first position of a physical camera relative to another electronic device in a physical environment. The processor may be configured to initiate positioning of the virtual camera at a second position within a computer-generated environment, wherein the second position relative to a representation of a person in the computer-generated environment coincides with the first position. The processor may be configured to receive an image frame captured by the physical camera and a virtual image frame generated by the virtual camera. The processor may be configured to generate a computer-generated reality image frame comprising a composite of at least a portion of the image frame and at least a portion of the virtual image frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 897,910, filed on September 9, 2019, and entitled “Positional Synchronization of Virtual and Physical Cameras,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0003] The present specification generally relates to computer-generated reality recording, including position synchronization or position alignment of virtual and physical cameras to generate a synthetic computer-generated reality recording. Background Art

[0004] Augmented reality technology aims to bridge the gap between computer-generated environments and physical environments by providing an augmented physical environment that is enhanced with electronic information. Thus, the electronic information appears to be part of the physical environment perceived by the user. In an example, augmented reality technology further provides a user interface to interact with the electronic information overlaid on the augmented physical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0006] Figure 1 An exemplary system architecture including various electronic devices that may implement the subject system is shown according to one or more implementations.

[0007] Figure 2 Example electronic devices that can be used for position synchronization of virtual and physical cameras according to one or more implementations are shown.

[0008] Figure 3 A flow chart illustrating an example process for automatic position synchronization of a virtual camera relative to a physical camera in accordance with one or more implementations.

[0009] Figure 4 A flow chart of an example process for providing guidance for position synchronization of a physical camera relative to a virtual camera in accordance with one or more implementations is shown.

[0010] Figure 5 A flowchart of an example process for providing a viewport into a computer-generated reality environment is shown in accordance with one or more implementations.

[0011] Figure 6An example environment is shown in which position synchronization of virtual and physical cameras can be achieved for computer-generated reality recording according to one or more implementations.

[0012] Figure 7 An example environment is shown in which synchronization of virtual and physical camera positions can be achieved to provide a computer-generated reality viewport in accordance with one or more implementations.

[0013] Figure 8 Exemplary electronic systems are shown that can be used to implement various aspects of the subject technology in accordance with one or more implementations. DETAILED DESCRIPTION

[0014] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0015] Computer generated reality (CGR) systems enable physical and virtual environments to be combined to varying degrees to facilitate real-time interaction by users. Thus, as described herein, such CGR systems may include various possible combinations of physical and virtual environments, including augmented reality that primarily includes physical elements and is closer to the physical environment than the virtual environment (e.g., without physical elements). In this way, the physical environment may be connected to the virtual environment through the CGR system. A user immersed in a CGR environment may navigate through such an environment, and the CGR system may track the user's viewpoint to provide visualization based on how the user is located in the environment.

[0016] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0017] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect body and / or head turns of a person, and in response, adjust graphical content and sound fields presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0018] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0019] Examples of CGR include virtual reality and mixed reality.

[0020] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0021] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0022] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0023] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a portion of the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment, and perceive virtual objects superimposed on a portion of the physical environment and / or behind a portion of the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video", meaning that the system uses one or more image sensors to capture images of the physical environment, and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0024] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0025] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0026] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include mobile devices, tablet devices, projection-based systems, head-up displays (HUDs), head-mounted systems, vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets or tablet devices, and desktop / laptop computers. For example, a head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0027] CGR systems enable physical and computer-generated environments to be combined to varying degrees to facilitate real-time interaction with users. Thus, as described herein, such CGR systems may include various possible combinations of physical and computer-generated environments. In this way, a physical environment may be connected to a computer-generated environment through a CGR system. For example, a user immersed in a computer-generated environment via an electronic device may navigate through such an environment, and the system may track the user's viewpoint to provide visualization based on how the user is located in the environment. The user may be represented in the computer-generated environment by, for example, an avatar.

[0028] A virtual camera may be positioned throughout the computer-generated environment to capture virtual images and / or video of the movement of the user's avatar throughout the computer-generated environment, while a physical camera (e.g., an image capture device) may be positioned throughout the physical environment surrounding the user to capture images and / or video of the surrounding physical environment. The subject system facilitates synchronization of the virtual camera and the physical camera relative to the position of the user's avatar in the computer-generated environment and the user in the physical environment, such that virtual images captured by the virtual camera are positionally aligned and / or perspectively aligned with physical images captured by the physical camera. In this manner, positionally aligned and / or perspectively aligned virtual and physical images may be synthesized to generate a computer-generated image and / or recording of reality.

[0029] For example, the subject system may facilitate synchronizing the position of a physical camera that captures images / video of the user's movement in a physical environment with the position of a virtual camera that simultaneously captures virtual images / video of the user's avatar's movement in a computer-generated environment. In this way, the user's images / videos may be segmented from images / videos captured from the physical environment and may be composited with the virtual images / videos to generate a computer-generated image / recording of reality in which the user's avatar is replaced by an image of the user in the physical environment.

[0030] The subject system may also be used to enable another user's device (such as a mobile device or tablet) to be used as a viewport into the CGR environment that the user is experiencing using and / or wearing the electronic device. For example, the subject system may place a virtual camera in the CGR environment at a location that is synchronized with the location of the other user's device in the physical environment. Virtual objects may then be segmented from the virtual image and composited with a physical image being simultaneously captured by the physical camera of the other user's device to provide the other user with a viewport into the CGR environment being experienced by the user.

[0031] Figure 1An exemplary system architecture 100 including various electronic devices that may implement the subject system according to one or more implementations is shown. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0032] The system architecture 100 includes an electronic device 105, a handheld electronic device 104, an electronic device 110, an electronic device 115, and a server 120. For the purpose of explanation, the system architecture 100 is shown in FIG. Figure 1 1. System architecture 100 is shown as including electronic device 105, handheld electronic device 104, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0033] The electronic device 105 can be implemented, for example, as a tablet device, a handheld and / or mobile device, or as a head-mounted portable system (e.g., worn by the user 101). The electronic device 105 includes a display system capable of presenting a visualization of a computer-generated reality environment to the user. The electronic device 105 can be powered by a battery and / or another power source. In one example, the display system of the electronic device 105 provides a stereoscopic presentation of the computer-generated reality environment to the user, so that a three-dimensional visual display of a particular scene rendering can be provided. In one or more specific implementations, instead of or in addition to using the electronic device 105 to access the computer-generated reality environment, the user can use a handheld electronic device 104, such as a tablet computer, a watch, a mobile device, etc.

[0034] The electronic device 105 may include one or more cameras, such as a camera 150 (e.g., a visible light camera, an infrared camera, etc.). In addition, the electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, depth sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight sensors, etc.), GPS sensors, Wi-Fi sensors, near-field communication sensors, radio frequency sensors, etc. In addition, the electronic device 105 may include hardware elements that may receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities, such as for initiating a coexistence session within an application. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc.

[0035] In one or more specific implementations, the electronic device 105 may be communicatively coupled to a base device, such as the electronic device 110 and / or the electronic device 115. Generally speaking, such base devices may include more computing resources and / or available power than the electronic device 105. In one example, the electronic device 105 may operate in various modes. For example, the electronic device 105 may operate in a standalone mode independent of any base device. When the electronic device 105 operates in a standalone mode, the number of input modalities may be constrained by the power and / or processing limitations of the electronic device 105 (such as the available battery power of the device). In response to power limitations, the electronic device 105 may deactivate certain sensors within the device itself to preserve battery power and / or free up processing resources.

[0036] The electronic device 105 may also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection) to work in conjunction with a given base device. The electronic device 105 may also operate in a connection mode in which the electronic device 105 is physically connected to the base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device charges the electronic device 105 when physically connected).

[0037] When the electronic device 105 is operated in a wireless connection mode or a connection mode, at least a portion of processing user input and / or rendering a computer-generated reality environment can be offloaded to the base device, thereby reducing the processing burden on the electronic device 105. For example, in one specific implementation, the electronic device 105 works in conjunction with the electronic device 110 or the electronic device 115 to generate a computer-generated reality environment, which includes physical objects and / or virtual objects that implement different forms of interaction (e.g., visual, auditory and / or physical or tactile interaction) between the user and the generated computer-generated reality environment in real time. In one example, the electronic device 105 provides a rendering of a scene corresponding to a computer-generated reality environment that can be perceived by the user and interacted with in real time, such as a host environment for a coexistence conversation with another user. In addition, as part of presenting the rendered scene, the electronic device 105 may provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and current system workload.

[0038] The electronic device 105 may also detect an event that has occurred within a scene of the computer-generated reality environment. Examples of such events include detecting the presence of a particular person, entity, or object in the scene. In response to the detected event, the electronic device 105 may provide an annotation (e.g., in the form of metadata) in the computer-generated reality environment corresponding to the detected event.

[0039] The network 106 may communicatively couple (directly or indirectly) electronic devices 104, 105, 110, and / or 115 to each other and / or to a server 120, for example. In one or more implementations, the network 106 may be an interconnected network of devices that may include the Internet or be communicatively coupled to the Internet.

[0040] The electronic device 110 may include a touch screen and may be, for example, a smart phone including a touch screen, a portable computing device such as a laptop computer including a touch screen, a companion device including a touch screen (e.g., a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (such as a watch, wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device with a touchpad. In one or more specific implementations, the electronic device 110 may not include a touch screen, but may support touch screen-like gestures, such as in a computer-generated reality environment. In one or more specific implementations, the electronic device 110 may include a touchpad. In Figure 1In the embodiment, by way of example, the electronic device 110 is depicted as a mobile smart phone device having a touch screen. In one or more specific implementations, the electronic device 110, the handheld electronic device 104 and / or the electronic device 105 can be and / or can include the electronic system described below (hereinafter with respect to the electronic system). Figure 8 In one or more specific implementations, the electronic device 110 may be another device, such as an Internet Protocol (IP) camera, a tablet computer, or a companion device such as an electronic stylus.

[0041] The electronic device 115 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a companion device (e.g., a digital camera, a headset), a tablet device, a wearable device such as a watch, a wristband, etc. Figure 1 In the embodiment, the electronic device 115 is depicted as a desktop computer by way of example. The electronic device 115 may be and / or may include the following with respect to Figure 8 All or part of the electronic system described.

[0042] The server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors and other dedicated or custom processors) for rendering and generating content of a computer-generated reality environment, such as graphics, images, video, audio, and multimedia files. In one specific implementation, the server 120 may be used as a cloud storage server that stores any of the aforementioned computer-generated reality content generated by the aforementioned devices and / or server 120.

[0043] In the following about Figure 6 In one or more specific implementations discussed further, a user entering a computer-generated reality environment using electronic device 105 and / or using electronic device 104 may wish to generate a recording that merges an image of their body in the physical environment with a virtual image frame, such as a virtual video, generated from the computer-generated environment and / or the computer-generated reality environment provided by electronic device 105. However, in order to composite the image of the user's body onto the virtual image frame, the position of a physical camera (e.g., electronic device 110) that captures the image of the user's body in the physical environment may need to be synchronized (e.g., aligned) with the position of a virtual camera that generates the virtual image frame from the computer-generated environment.

[0044] The subject system facilitates the user to synchronize the positions of the physical camera and the virtual camera by automatically positioning the virtual camera in the computer-generated environment at a position that coincides with the position of the physical camera in the physical environment (hereinafter referred to as Figure 3) and / or providing guidance to the user to position a physical camera (e.g., electronic device 110) in the physical environment at a position that coincides with the position of a virtual camera in the computer-generated environment (described below with respect to example processes discussed further in Figure 4 ). The subject system may then synthesize at least a portion of the image frame captured by the physical camera with the virtual image frame generated by the virtual camera to generate a computer-generated reality image frame.

[0045] In the following about Figure 7 In one or more embodiments discussed further, when a user is using and / or wearing electronic device 105 to access a computer-generated reality environment, another user may wish to view the computer-generated reality environment that the user is experiencing. The subject system enables other users to use their electronic devices (such as electronic device 110) as viewports into the computer-generated reality environment provided by electronic device 105 by positioning a virtual camera in the computer-generated reality environment at a position that coincides with the position of the physical camera of the other user's electronic device in the physical environment (relative to electronic device 105). Figure 5 Further discussion provides an example process for entering a viewport into a computer-generated reality environment experienced by another user.

[0046] In one or more embodiments, a user using and / or wearing an electronic device to experience a computer-generated reality environment may wish to generate a computer-generated reality image of themselves in the computer-generated reality environment (e.g., a computer-generated reality "selfie"), wherein the computer-generated reality image includes an image of their physical body from the physical environment synthesized with the computer-generated reality environment being experienced. However, in order to align the image captured from the physical environment with the virtual image generated from the computer-generated environment, the subject system positions a virtual camera in the computer-generated environment at a position that is synchronized and / or aligned with the position of the physical camera in the physical environment (relative to the electronic device 105) so that image frames from the physical camera can be synthesized with virtual image frames from the virtual camera.

[0047] For purposes of explanation, the subject system is described herein with respect to synchronizing the position of a virtual camera to a physical camera. However, the subject system can be used to synchronize the position of one or more virtual cameras to one or more physical cameras. In one or more specific implementations, a user can utilize the subject system to position a virtual camera in a computer-generated environment without using any physical camera.

[0048] Figure 2An exemplary electronic device 110 that can be used for position synchronization of virtual and physical cameras according to one or more specific implementations is shown. However, not all of the components depicted can be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided. In one or more specific implementations, one or more components of the electronic device 105, the electronic device 112, the electronic device 115, and / or the server 120.

[0049] The electronic device 110 may include a host processor 202, a memory 204, a wireless interface 206, an image capture device 208, and one or more position sensors 210. In one or more implementations, the electronic device 110 may utilize the wireless interface 206 as a position sensor and may or may not include any additional position sensors 210.

[0050] The wireless interface 206 may include one or more antennas and one or more transceivers for transmitting / receiving wireless communications. In one or more specific implementations, the wireless interface 206 may be configured to perform wireless ranging operations with another device (such as the electronic device 105). The wireless ranging operations may include, for example, ranging operations performed by exchanging ultra-wideband signals (e.g., 500Mhz signals) that provide millimeter and / or sub-millimeter positioning accuracy, such as based on arrival times and / or arrival angles determined from the exchanged signals.

[0051] The image capture device 208 may be and / or may include, for example, one or more image sensors. The image capture device 208 may further include one or more lighting devices, such as an infrared device, a light emitting diode device, or generally any lighting device. In one or more specific implementations, the image capture device 208 may be referred to as a physical camera in part and / or in its entirety. In one or more specific implementations, the image capture device 208 may be used to determine the position of the electronic device 110 relative to another device, such as the electronic device 105. For example, one or more image sensors of the image capture device 208 may be used to generate a depth map and / or otherwise determine the depth of another device, such as the electronic device 105.

[0052] In one or more specific implementations, the image capture device 208 can also be used to segment the image of the user's body from the image of the physical environment. For example, one or more depth maps can be generated from the images captured by the multiple image sensors of the image capture device 208, and the one or more depth maps can be used to identify / segment objects in the image, such as the user.

[0053] The one or more position sensors 210 may include one or more sensors that may provide information that may be used to determine the position of the electronic device 110, such as relative to another device (e.g., the electronic device 105). For example, the one or more position sensors may include one or more gyroscopes, accelerometers, magnetometers, ultrasonic transceivers, radar / lidar transceivers, or generally any sensor that may be helpful in determining the position of the electronic device 110 relative to another device (such as the electronic device 105). In one or more specific implementations, the one or more position sensors 210 may also be used to determine the orientation of the electronic device 110, for example, along the x-axis, y-axis, and / or z-axis.

[0054] The host processor 202 may include appropriate logic components, circuits, and / or code that enable processing of data and / or control of the operation of the electronic device 110. In this regard, the host processor 202 may be enabled to provide control signals to various other components of the electronic device 110. The host processor 202 may also control the transmission of data between the various parts of the electronic device 110. In addition, the host processor 202 may enable the implementation of an operating system or otherwise execute code to manage the operation of the electronic device 110. In one or more specific implementations, the host processor 202 may facilitate the use of computer vision to determine the position of the electronic device 110 relative to another device (such as the electronic device 105). For example, the host processor (and / or one or more remote processing units, such as at the server 120) may analyze one or more images captured by the image capture device 208 to determine the position of the electronic device 110 relative to another device (such as the electronic device 105) shown in the one or more images.

[0055] The memory 204 may include suitable logic components, circuits, and / or code to enable storage of various types of information, such as received data, generated data, code, and / or configuration information. The memory 204 may include, for example, a random access memory (RAM), a read-only memory (ROM), flash memory, and / or a magnetic storage device.

[0056] In one or more specific implementations, one or more of the host processor 202, memory 204, wireless interface 206, image capture device 208, one or more position sensors 210, and / or one or more portions thereof may be implemented in software (e.g., subroutines and code), in hardware (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, a gated logic component, a discrete hardware component, or any other suitable device), and / or a combination of both.

[0057] Figure 3A flowchart of an example process 300 for automatic position synchronization of a virtual camera relative to a physical camera according to one or more specific implementations is shown. Figure 1 Process 300 is described with reference to electronic device 105 of the present invention. However, electronic device 105 is presented as an exemplary device, and one or more of the operations described herein may be performed by any suitable device. For example, process 300 may be performed by electronic device 110 or any electronic device in general. Further for the purpose of illustration, the operations of process 300 are described herein as occurring sequentially or linearly. However, multiple operations of process 300 may occur in parallel. In addition, the operations of process 300 do not have to be performed in the order shown, and / or one or more operations of process 300 do not have to be performed and / or may be replaced by other operations.

[0058] Example process 300 may be initiated by a user experiencing a computer-generated environment and / or a computer-generated reality environment utilizing electronic device 105, who desires to generate a recording that merges an image of his or her body in the physical environment with a virtual image frame, such as a virtual video, generated from the computer-generated environment and / or the computer-generated reality environment provided by electronic device 105. For example, the user may position another electronic device (such as electronic device 110) that includes a physical camera so that the field of view of the physical camera includes the user using and / or wearing electronic device 105 and / or utilizing electronic device 104 in the physical environment, such as Figure 6 The user may then select a configuration option to initiate the example process 300 via, for example, the electronic device 105 and / or the electronic device 110 .

[0059] Once the example process 300 is initiated, the electronic device 105 determines the position (e.g., location and orientation) of a physical camera (such as the electronic device 110) in the physical environment relative to the electronic device 105 (302). For example, when the physical camera includes a wireless interface that supports ranging operations, the electronic device 105 can initiate ranging operations, such as ultra-wideband ranging operations, through the physical camera. In one or more specific implementations, the electronic device 105 can utilize one or more image sensors to determine the position of the physical camera, such as by utilizing computer vision, depth maps, etc. In one or more specific implementations, the electronic device 105 can receive the position of the physical camera relative to the electronic device 105, for example, from the physical camera. For example, the physical camera can include an additional position sensor, which can be used to determine its position relative to the electronic device 105.

[0060] The electronic device 105 may initiate positioning of a virtual camera within a computer-generated reality environment provided by the electronic device 105 to coincide with a position of a physical camera in the physical environment relative to the electronic device 105 (304). The virtual camera may be positioned to have an orientation and position that coincides with the orientation and position of the physical camera relative to the electronic device 105, such that image frames captured by the physical camera are aligned with virtual image frames generated by the virtual camera. For example, the position of the virtual camera relative to a virtual representation (such as an avatar of a user) corresponding to the electronic device 105 in the computer-generated environment may coincide with the position and orientation of the physical camera in the physical environment relative to the electronic device 105.

[0061] The electronic device 105 may initiate recording of the physical environment by the physical camera while simultaneously initiating the virtual camera to generate a virtual recording of the computer-generated environment (306). For example, the electronic device 105 may transmit a message to the physical camera indicating a time when the physical camera should initiate recording. In one or more specific implementations, the recordings may not be initiated simultaneously; however, the virtual and / or physical cameras may be synchronized in time so that the images may be later time-aligned, e.g., based on a timestamp.

[0062] The electronic device 105 may receive an image frame captured by a physical camera and a virtual image frame captured by a virtual camera (308). For example, the physical camera may stream the image frame to the electronic device 105 via a wired and / or wireless connection. The electronic device 105 may segment at least one of the image frame or the virtual image frame (310). For example, the electronic device 105 may segment the image of the user from the physical image frame, such as by utilizing computer vision / object recognition and / or by utilizing additional data corresponding to the image frame provided by the electronic device 110 (such as a depth map or other information that can be used to identify and segment the image of the user from the image frame). In one or more specific implementations, the user may stand in front of a uniformly colored screen (such as a green screen) that facilitates segmenting the image of the user from the image frame.

[0063] In one or more specific implementations, the electronic device 110 may include one or more sensors, such as multiple image sensors, which may be used to generate a depth map or otherwise generate information that can be used to segment the image of the user from the image frame. The electronic device 110 may provide the segmented image of the user to the electronic device 105 so that the electronic device 105 may bypass the segmentation and synthesize the segmented user image onto the virtual image.

[0064] The electronic device 105 generates a computer-generated reality frame from at least a portion of the image frame (e.g., a segmented user image) and at least a portion of the virtual image frame (such as the entire virtual image frame excluding a virtual representation of the user, such as an avatar of the user) (312). In one or more specific implementations, the electronic device 105 may overlay the image of the user onto the avatar of the user in the corresponding virtual image frame to generate the computer-generated reality image frame.

[0065] The electronic device 105 provides a computer-generated reality image frame, such as for display to a user, for local and / or remote storage, and / or for streaming to one or more other users (314). Operations 308-314 may be repeated for each image frame and corresponding virtual image frame to generate a computer-generated reality video stream. Thus, for example, a computer-generated reality image frame and / or video stream may be displayed to a user by the electronic device 105 so that the user can preview the computer-generated reality video stream and make any physical and / or virtual camera adjustments that may be needed and / or desired.

[0066] In one or more implementations, the electronic device 105 can continuously monitor the position of the physical camera relative to the electronic device 105 (302), and can continuously adjust the position of the virtual camera to coincide with changes in the position of the physical camera. Thus, if another user is holding the physical camera, and the other user is moving, shaking, or otherwise not holding the physical camera in a fixed position, the position of the virtual camera will also move and / or shake, so that the entire computer-generated reality image frame will reflect the movement / shake, rather than just the physical image frame portion of the computer-generated reality image frame.

[0067] In one or more implementations, the example process 300 may be performed in whole or in part by the electronic device 110. For example, the electronic device 110 may transmit instructions to the electronic device 105 and may receive a stream of virtual image frames generated by a virtual camera positioned by the electronic device 105. In one or more implementations, at least a portion of the example process 300 may be performed and / or facilitated by the server 120. For example, the server may receive image frames captured by a physical camera and virtual image frames generated by a virtual image camera, may generate a computer-generated reality image frame, and may transmit the computer-generated reality image frame to the electronic device 110, the electronic device 105, and / or one or more additional electronic devices, such as electronic devices of other users authorized to view the user's computer-generated reality video stream.

[0068] In one or more specific implementations, the electronic device 105 and the physical camera (such as the electronic device 110) can be associated with and / or registered to the same user account. The electronic device 105 and / or the electronic device 110 can confirm that the other device is associated with the same user account before initiating the example process 300. Alternatively and / or in addition, the electronic device 105 and the electronic device 110 can be associated with different user accounts and can participate in an initial pairing / authorization operation before initiating the example process 300.

[0069] Figure 4 A flowchart of an example process for providing guidance for synchronizing the position of a physical camera relative to a virtual camera according to one or more specific implementations is shown. Figure 1 Process 400 is described with reference to electronic device 105 of the present invention. However, electronic device 105 is presented as an exemplary device, and one or more of the operations described herein may be performed by any suitable device. For example, process 400 may be performed by electronic device 110 or any electronic device in general. Further for the purpose of illustration, the operations of process 400 are described herein as occurring sequentially or linearly. However, multiple operations of process 400 may occur in parallel. In addition, the operations of process 400 do not have to be performed in the order shown, and / or one or more operations of process 400 do not have to be performed and / or may be replaced by other operations.

[0070] The example process 400 may be initiated by a user experiencing a computer-generated environment and / or a computer-generated reality environment using and / or wearing the electronic device 105, the user desiring to generate a recording that merges an image of their body in the physical environment with a virtual image frame, such as a virtual video, generated from the computer-generated environment and / or the computer-generated reality environment provided by the electronic device 105. For example, the user may position a virtual camera in a particular position within the computer-generated environment having a desired angle or field of view. The process 400 may then be initiated, such as by selecting a user interface element, to facilitate the user to effectuate positioning of a physical camera, such as the electronic device 110, in the physical environment in a position that coincides with the position of the virtual camera in the computer-generated environment, such as by selecting a user interface element. Figure 6 shown.

[0071] Once the example process 400 is initiated, the electronic device 105 determines a position of a virtual camera in a computer-generated environment provided by the electronic device 105 relative to a position in the computer-generated environment corresponding to the electronic device 105 (402). The position in the computer-generated environment corresponding to the electronic device 105 may be, for example, a position of an avatar of a user in the computer-generated environment.

[0072] The electronic device 105 then provides guidance to the user for positioning the physical camera in the physical environment at a location that coincides with the location of the virtual camera in the computer-generated environment relative to the electronic device 105 (404). For example, the electronic device 105 may provide the user with an augmented reality display that indicates the location in the physical environment where the physical camera should be positioned.

[0073] In one or more implementations, process 400 may be performed by a physical camera, such as electronic device 110. For example, the physical camera may receive location information from electronic device 105, and may then facilitate a user to place the physical camera at an appropriate location in a physical environment. In one or more implementations, the physical camera may be included in an autonomous mobile device, such as a drone or a robotic device, which may automatically move the physical camera to a location indicated by electronic device 105.

[0074] Figure 5 A flowchart of an example process for providing a viewport into a computer-generated reality environment according to one or more specific implementations is shown. For the purpose of explanation, this article mainly refers to Figure 1 Process 500 is described with reference to electronic device 110 of the present invention. However, electronic device 110 is presented as an exemplary device, and the operations described herein may be performed by any suitable device. For example, process 500 may be performed by electronic device 112 or any electronic device in general. Further for the purpose of explanation, the operations of process 500 are described herein as occurring sequentially or linearly. However, multiple operations of process 500 may occur in parallel. In addition, the operations of process 500 do not have to be performed in the order shown, and / or one or more operations of process 500 do not have to be performed and / or may be replaced by other operations.

[0075] The example process 500 may be initiated when a user is experiencing a computer-generated reality environment via the electronic device 105 and another user wants to view the computer-generated reality environment. Figure 7 As shown, a user using and / or wearing electronic device 105 may be playing a virtual game with a virtual game board positioned on (and / or adjacent to) a physical object (such as a table) in a physical environment. Although other users in the room can see the physical table, they cannot see the virtual game that the user is playing on the physical table using and / or wearing electronic device 105. Therefore, other users can request to receive a viewport into the computer-generated real environment provided by electronic device 105 via electronic device 110.

[0076] In one or more specific implementations, other users may be automatically granted access to the viewport if electronic device 110 is registered to and / or associated with the same user account as electronic device 105. However, if electronic device 110 is registered to and / or associated with a different user account than electronic device 105, electronic device 110 may transmit a request for authorization to access the viewport to electronic device 105. For example, electronic device 105 may present the request to a user using and / or wearing electronic device 105, and the user may accept or deny the request.

[0077] When the electronic device 110 is granted access to a viewport into a computer-generated real environment generated by the electronic device 105, the electronic device 110 determines a position of a physical camera on the electronic device 110 in the physical environment relative to the electronic device 105 (502). For example, the position may be determined by utilizing the electronic device 105 to perform an ultra-wideband ranging operation and / or any other suitable positioning technique described herein.

[0078] Electronic device 110 may transmit its position to electronic device 105, and / or electronic device 105 may use suitable positioning technology to determine the position of electronic device 110 (504). Electronic device 105 may position the virtual camera in the computer-generated real environment at a position that coincides with the position of the physical camera of electronic device 110 in the physical environment.

[0079] Then, the electronic device 110 receives at least a portion of the virtual image frame generated by the virtual camera from the electronic device 105 (506). For example, the virtual camera may capture only the virtual object in the computer-generated real environment, and / or the electronic device 105 may segment the virtual object from the virtual image and provide only the virtual object to the electronic device 110. The electronic device 110 also receives an image frame captured by a physical camera of the electronic device 110 (508) and synthesizes at least a portion of the virtual image onto the physical image frame, such as by overlaying at least the portion of the virtual image onto the physical image frame, to generate a computer-generated real image frame (510).

[0080] The electronic device 110 then displays the computer-generated reality image frame on the display of the electronic device 110 to provide a viewport into the computer-generated reality environment provided by the electronic device 105 (512). Operations 506-512 may be repeated, such as for each image frame, to display a computer-generated reality video stream that provides a real-time viewport into the computer-generated reality environment. In one or more specific implementations, the electronic device 105 may display an indication (such as a red icon) to the user using and / or wearing the electronic device 105, indicating that a viewport into the computer-generated reality environment is being provided to another user, such as via the electronic device 110.

[0081] Figure 6 An example environment is shown in which position synchronization of virtual and physical cameras can be achieved for computer-generated reality recording according to one or more implementations. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0082] In the example environment 600, the user has positioned the electronic device 110 including a physical camera at a location where the field of view of the physical camera encompasses the user using and / or wearing the electronic device 104 (or the electronic device 105). The electronic device 104 may then perform, for example, the above-described process 300 to position the virtual camera in the computer-generated real environment provided by the electronic device 104. Alternatively and / or in addition, the user may have pre-configured the position of the virtual camera in the computer-generated environment, and the process 400 may have been used to determine an appropriate location to place the electronic device 110.

[0083] Once the position of the physical camera of the electronic device 110 and the position of the virtual camera in the computer-generated environment provided by the electronic device 104 are synchronized, the image frames captured by the physical camera can be synthesized with the virtual image frames generated by the virtual camera to generate one or more computer-generated reality image frames.

[0084] In one or more implementations, physical and virtual recording may not occur simultaneously. For example, physical camera recording may be performed first, and virtual recording may be added later, or vice versa. In this case, the positions of the virtual and physical cameras will still be synchronized to allow subsequent recording synthesis; however, the recording may be performed asynchronously.

[0085] Figure 7An example environment 700 is shown in accordance with one or more implementations in which synchronization of the positions of virtual and physical cameras can be achieved to provide a computer-generated reality viewport. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0086] In example environment 700, a user is using and / or wearing electronic device 105 to experience a computer-generated reality environment that includes virtual objects positioned on and / or adjacent to a physical table 704. For example, electronic device 110 may perform process 500 to provide another user with a viewport 702 into the computer-generated reality environment being experienced by the user using and / or wearing electronic device 105. In one or more specific implementations, the image of electronic device 105 may be removed from the computer-generated reality image frame providing the viewport and may instead be replaced with an image of the user's actual face and / or avatar.

[0087] In one or more specific implementations, a user using and / or wearing the electronic device 105 may be experiencing a computer-generated reality environment and / or may be participating in a co-presence session with another user. The user may wish to use a physical camera on their electronic device 110 to generate a self-captured image of the computer-generated environment, which may be colloquially referred to as a "selfie."

[0088] Thus, the electronic device 105 and / or the electronic device 110 may perform the process 300 to place the virtual camera in the computer-generated environment at a position that coincides with the position of the physical camera of the electronic device 110 in the physical environment relative to the electronic device 105. In this way, the virtual image frames generated by the virtual camera may be synthesized with the image frames generated by the physical camera of the electronic device 110 to generate a mixed reality self-captured image or a mixed reality selfie.

[0089] In one or more specific implementations, the image of the electronic device 105 can be removed from the mixed reality image and replaced with an image of the face and / or avatar of the user using and / or wearing the electronic device 105. If the user is participating in a co-presence session with one or more other users and is capturing a mixed reality selfie of the one or more other users, the electronic device being used and / or worn by the other users can also be replaced with the face and / or avatar of the other users in the computer-generated reality image.

[0090] As described above, one aspect of the present technology is to collect and use data from various sources. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, social network identifiers, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0091] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. The present disclosure also contemplates uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals using technology to pursue health goals.

[0092] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0093] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, the present technology may be configured to allow users to “opt in” or “opt out” of participating in the collection of personal information data at any time during or after registration for a service. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0094] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0095] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user based on non-personal information data or a small amount of personal information, such as content requested by a device associated with the user, other non-personal information, or publicly available information.

[0096] Figure 8 An exemplary electronic system 800 is shown that can be used to implement various aspects of the subject technology according to one or more specific implementations. The electronic system 800 can be Figure 1One or more of the electronic device 105, electronic device 104, electronic device 110, electronic device 112, electronic device 115 and / or server 120 shown and / or may be a part thereof. The electronic system 800 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 800 includes a bus 808, one or more processing units 812, a system memory 804 (and / or a cache), a ROM 810, a permanent storage device 802, an input device interface 814, an output device interface 806, and one or more network interfaces 816, or a subset and variation thereof.

[0097] The bus 808 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 800. In one or more specific implementations, the bus 808 communicatively connects one or more processing units 812 with the ROM 810, the system memory 804, and the permanent storage device 802. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different specific implementations, the one or more processing units 812 can be a single processor or a multi-core processor.

[0098] ROM 810 stores static data and instructions required by one or more processing units 812 and other modules of the electronic system 800. On the other hand, permanent storage device 802 can be a read-write memory device. Permanent storage device 802 can be a non-volatile memory unit that stores instructions and data even when the electronic system 800 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as permanent storage device 802.

[0099] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 802. Like the permanent storage device 802, the system memory 804 may be a read-write memory device. However, unlike the permanent storage device 802, the system memory 804 may be a volatile read-write memory, such as a random access memory. The system memory 804 may store any of the instructions and data that one or more processing units 812 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 804, the permanent storage device 802, and / or the ROM 810. The one or more processing units 812 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0100] The bus 808 is also connected to an input device interface 814 and an output device interface 806. The input device interface 814 enables a user to transmit information and select commands to the electronic system 800. Input devices that can be used with the input device interface 814 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 806 may, for example, enable the display of images generated by the electronic system 800. Output devices that can be used with the output device interface 806 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0101] Finally, if Figure 8 As shown, bus 808 also couples electronic system 800 to one or more networks and / or to one or more network nodes via one or more network interfaces 816. In this manner, electronic system 800 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of electronic system 800 may be used with the subject disclosure.

[0102] The above functions can be implemented in computer software, firmware or hardware. The technology can be implemented using one or more computer program products. The programmable processor and computer can be included in a mobile device or packaged as a mobile device. The process and logic flow can be performed by one or more programmable processors and one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through a communication network.

[0103] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files including higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0104] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0105] As used in this specification and any claims of this patent application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms exclude people or groups of people. For the purposes of this specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, tangible objects that store information in a form that can be read by a computer. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0106] To provide interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0107] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with a specific implementation of the subject matter described in this specification, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and may interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some specific implementations, the server transmits data (e.g., an HTML page) to a client device (e.g., in order to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., a result of a user interaction) may be received at the server from the client device.

[0109] Those skilled in the art will recognize that various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The functions can be implemented in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways), without departing from the scope of the subject technology.

[0110] It should be understood that the specific order or hierarchy of steps in the process disclosed by the present invention is an illustration of an exemplary method. Based on design preference requirements, it should be understood that the specific order or hierarchy of steps in the process can be rearranged. Some of the steps can be performed simultaneously. The attached method claims present elements of various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0111] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications of these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to the singular element is not intended to mean "only one", but to refer to "one or more", unless specifically indicated. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are only for convenience and do not limit the present invention described herein.

[0112] As used herein, the term website may include any aspect of a website, including one or more web pages, one or more servers for hosting or storing web-related content, and the like. Thus, the term website may be used interchangeably with the terms web page and server. The predicate words "configured to," "capable of operating to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is capable of operating to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code or is capable of operating to execute code.

[0113] As used herein, the term automatically may include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0114] Phrases such as "aspects" do not mean that this aspect is essential to the subject technology or that this aspect applies to all configurations of the subject technology. The disclosure related to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "implementation" do not mean that this implementation is essential to the subject technology or that this implementation applies to all configurations of the subject technology. The disclosure related to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, and vice versa. Phrases such as "configuration" do not mean that this configuration is essential to the subject technology or that this configuration applies to all configurations of the subject technology. The disclosure related to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, and vice versa.

Claims

1. An apparatus for generating a synthetic computer-generated record of reality, comprising: Memory; as well as at least one processor, the at least one processor being configured to: determining a first position of a physical camera in a physical environment relative to the device; initiating positioning of a virtual camera in a second position within the computer-generated environment, wherein the second position of the virtual camera in the computer-generated environment relative to a virtual representation corresponding to a person coincides with the first position of the physical camera in the physical environment relative to the device, the person being associated with the device; receiving at least one image frame captured by the physical camera of the physical environment and at least one virtual image frame generated by the virtual camera of the computer-generated environment; as well as Generate at least one computer-generated real image frame, the at least one computer-generated real image frame comprising a synthesis of at least a portion of the at least one image frame and at least a portion of the at least one virtual image frame, wherein the at least a portion of the at least one image frame comprises a segmented image of the person, and the segmented image of the person replaces the virtual representation corresponding to the person in the at least one computer-generated real image frame.

2. The device of claim 1 , wherein the virtual representation corresponding to the person includes an avatar of the person, a field of view of the virtual camera includes the avatar of the person, a field of view of the physical camera includes the device, and the at least one processor is further configured to: The image of the person is segmented from the at least one image frame captured by the physical camera.

3. The apparatus of claim 2, wherein the at least one processor is further configured to: The segmented image of the person is synthesized with the at least portion of the at least one virtual image frame to generate the at least one computer-generated real image frame, the segmented image of the person replacing the avatar of the person in the at least one computer-generated real image frame.

4. The apparatus of claim 1 , wherein the at least one processor is further configured to: segmenting at least one virtual object from the at least one virtual image frame; and The at least one virtual object is composited with the at least portion of the at least one image frame to generate the at least one computer generated reality image frame.

5. The apparatus of claim 4, wherein the at least one virtual object comprises another avatar of another user in the computer-generated environment.

6. The apparatus of claim 4, wherein the computer-generated environment comprises a computer-generated reality environment, and the at least one virtual object is displayed in the computer-generated reality environment as being adjacent to a physical object in the physical environment.

7. The apparatus of claim 1, wherein the at least one processor is further configured to: detecting a change in the first position of the physical camera; and A change of the second position of the virtual camera is initiated to coincide with the changed first position of the physical camera.

8. The apparatus of claim 1, wherein the at least one processor is further configured to: exchanging one or more signals between the physical camera and the device; and The first position of the physical camera in the physical environment is determined based at least in part on one or more times of arrival of one or more exchanged signals.

9. The apparatus of claim 1 , wherein the at least one processor is further configured to: Computer vision is utilized to determine the first position of the physical camera relative to the device in a physical environment.

10. The device of claim 1, wherein the physical camera and the device are both associated with the person's account, and the device includes at least one of the physical camera or the device.

11. The device of claim 1 , wherein the at least one processor is further configured to: The at least one computer generated reality image frame is displayed.

12. A method of generating a synthetic computer-generated reality record, comprising: Determining a first position of a physical camera relative to the device in a physical environment; initiating positioning of a virtual camera in a second position within the computer-generated environment, wherein the second position of the virtual camera in the computer-generated environment relative to a virtual representation corresponding to a person coincides with the first position of the physical camera in the physical environment relative to the device, the person being associated with the device; receiving at least one image frame captured by the physical camera of the physical environment and at least one virtual image frame generated by the virtual camera of the computer-generated environment; as well as Generate at least one computer-generated real image frame, the at least one computer-generated real image frame comprising a synthesis of at least a portion of the at least one image frame and at least a portion of the at least one virtual image frame, wherein the at least a portion of the at least one image frame comprises a segmented image of the person, and the segmented image of the person replaces the virtual representation corresponding to the person in the at least one computer-generated real image frame.

13. The method of claim 12, wherein the virtual representation corresponding to the person includes an avatar of the person, a field of view of the virtual camera includes the avatar of the person, a field of view of the physical camera includes the device, and the method further comprises: The image of the person is segmented from the at least one image frame captured by the physical camera.

14. The method according to claim 13, further comprising: The segmented image of the person is synthesized with the at least portion of the at least one virtual image frame to generate the at least one computer-generated real image frame, the segmented image of the person replacing the avatar of the person in the at least one computer-generated real image frame.

15. The method according to claim 12, further comprising: segmenting at least one virtual object from the at least one virtual image frame; as well as The at least one virtual object is composited with the at least portion of the at least one image frame to generate the at least one computer generated reality image frame.

16. The method of claim 15, wherein the at least one virtual object comprises another avatar of another user in the computer-generated environment.

17. The method of claim 15, wherein the computer-generated environment comprises a computer-generated reality environment, and the at least one virtual object is displayed in the computer-generated reality environment adjacent to a physical object in the physical environment.

18. The method according to claim 12, further comprising: detecting a change in the first position of the physical camera; as well as A change of the second position of the virtual camera is initiated to coincide with the changed first position of the physical camera.

19. The method according to claim 12, further comprising: exchanging one or more signals between the physical camera and the device; as well as The first position of the physical camera in the physical environment is determined based at least in part on one or more times of arrival of one or more exchanged signals.

20. The method of claim 12, further comprising: Computer vision is utilized to determine the first position of the physical camera relative to the device in a physical environment.

21. The method of claim 12, wherein the physical camera and the device are both associated with an account of the person, and the device comprises at least one of the physical camera or the device.

22. The method of claim 12, further comprising: The at least one computer generated reality image frame is displayed.

23. A non-transitory machine-readable medium comprising code that, when executed by one or more processors, causes the one or more processors to perform operations, the code comprising: code for determining a first position of a physical camera in a physical environment relative to the device; code for initiating positioning a virtual camera in a second position within a computer-generated environment, wherein the second position of the virtual camera in the computer-generated environment relative to a virtual representation corresponding to a person coincides with the first position of the physical camera in the physical environment relative to the device, the person being associated with the device; code for receiving at least one image frame captured by the physical camera of the physical environment and at least one virtual image frame generated by the virtual camera of the computer-generated environment; as well as Code for generating at least one computer-generated reality image frame, the at least one computer-generated reality image frame comprising a synthesis of at least a portion of the at least one image frame and at least a portion of the at least one virtual image frame, wherein the at least a portion of the at least one image frame comprises a segmented image of the person, the segmented image of the person replacing the virtual representation corresponding to the person in the at least one computer-generated reality image frame.

24. The machine-readable medium of claim 23, wherein the virtual representation corresponding to the person includes an avatar of the person, a field of view of the virtual camera includes the avatar of the person, a field of view of the physical camera includes the device, and the code further comprises: Code for segmenting an image of the person from the at least one image frame captured by the physical camera.

25. The machine-readable medium of claim 24, the code further comprising: Code for synthesizing the segmented image of the person with the at least portion of the at least one virtual image frame to generate the at least one computer-generated real image frame, the segmented image of the person replacing the avatar of the person in the at least one computer-generated real image frame.

26. The machine-readable medium of claim 23, the code further comprising: code for segmenting at least one virtual object from the at least one virtual image frame; as well as Code for compositing the at least one virtual object with the at least portion of the at least one image frame to generate the at least one computer generated reality image frame.

27. The machine-readable medium of claim 26, wherein the at least one virtual object comprises another avatar of another user in the computer-generated environment.

28. The machine-readable medium of claim 26, wherein the computer-generated environment comprises a computer-generated reality environment, and the at least one virtual object is displayed in the computer-generated reality environment adjacent to a physical object in the physical environment.

29. The machine-readable medium of claim 23, the code further comprising: code for detecting a change in the first position of the physical camera; as well as Code for initiating a change of the second position of the virtual camera to coincide with the changed first position of the physical camera.

30. The machine-readable medium of claim 23, the code further comprising: code for exchanging one or more signals between the physical camera and the device; as well as Code for determining the first position of the physical camera in the physical environment based at least in part on one or more times of arrival of one or more exchanged signals.

31. The machine-readable medium of claim 23, the code further comprising: Code for utilizing computer vision to determine the first position of the physical camera relative to the device in a physical environment.

32. The machine-readable medium of claim 23, wherein the physical camera and the device are both associated with an account of the person, and the device comprises at least one of the physical camera or the device.

33. The machine-readable medium of claim 23, the code further comprising: Code for displaying the at least one computer generated reality image frame.

Images