User-controlled 3D scenes

Abstract

The present disclosure generally relates to systems and methods for a user to control a virtual representation of himself / herself within a three-dimensional virtual world. The systems and methods utilize a user's two-dimensional image or video data along with extracted depth information to allow the user to position himself / herself within the three-dimensional scene. A control system and method is also provided for a user to control a virtual representation of himself / herself within a three-dimensional space that contains the virtual representation of himself / herself, using the output video as a visualization feedback mechanism. The user interacts with other virtual objects or items within the scene, or even with other users visualized within the scene.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to systems and methods for a user to control a virtual representation of himself or herself within a three-dimensional virtual world. [Background technology]

[0002] Interacting with the virtual world currently involves the use of a virtual reality (VR) headset, utilizing specialized 3D sensors, typing on a keyboard and computer, or swiping on a mobile phone to control an avatar that navigates through and interacts with the virtual environment.

[0003] The present disclosure can be used to insert an image or representation of a user / actor into a three-dimensional virtual world, allowing the user to move in their own physical world as they see themselves in that virtual world and control the movement of their own image or representation and interaction with objects in the virtual world they see.

[0004] This disclosure provides systems and methods that are useful for many situations where an actor needs to be present on screen in real time with other content, but without overlapping and occluding any objects in the virtual world. This is accomplished by inserting a representation of the actor into a "scene." Such a scene consists of content at many different levels or layers of depth, where, from the viewer's perspective, some of the content is behind the actor and some content is in front of the actor and therefore not occluded by the actor.

[0005] A scene may be composed of voxels (three-dimensional pixels) in a three-dimensional space of xyz, and a user's representation is inserted as a set of voxels contained within the scene. This disclosure provides systems and methods that allow a user(s) to interact in real time with other objects or items in the scene, or even with each other in a multi-user case.

[0006] Often, green screen and virtual green screen techniques are now used to place an image of a person over a background image or video. Much like a meteorologist on the evening news, the "weatherman" effect occludes the view of the subject material, so that a person who needs to move side to side can point to parts of the background without blocking them.

[0007] This is especially problematic for remote presentations: today, an actor's video stream is typically displayed in a window that is completely separate from the content, making it more difficult for the actor to point at the content (with anything other than a mouse pointer) and communicate more effectively with the audience. When an actor is placed in front of the content (such as by using a chromakey green screen), the actor will occlude part of the very content he is trying to present. Summary of the Invention [Problem to be solved by the invention]

[0008] The subject matter of this application may, in some cases, involve interrelated products, alternative solutions to a particular problem, and / or multiple different uses of a single system or product. This disclosure therefore includes, without limitation, the following example embodiments. [Means for solving the problem]

[0009] The present disclosure relates generally to systems and methods for a user to control a virtual representation of themselves within a three-dimensional virtual world that utilizes two-dimensional image or video data of a user along with extracted depth information to enable the user to position themselves within the three-dimensional scene.

[0010] Some example embodiments provide a computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method including: receiving two-dimensional video stream data of a user using an imaging device, the two-dimensional video stream data at least partially capturing the user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separating a user representation of the user from the two-dimensional video stream data; identifying user position information from the two-dimensional video stream, the position information being identified based on a position of the user relative to the imaging device; using the position information to add the user representation to the three-dimensional scene by modifying data associated with voxels that comprise the three-dimensional scene; and displaying the three-dimensional scene and the added user representation on a display device, wherein the added user representation displayed on the display device is controlled based on detection of a change to at least one of the user and the position information.

[0011] Some example embodiments provide a computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method including: receiving, using a first imaging device, first two-dimensional video stream data of a first user, the first two-dimensional video stream data at least partially capturing the first user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; isolating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, the first position information being identified based on a relative position of the first user with respect to the first imaging device; receiving, using a second imaging device, second two-dimensional video stream data of a second user, the second two-dimensional video stream data at least partially capturing the second user; and isolating a second user representation of the second user from the second two-dimensional video stream data. identifying second position information of a second user from the second two-dimensional video stream, the second position information being identified based on a relative position of the second user with respect to the second imaging device; adding first and second user representations to the three-dimensional scene by modifying data associated with voxels comprising the three-dimensional scene, the data associated with the first user representation using the first position information and the data associated with the second user representation using the second position information; and displaying the three-dimensional scene, the added first user representation, and the added second user representation on the first display device and the second display device, the first added user representation displayed on the first display device being controlled based on detection of a change to at least one of the first user and the first position information and the second added user representation displayed on the second display device being controlled based on detection of a change to at least one of the second user and the second position information.

[0012] Some example implementations provide a system for controlling a virtual representation of a user within a three-dimensional scene, comprising: an imaging device that captures two-dimensional video stream data of the user; a storage device having computer-readable program instructions stored therein; and a processor in communication with the storage device and configured to execute the computer-readable instructions to cause the system to at least: receive two-dimensional video stream data of the user from the imaging device, the two-dimensional video stream data at least partially capturing the user; receive a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separate a user representation of the user from the two-dimensional video stream data; identify position information of the user from the two-dimensional video stream, the position information being identified based on a position of the user relative to the imaging device; add the user representation to the three-dimensional scene by modifying data associated with voxels that comprise the three-dimensional scene using the position information and pose information; and display the three-dimensional scene and the added user representation on a display device, wherein the added user representation displayed on the display device is controlled based on detection of a change to at least one of the user and the position information. Effect of the Invention

[0013] These and other features, aspects, and advantages of the present disclosure will be apparent from reading the following detailed description in conjunction with the accompanying drawings, which are briefly described below. The present disclosure includes any combination of two, three, four, or more features or elements described in the present disclosure, regardless of whether such features or elements are explicitly combined in a particular example embodiment described herein or otherwise recited. Unless otherwise clearly indicated in the context of the present disclosure, the present disclosure is intended to be read as a whole such that any separable features or elements of the present disclosure should be considered as combinable in any of its aspects and example embodiments.

[0014] It will therefore be understood that this Summary is provided solely for the purpose of summarizing some example implementations in order to provide a basic understanding of some aspects of the present disclosure. Accordingly, it will be understood that the foregoing example implementations are merely examples, and should not be construed as narrowing the scope or spirit of the present disclosure in any way. Other example implementations, aspects and advantages will become apparent from the following detailed description, taken in conjunction with the accompanying drawings which illustrate, by way of example, the principles of some described example implementations.

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate aspects of the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]

[0016] [Figure 1] 1 illustrates a system according to some example embodiments. [Figure 2-3] 4 is a flow diagram illustrating various steps in a method for generating a multi-layered scene, according to various example embodiments of the present disclosure. [Figure 4-9] 1 illustrates an example embodiment in accordance with the systems and methods of the present disclosure. [Figure 10-11] 1 illustrates an example implementation of an actor interacting with an interacting object in accordance with the disclosed systems and methods. [Figure 12A-12B] 1 illustrates an example embodiment of multiple actors interacting with an interacting object in accordance with the systems and methods of the present disclosure. [Figure 13A-29] 1 illustrates various embedding examples in accordance with the systems and methods of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] It should be noted that some details of the drawings have been drawn in a simplified manner for ease of understanding rather than maintaining strict structural accuracy, detail, and scale.

[0018] The detailed description set forth below in connection with the accompanying drawings or figures is intended as a description of the presently preferred embodiments of the disclosure and is not intended to represent the only manner in which the disclosure may be constructed and / or utilized. The description sets forth the functions and the sequence of steps for constructing and operating the disclosure in connection with the illustrated examples.

[0019] In this description, specific details are set forth in order to provide a thorough understanding of the disclosed examples. In other instances, well-known methods, procedures, components and materials have not been described in detail so as not to unnecessarily lengthen the disclosure.

[0020] These and other features, aspects, and advantages of the present disclosure will be apparent from reading the following detailed description in conjunction with the accompanying drawings, which are briefly described below. The present disclosure includes any combination of two, three, four, or more features or elements described in the present disclosure, regardless of whether such features or elements are explicitly combined in a particular example embodiment described herein or otherwise recited. Unless otherwise clearly indicated in the context of the present disclosure, the present disclosure is intended to be read as a whole such that any separable features or elements of the present disclosure should be considered as combinable in any of its aspects and example embodiments.

[0021] When an element or component is referred to herein as "on," "in contact," "against," "in communication," "connected," "attached," or "coupled" to another element or component, it should be understood that it is directly on, in contact with, in communication with, connected, attached, or coupled to the other element or component, or there may be intervening elements or components. When used, the term "and / or," when so defined, includes any and all combinations of one or more of the associated listed items.

[0022] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It should be further understood that the terms "include" and / or "including", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof not expressly stated.

[0023] Further, it should be understood that unless otherwise specified, the terms "data," "content," "digital content," "information," and similar terms may be used interchangeably at times.

[0024] Additionally, unless otherwise specified, the term "real-time" should be understood to refer to processing, displaying, capturing, detecting, identifying, or analyzing data without intentional delay, given the processing limitations of the system.

[0025] Various operations may be described in sequence as multiple separate operations in a manner that may be helpful to understanding the embodiments, but the order of description should not be construed to imply that these operations are order dependent.

[0026] Spatially relative terms such as "below," "below," "below," "downward," "upward," "proximal," "distal," "rear," "front," and the like may be used herein for ease of explanation and / or illustration to describe the relationship of one element or feature to another element(s) or feature(s) as shown in the various figures. However, it should be understood that the spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation shown in the figures. For example, if the device in the figures is turned over, an element described as "below" or "below" the other element or feature would thereby be oriented "above" the other element or feature. Thus, a relative spatial term such as "below" may encompass both an above or below orientation. The device may be oriented differently (rotated 90 degrees or to other orientations) and the spatially relative descriptors used herein should be interpreted accordingly. Similarly, the relative spatial terms "proximal" and "distal" may also be interchangeable where applicable. Such descriptors are used for ease of explanation only and are not intended to limit the application of the disclosed embodiments.

[0027] Terms such as first, second, third, etc. may be used herein to describe various elements, components, regions, parts, and / or sections. It should be understood that these elements, components, regions, parts, and / or sections should not be limited by these terms. These terms are used only to distinguish one element, component, region, part, or section from another region, part, or section. Thus, a first element, component, region, part, or section described below may be referred to as a second element, component, region, part, or section without departing from the teachings of this specification.

[0028] Some examples of the present disclosure may be implemented on a computer system that generally includes one or more processors for processing information and instructions, RAM for storing information and instructions, ROM for storing static information and instructions, data storage devices such as magnetic or optical disks and disk drives for storing information and instructions, modules as software units that execute on the processors, optional user output devices such as a display screen device (e.g., a monitor) for displaying screening information to a computer user, and optional user input devices.

[0029] As can be appreciated by those skilled in the art, the present examples may be embodied, at least in part, in a computer program product embodied in any tangible means of expression having computer usable program code stored therein. For example, some embodiments described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products may be implemented by computer program instructions. The computer program instructions may be stored in a computer readable medium that can direct a computer, controller or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium constitute a product including instructions and processes that implement the functions / operations / steps specified in the flowcharts and / or block diagrams. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine, such that the instructions, executing via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in the flowchart and / or block diagram block or blocks.

[0030] In the following description, reference is made to the accompanying drawings, which are illustrations of embodiments in which the disclosed system may be practiced, however, it should be understood that those skilled in the art may develop other structural and functional modifications without departing from the novelty and scope of the present disclosure.

[0031] The system may include one or more computers or computer-controlled elements that communicate and cooperate to perform different functions of the system. The systems and methods contemplated herein may further include a storage device, such as a non-transitory computer-readable storage medium, configured to instruct a computer or multiple computers to perform the steps and functions of the system and method as described herein. In some embodiments, communication between one or more computers or similarly one or more processors may support multiple encryption / decryption methods and mechanisms of various types of data.

[0032] A computer-controlled user interface may be comprised of one or more computing devices in network communication with each other. The computer or computers of the computer-controlled user interface contemplated herein may include a memory, a processor, and an input / output system. In some embodiments, the computer may further include a network connection and / or a display screen. These computer-controlled elements may cooperate in a network to provide functionality to the computer-controlled user interface. The computer-controlled user interface may be any type of computer-controlled interface known in the art that can allow a user to input data and receive feedback therefrom. The computer-controlled user interface may further provide output executed by the system contemplated herein.

[0033] Storage or data contemplated herein may be in any format including, but not limited to, XML, JSON, CSV, 2-Value, over any protocol: UDP, TCP, and the like, over any connection type: serial, Ethernet, etc.

[0034] Computers or computing devices contemplated herein may include, but are not limited to, virtual systems, cloud / remote systems, desktop computers, laptop computers, tablet computers, handheld computers, smartphones and other mobile phones and similar Internet-enabled mobile devices, digital cameras, special purpose computing devices configured specifically to perform the methods contemplated in this disclosure, and the like.

[0035] Networks contemplated herein may include, for example, one or more of the Internet, a wide area network (WAN), a local area network (LAN), analog or digital wired and wireless telephone networks (e.g., PSTN, Integrated Services Digital Network (ISDN), mobile networks, and digital subscriber lines (xDSL)), radio, television, cable, satellite, and / or any other distribution or tunneling mechanism for carrying data. A network may include multiple networks or sub-networks, each of which may include, for example, wired or wireless data pathways. A network may include a circuit-switched voice network, a packet-switched data network, or any other network capable of carrying electronic communications. Examples include, but are not limited to, Picture Transfer Protocol (PTP) over Internet Protocol (IP), IP over Bluetooth, IP over WiFi, and PTP over IP networks (PTP / IP).

[0036] Imaging devices or video capture devices contemplated herein may include, but are not limited to, webcams, cell phone cameras, depth-sensing imaging devices (including, but not limited to, stereoscopic cameras, multiple cameras used together, 2D cameras supplemented with lidar, cameras on cell phones with depth-sensing technology, and the like), DSLRs, non-SLR digital cameras (e.g., but not limited to, compact digital cameras and SLR-like bridge digital cameras (also known as modern digital cameras), and SLR-like interchangeable lens digital cameras), and video recorders (e.g., but not limited to, camcorders, 2D analog cameras and 2D IP cameras, 3D cameras, and the like; devices that can provide a video feed of any duration, such as DVRs; portable computing devices with cameras, such as tablet computers, laptop computers, smartphones, and the like). The position or location of the imaging device may also vary. The imaging device may be oriented to capture the front of the actor, the face of the actor, a side view of the actor, a top view of the actor, a back view of the actor, a perspective view of the actor, and the like. Similarly, the imaging device can zoom in and out to change the captured area of ​​an actor or object.

[0037] In some embodiments, the system may utilize a display that is positioned away from the user such that the user's movements are not hindered by any wearable device while using the system.Similarly, the system may include an imaging device that is positioned away from the user such that the user's movements are not restricted by any such wearable device.

[0038] An image or video file contemplated herein may be in any digital image format interpretable by a computer or computing device. Examples of image files contemplated herein include, but are not limited to, JPEG, GIF, TIFF, PNG, Bitmap, RAW, PNM, WEBP, etc.

[0039] The present disclosure provides systems and methods for allowing a user to be represented in a three-dimensional scene and control the movement and interaction of their own representation within the three-dimensional scene through the use of an imaging device (including a two-dimensional imaging device). More specifically, the present disclosure provides systems and methods for adding a representation or image of a user (such as separated from a two-dimensional video stream captured by an imaging device) into a three-dimensional scene by modifying the voxels of the scene, and then allowing the representation to interact with virtual objects (including representations of other users) within the three-dimensional scene in response to changes in the position, pose, and / or gesture of the real user. The final resulting three-dimensional scene can be viewed using a display device.

[0040] The present disclosure also generally provides systems and methods for generating multi-layered scenes. More specifically, the present disclosure provides systems and methods for populating or inserting a data feed (e.g., one or more images, video streams, live video streams) such as a video capturing an actor (a person digitally captured or captured by an imaging device) into a multi-layered scene to generate a video (e.g., a composite video) approximating a three-dimensional space with depth by displaying the actor in front of one or more layers of the multi-layered scene and behind other layers of the multi-layered scene. The present disclosure describes systems and methods that can be implemented upon receiving multiple media inputs (such as images, videos, and the like) to generate a displayable scene, where the displayable scene includes multiple layers of graphical or visual data. The final output video of the multi-layered scene can be viewed using a display device.

[0041] In particular, the system displays, populates, and / or positions data feeds at various degrees of depth (i.e., each of the layers of the multi-layer scene, including in front of, behind, between one or more layers of the multi-layer scene) such that the data feeds, such as data feeds representing actors, coexist, are positioned, or are displayed on the display without obstructing at least a portion or one or more of the layers of the multi-layer scene. In addition, the system enables the populated data feeds to interact with at least a portion of one or more layers. For example, an actor or user can be captured as a data feed, and the actor's movements can cause virtual interactions with objects presented in or by another layer of the multi-layer scene into which the actor's image or video feed is populated. Such a merge of the multi-layer scene and the data feeds is displayed as an output video or image on the display. As a non-limiting example, an object can appear as a solid object, an opaque object, a semi-transparent object, a transparent object, or an object displayed by the display with any various degrees of transparency or semi-transparency.

[0042] In the detailed description of the present disclosure, examples of the present system and method are primarily presented to display data feeds of actors (i.e., data feeds representing actors, such as users, captured by imaging devices, such as video streams of users) in a multi-layered scene. It should be noted that some details in the drawings are simplified and depicted for ease of understanding rather than maintaining strict architectural accuracy, detail, and scale. In particular, video streams of people (i.e., users, actors, etc.) are depicted in simplified drawings or representations instead of actual photographs, videos, or other types of data recordings of the recorded subjects or objects. This is for clarity and ease of illustrating the functionality of the present disclosure. The visual representations contemplated herein are primarily directed to actual recordings of people, objects, or other visible things. Such simplified characterizations in the drawings are representative, rather than literal demonstrations of the present disclosure in actual implementations.

[0043] The words "user" and "actor" are used interchangeably in this specification and claims.

[0044] The terms "actual" or "physical" are intended to mean "non-virtual" or "tangible" or "in the real world" as opposed to "virtual". The words "add" and "position" are used interchangeably herein with respect to a user, actor, or actor layer. With respect to "adding to" or "positioning within" a voxel-based three-dimensional scene, "adding" or "positioning" is understood to include changing data or values ​​for a set of voxels (three-dimensional pixels in xyz space) within the three-dimensional scene.

[0045] The term "multi-layer" is also understood to mean "multi-depth" and "three-dimensional."

[0046] "Pose" and "gesture" are very closely related as complementary features of the human body. "Pose estimation or recognition" can be understood as the recognition or identification of the shape or posture of the human body, including the position and orientation of both larger body parts such as arms and legs, and finer details such as fingers and facial points and eyes. Gesture recognition involves further interpretation of the pose information. For example, the pose information of a hand showing an index finger extended and other fingers curled can be interpreted by gesture recognition as a typical pointing gesture. When "pose estimation" or "pose recognition" or the like is mentioned, it is understood that it may be followed by a step of "gesture recognition". Similarly, when "gesture recognition" is mentioned, it is understood that it is often preceded by a step of "pose estimation". Well-known pose estimation techniques use convolutional neural networks and heat maps to estimate the pose of a user.

[0047] Referring now to FIG. 1, a system for displaying data in a multi-layered scene, hereinafter also referred to as data processing system 100, includes an imaging device 102, a data acquisition device 104, a storage device 106, a controller 108, and a display device 110. Such system component units represent software units executing on a processor or processing circuit of a computer system, one or more hardware components performing the methods described herein, or any combination of the two. Although FIG. 1 shows system component units grouped as a single system, such association is not limited to forming a single physical hardware unit (such as individual units formed on a single computing device), but also represents a functional association of system units that collectively perform the methods described herein. The imaging device 102 captures data feeds, such as image(s) and video(s). In some examples, the imaging device 102 records or captures images or videos of an actor (or user). The data acquisition device 104 controls and receives data captured by the imaging device 102. The controller 108 processes and implements the methods described herein using the storage device 106. The display device 110 outputs the resulting data after processing as the output video.

[0048] 2 and 3 illustrate how various data types (e.g., data feed 204 and multi-layered (3D) input data 202) are processed by the system to display an output video 218 including a multi-layered scene. Although the process is described sequentially, different input data may be processed individually or in parallel, provided that no one of the input data is dependent on the process or progress at which the other data feed is processed. As illustrated in FIG. 2, the system (which may also be referred to as data processing system 100) receives data feed 204 and multi-layered (3D) input data 202 via data acquisition device 104 of data processing system 100. Data feed 204, such as video stream 304, is acquired using imaging device 102. During data acquisition phase 220, input data (e.g., multi-layered (3D) input data 202 and composite input data 302) are acquired by data acquisition device 104. The data controller 108 identifies various types of input data and data captured by the imaging device 102, as well as its scene layers 202, and extracts and merges them during the data control phase 222. Finally, the display device 110 presents the rendered multi-layered scene 214 as the output video 218. In some examples, the input data is three-dimensional input data 202 contained in a computer memory or data storage device. In some examples, the input data is multi-layered input data 202 including at least a set of separate but coordinated input scene layers 202. The multi-layered input data 202 with coordinated input scene layers can be input data time-synchronized with multiple layers (i.e., multi-layered input data). For example, the multi-layered input data includes a time-synchronized background layer 210 video feed and a subject layer 208 video feed. In some examples, the scene layers of the multi-layered input data 202 can be coordinated by association of various objects captured in the input data. For example, the background layer 210 can include an object that is associated or correlated with another object captured by the subject layer 208, and vice versa.

[0049] In some examples, the input data is composite input data 302 that includes a variety of data, such as fixed images, slides, videos, and the like. As shown in Figure 3, the composite input data 302 is processed to extract background and subject layers (i.e., layer extraction 306). Thus, layer extraction 306 may be necessary when the input data does not include layered data, such as multi-layer input data 202.

[0050] In some embodiments, as shown in FIG. 2, the system may also receive multi-layer input data 202 including multiple scene layers, such as a background layer 210 and a subject layer 208. In this example, the multi-layer input data 202 includes two layers. The multi-layer input data 202 may be a collection of the background layer 210 and the subject layer 208. The multiple such layers enhance the three-dimensional effect and details of the multi-layer (three-dimensional) input data 202 scene. Similarly, the multi-layer input data 202 may be two separate input data, the background layer 210 and the subject layer 208.

[0051] From a data feed 204, such as a video stream 304, the system extracts images (or a collection of images, such as a video clip) capturing one or more actors (i.e., extracting actors 206). The data feeds 204 (such as video streams 304) captured by these imagers contain at least a portion of an actor, which is processed by the system to isolate portions of the data feed that contain at least a portion of the actor. In some examples, actor extraction or isolation or segmentation can use chromakey green screen, virtual green screen techniques, skeletal / human pose recognition / estimation, neural networks trained to isolate actors in images or video streams, and the like. Various known image processing and video processing techniques can be applied to extract or isolate actors from a data feed 204 or video stream 304. The extracted actor representations are then added to an actor layer 212, which consists of non-transparent areas that contain isolated actors and transparent areas that do not contain isolated actors. In some embodiments, the extracted actor representations are added into the three-dimensional virtual scene by modifying data associated with appropriate three-dimensional locations or voxels (three-dimensional or "volumetric" pixels) in the three-dimensional virtual scene. See, for example, three-dimensional scene 1302 in Figures 13A and 13B.

[0052] Once the actor layer 212 is identified from the data feed 204, such as the video stream 304, and multiple layers 202 are identified from the multi-layer input data 202 or the composite input data 302, the actor layer 212 including the isolated actor 206 is added before the background layer 210 and after the subject layer 208, thereby generating a scene 214 including three layers (i.e., a multi-layer scene). In other words, the actor layer 212 is added between the background layer 210 and the subject layer 208. Such a collection of at least three layers, the background layer 210, the actor layer 212, and the subject layer 208, represents visual or displayable data to form the multi-layer scene 214. A multi-layer scene refers to one or more layered visual representations of data, such as image(s) and video(s). The background layer 210, the actor layer 212, and the subject layer 208 are rendered / merged (216), e.g., flattened, to render an output video 218, which is then displayed (224) using the display device 110. In some examples, the flattening of layers in the scene 214 may be performed such that any region of the resulting output video 218 or image is composed of video or image from the foreground layer of that same region where that foreground layer is not transparent. If the foreground layer of that region is semi-transparent, that region of the output video 218 or image is composed of a visual combination (e.g., the colors of the layers are averaged, blended, or otherwise visually combined) of the foreground layer and the recursively flattened layer(s) behind it. The display device 110 may be a monitor, a projector, a mobile phone display, a three-dimensional display, augmented reality glasses, a virtual reality headset, a display presented by another wearable, or any other device that displays an image(s) or video(s) for visual presentation.

[0053] In some embodiments, as illustrated in FIG. 13A and FIG. 13B, the system may receive three-dimensional input data 1302 (also called a "three-dimensional scene") that includes objects (including interacting or passive and moving or fixed, including fixed passive features of the environment such as slopes or hills) at multiple positions in three-dimensional space, such as an object 1310 further back and an object 1308 further forward. In this example, the three-dimensional input data 1302 includes objects at two different positions in three-dimensional space. The three-dimensional input data 1302 may be a collection of objects 1308 and 1310 arranged in three-dimensional space. Such objects at multiple positions increase the perceived richness and detail of the three-dimensional input data 1302. Similarly, the three-dimensional input data 1302 may also be sparsely populated with very few objects. As another characteristic, the three-dimensional objects 1308 and 1310 may have different levels of transparency and opacity, which will be taken into account during rendering (1316). The three-dimensional input data 1302 may be stored in the storage device 106. The components of the data processing system 100 communicate directly or indirectly via a variety of communication protocols.

[0054] From a data feed 204, such as a video stream 304, the system extracts (206) an image (or a collection of images, such as a video clip) capturing one or more physical actors 1301. The data feed 204 (such as the video stream 304) captured by these imagers contains at least a portion of the actor, which is processed by the system to isolate portions of the data feed that contain at least a portion of the actor. In some examples, actor extraction or isolation or segmentation can use chromakey green screen, virtual green screen techniques, skeletal / human pose recognition / estimation, neural networks trained to isolate actors in images or video streams, and the like. A variety of known image and video processing techniques can be applied to extract or isolate actors from a data feed 204 or video stream 304.

[0055] Once the extracted actor representation 1312 has been identified from a data feed 204, such as a video stream 304, and the three-dimensional scene 1302 is available, the extracted actor representation 1312 is then positioned 1313 within the three-dimensional scene 1302. Positioning is performed by modifying the data or values ​​of the appropriate voxels (or three-dimensional pixels) within the three-dimensional space / scene, thereby generating a three-dimensional scene / space 1314 that contains the various objects and actor representations. In other words, the actor representation 1312 is positioned between objects of the three-dimensional scene, such as 1308 and 1310.

[0056] The positioning and updating step 1313 may optionally (as indicated by the dotted lines in the figure) involve taking as input additional actor information, including but not limited to position / depth information 1309 (information regarding the actor's physical location relative to the camera, including depth or distance and lateral and vertical position relative to the camera, in other words x, y, z coordinates), identified from the video stream 304, and pose and / or gesture information 1311 regarding the actor's physical poses and / or gestures. The position / depth information 1309 may be determined (1303) based on the user captured in the video stream 304. The pose and gesture information 1311 may be identified (1304) based on the user captured in the video stream 304.

[0057] As part of the update, in step 1313, the system may also take and utilize additional actor information as input to update the scene 1302 based on the position, pose, or gesture of the actor 1301. For example, by utilizing the position / depth information 1309 and / or the pose & gesture information 1311, the positioning and updating step 1313 involves determining whether and how the virtual actor contacts an interacting object in the three-dimensional scene to trigger a corresponding action. Once an action is triggered, in step 1313, the system updates the three-dimensional scene accordingly to reflect the action (e.g., visual state change, etc.). For example, in one embodiment, if the virtual position and pose of the virtual actor indicates that the virtual actor is in contact in the three-dimensional virtual space with a button that changes color when pressed, in step 1313, the system considers the button to have been pressed and triggers and executes the corresponding action to update the virtual three-dimensional space or scene to reflect that the button has changed color. As another example, in another embodiment, if a physical actor is determined to be making a pinching gesture with his / her fingers as if holding a pen, then in step 1313, the system maps the physical location of the physical actor to the corresponding virtual location of the virtual actor in the virtual three-dimensional space, whereby recognition of the pinching gesture (i.e., a pen gesture) in step 1313 causes the system to update the virtual three-dimensional space to show the virtual location of the virtual actor's hand (or the virtual pen held in that hand) writing or drawing in the virtual three-dimensional space. Thus, some visually perceptible change of the user 1301, such as the additional actor information described above, is utilized by the system to effect the positioning or update as described by step 1313. In other words, the user can control the actor representation 1312 by using the output video 218 provided by the system and introducing visually perceptible changes to the system via the video stream 304 data input.

[0058] In some embodiments, the imaging device or camera is positioned to capture the user's entire body to provide for the user's entire body to be included within the video stream.

[0059] In some embodiments, an image capture device or camera is positioned to capture at least a portion of the user himself / herself to provide a partial view of the user that is included within the video stream.

[0060] A three-dimensional scene refers to a three-dimensional space with one or more objects at multiple depths or distances from the viewpoint. The further back objects 1310, the actor representations 1312, and the closer forward objects 1308 are rendered (step 1316), such as by 2D projection (e.g., orthogonal, oblique, perspective, etc.) to render an output video 218, which is then displayed (224) using the display device 110. The display device 110 may be a monitor, a projector, a display presented by augmented reality glasses, virtual reality headset, other wearable, or any other device that displays image(s) or video(s) for visual presentation. In some embodiments, such as when the display device 110 is capable of receiving three-dimensional content, the rendering (1316) may leave the generated three-dimensional scene 1316 substantially unchanged as it passes it directly to the display device 110 where it is displayed.

[0061] Computer program instructions may be stored in a computer-readable medium that can direct the system, its components, or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium constitute a product including instructions and processes that implement the functions / actions / steps specified in the flow charts and / or block diagrams provided herein. These computer program instructions may be provided to a processor of the system, such that the instructions create means for implementing the functions / actions specified in a block or blocks of the flow charts and / or block diagrams. In the following description, reference is made to the accompanying drawings, which are illustrations of embodiments in which the disclosed system may be implemented. The system of the present disclosure is specifically configured to perform the methods contemplated in the present disclosure. However, it should be understood that those skilled in the art may develop other structural and functional modifications without departing from the novelty and scope of the present disclosure.

[0062] In some embodiments, as shown in Figure 16, the system acts as a user control mechanism by displaying to the physical user a real-time representation of the virtual user as seen in the virtual world, allowing the physical user to control his / her virtual self (i.e., the virtual user or user representation generated on the display) within the virtual three-dimensional world and also to interact with objects within that virtual world. To the extent that the user can see an object with which the user wishes to interact, the user can move his / her physical body (or make gestures to produce a similar effect) in the physical world, thereby enabling the virtual user representation of his / herself to similarly move its body to take such action within the virtual world.

[0063] As shown in FIG. 16, in some embodiments, the user control mechanism operates as follows:

[0064] In step 1602, the system receives video data, such as frames of video (containing images of the actual user), from an imaging device. The imaging device can be a 2D or 3D camera or multiple cameras. Camera types include, but are not limited to, webcams on laptops and cameras on cell phones (including "front-facing" or "selfie" cameras on cell phones).

[0065] At step 1604, the system separates the user representation from the video data received in step 1602. In some embodiments, the separation is performed by body segmentation or "selfie segmentation" as commonly used by videoconferencing software to separate or segment an image of a user from an image of the actual environment around them in order to remove it and replace it with a more favorable background.

[0066] Once the image of the user is isolated, some embodiments of the system may use all or a portion of this image as a user representation (also referred to as a "virtual user" or "virtual user representation"). In some embodiments, the virtual user representation may be a completely "real-world" image of a physical user. In other embodiments, the virtual user representation may be a partially "real-world" image (e.g., just the face), but other parts of the virtual user may be virtual (such as an avatar's body) or augmented (such as wearing virtual clothing or holding virtual objects). Also, in some embodiments, the virtual user may be completely virtual, such as a virtual avatar (including, but not limited to, an avatarized representation of a physical user). In some embodiments, the user representation may be flat (like a papercut) or have a certain thickness. In other embodiments, it may be rounded or more similar to a real body. In some embodiments, a user representation, such as an avatarized user representation, may be based on multiple captured images of the user, either with an explicit training step (such as the user being asked to stand in a "T-pose" in front of the camera and perhaps turn around), or by capturing multiple images during a natural pose of the user using the system.

[0067] After receiving the user representation in step 1604, the system then adds the user representation or virtual user to the three-dimensional scene in step 1606. A three-dimensional scene, also referred to as a "virtual world / scene / space" or "three-dimensional space," is a three-dimensional space that may also include various features, such as, for example, objects, environmental features (hills, mountains, trees, etc.), other virtual users, etc. A three-dimensional scene / space may be a three-dimensional representation made up of voxels (i.e., three-dimensional or "volumetric" pixels) and / or vectors with xyz coordinate positions, and may be held in computer memory. A three-dimensional scene may be static or dynamic. In some embodiments, it may be created by a virtual reality application, such as a game.

[0068] In some embodiments, step 1606 may involve adding the virtual user representation to the three-dimensional scene on a "frame-by-frame" basis, thereby enabling the physical user to control the virtual user by the real, physical user's own frame-by-frame movements or actions. However, as illustrated in Figures 20B and 20C, in some embodiments, rather than adding and re-adding the virtual user representation again each and every frame, step 1606 may include step 1606a of instantiating and adding the virtual user representation once, initially as an object in the three-dimensional scene, and then updating the virtual user representation object in the three-dimensional scene in a subsequent iteration (step 1606b), which may correspond to a subsequent "frame." In this manner, the virtual user representation object is added, instantiated, and updated only once (or less frequently than per frame), rather than being re-instantiated and re-added every frame (as frequently as per frame).

[0069] Now, upon receiving the three-dimensional scene including the virtual user or user representation (such as by a pointer to computer memory where it is stored), the system displays the three-dimensional scene back to the physical or real user in step 1608. This real-time feedback to the user allows them to maintain control of their virtual self within the virtual three-dimensional world. In some embodiments, the display is via a two-dimensional display, such as a 2D monitor, where the user can see an image of himself in the virtual world (when displayed on a 2D monitor, the three-dimensional scene (including the user representation) is first rendered by typical means of two-dimensional projection, such as orthogonal, oblique, perspective, etc.). In other embodiments, the display may be via a three-dimensional display device.

[0070] After step 1608, the process loops back to beginning a new iteration at step 1602 to continue to provide control and visual feedback to the user for controlling his or her virtual self within the three-dimensional virtual world.

[0071] FIG. 21 shows an example of applying the steps described in FIG. 16. The user is physically standing in front of the camera in his / her real room. In step 1602, the system receives the first frame of the video from the camera. Upon receiving the video, in step 1604, the system separates the user's image from the video by body segmentation and starts using this image as part of the user representation. More specifically, in this embodiment illustrated in FIG. 21, in step 1604, the system uses three copies of the user's body image (with fixed offsets in position and scale) as the user representation. Given an "empty" (i.e., no virtual user representation) three-dimensional scene or space of the virtual dance hall, the system adds the user representation (e.g., for this particular example, with three copies of the user's current body image from the most recent frame of the video from the camera. For clarity, any number of copies of the user's current body image can be used as the user representation, including a single copy) to the three-dimensional scene of the virtual dance hall in step 1606. In step 1608, the system renders the three-dimensional scene into a 2D image, which is then displayed on a display device, such as a monitor screen, for the user to view. Once the user knows how her current virtual user representation looks in the virtual world, she very slightly adapts her body position and pose. As she does so, the process loop starts over again at step 1608, which receives the next frame of video with the adapted pose and position of the user (i.e., a visually perceptible change in the user). Based on this latest frame of video as input, the process again proceeds with step 1604 of isolating the user image and creating a user representation (with three images of the user's body), step 1606 of adding the latest user representation to the empty virtual scene, and step 1608 of displaying to the user an updated 2D rendering that reflects how the virtual user currently looks in the virtual three-dimensional space.

[0072] 17, some embodiments may additionally include a step 1710 for updating the three-dimensional scene. In some embodiments, step 1710 involves updating the three-dimensional scene separately from the virtual user representation. In some embodiments, step 1710 involves updating the three-dimensional scene, including updating or modifying the virtual user representation.

[0073] At step 1710, the system enables the three-dimensional scene to be dynamic. In some embodiments, at step 1710, the system updates the scene based on the passage of time or randomly over an extended period of time. For example, without step 1710 in the process, virtual users could be added to a static ballroom virtual space that never changes. With step 1710 as part of the process, the virtual ballroom could have a rotating mirror ball and a floor made of tiles that pulsate with different colors. In some embodiments, the three-dimensional scene could be updated based on external inputs, such as outdoor weather, the stock market, or signals sent from a keyboard or game controller.

[0074] As shown in Figure 18A, some embodiments may include a step 1812 of recognizing the physical location of the user. The location may include, but is not limited to, one or more of lateral or "x" location, vertical or "y" location, depth or "z" location, and rotation or orientation about all three axes. In some embodiments, the location is determined with respect to a camera or imaging device. In other embodiments, the location is determined with respect to a room or physical space within which the user is located. The location may be determined by techniques for body tracking, depth estimation, etc.

[0075] With regard to depth estimation, in some embodiments, a depth-sensing imager (including, but not limited to, a time-of-flight (TOF) camera / sensor, stereoscopic camera, multiple cameras used together, 2D cameras supplemented with lidar, cameras on cell phones equipped with depth sensing technology, and the like) may be used to extract depth information from a scene captured by the imager. In other embodiments, an imager, such as a two-dimensional camera, may be used in conjunction with recognition of depth cues in the scene to extract depth information from the scene. Examples of depth cues include the distance between detectable features on the human body, such as the distance between the eyes or various joints of the body. Such features appear larger the closer they are to the camera and smaller the further they are from the camera, and thus may be useful in estimating depth or "z" distance from the user.

[0076] After generating the position information in step 1812, the system may utilize this information in step 1606 to determine where exactly to position the user representation or virtual user when adding it to the three-dimensional scene. In one embodiment, a physical actor shifting one meter to the left (as determined in step 1812) may cause the system to add the virtual user in step 1606 to a position shifted to the left in the virtual space. However, in another embodiment, the same leftward shift by the physical user may cause the system in step 1606 to add the virtual user representation to the right (instead of the left), exhibiting a "mirror" effect, or in another embodiment, to add the user to an entirely different region in three-dimensional space, which may exhibit a "wormhole" or "teleport" effect.

[0077] The changes in position and orientation effected by the system in step 1812 may be referred to as "shifts." In some embodiments, step 1606 may involve "mapping" a certain type of shift of the real, physical user to a certain corresponding type of shift by the virtual user representation. In some embodiments, a left-to-right shift by the user may result in a left-to-right shift by the virtual user representation, while in other embodiments it may be the other way around. In some embodiments, a user shifting forward toward the camera or imaging device may shift the virtual user representation forward, and vice versa. Alternatively, in other embodiments, a user shift toward the camera (i.e., a change in depth or "z" distance) may be completely ignored (i.e., not mapped to any corresponding virtual shift), thereby allowing the virtual user representation to shift only left and right, and not forward or backward. Changes in orientation or rotation may similarly be mapped in various ways or ignored entirely. In this manner, the mapping of positions and / or shifts between the real user's physical world and the virtual world of the user representation may be as varied as the particular embodiment being implemented.

[0078] Figure 22 shows an example embodiment of a user playing a virtual reality game. In the following description of Figure 22, the steps presented in Figure 19 correlate with the specific example of Figure 22. For example, on the display of his laptop or mobile phone, the user sees his virtual self in a virtual world. Seeing his virtual self on the screen, as seen in the four snapshots shown in Figure 22, the user calibrates and controls the movement of his virtual self frame by frame to carefully step from a virtual stone on the left to another virtual stone on the right without missing and falling into the water.

[0079] For the first snapshot 2201: the user sees on the display that his virtual self needs to move to the adjacent stone. The real user physically extends his first leg. In step 1602, the system receives a first frame of video (captured by the front "selfie" camera of the user's laptop or cell phone) showing the real user in her real room. In step 1604, the system separates or segments the user image from the image of her in her room and begins to use the user image as the user representation. In step 1812, the system determines the physical location, including depth, of the real user in her real room. Having mapped that physical location to a corresponding virtual location in the virtual scene, in step 1606, the system then adds her image (i.e., the user representation) just above the left stone in the virtual three-dimensional scene. The system then renders the three-dimensional scene from the 3D representation to a 2d image for display on the user's laptop or cell phone screen in step 1608. The next iteration of the process loop is then ready to resume in step 1602.

[0080] For the second snapshot, once the user receives clear feedback via the display device in the 2D image of the virtual user extending her foot to a location to the right (as shown in the first snapshot 2201), the real user begins to lower her physical foot, so that her virtual user now has her feet firmly planted on both stones (as shown in the second snapshot 2202). This occurs as the real user lowers her physical foot, and the system captures that image in step 1602, separates and segments that image from the rest of her room in step 1604, determines her actual location, including depth, in step 1612, adds the virtual user representation (her image) in the appropriate position (directly above the gap between the two stones) in step 1606, and renders the updated three-dimensional scene in step 1608 and displays it as the second snapshot 2202. The next iteration of the process loop is then ready to resume in step 1602. This is an example of a user controlling his virtual self by utilizing the displayed scene as feedback. As the user moves, the system mirrors the corresponding movements in the virtual self depicted on the display.

[0081] With clear feedback that she appears to be successfully crossing from one step to the next, the process loop can continue to iterate and repeat (as described in Figures 16-19) over the next two snapshots 2203 and 2204, where the virtual user is seen to first shift her weight (2203) to the right stone (as controlled by the physical user), then place both feet on the right stone (2204). This is an example of a user controlling his virtual self by utilizing the displayed scene as feedback. As the user moves, the system reflects the corresponding movements in the virtual self displayed on the display.

[0082] As shown in Figure 18b, some embodiments may include a step 1814 of recognizing a pose or gesture of a user from their image from the video data. Pose estimation can be performed by a variety of well-documented techniques. Gesture recognition, which is also well-documented, can be used to interpret the pose information to recognize various gestures of a subject.

[0083] After recognizing the pose and / or gesture information in step 1814, the system may use this information to determine where exactly to position the user representation or virtual user when adding it to the three-dimensional scene in step 1606. For example, a user alternatingly bending each knee may be recognized by the system in step 1814 as making a walking gesture with his / her legs. Accordingly, in some embodiments, for each successive piece of video data (i.e., video frame) being processed by the iterative loop of the process, the system may add a virtual user in step 1606 that is shifted forward from its previous virtual position for each iteration of the loop (i.e., for each video frame), thereby shifting its position forward as if the virtual user were walking in the virtual three-dimensional scene. In another embodiment, a user that is recognized by the system in step 1814 as making a jump-like gesture may be added upward from its previous position for the next few successive iterations of the loop to make it appear that the virtual user is jumping upward in the virtual three-dimensional scene. In another embodiment, a user's facial gesture or expression that is recognized by the system in step 1814 as a frown may cause the system to generate the user representation in step 1604 with a bluish tint, while the system recognizing a smile in step 1814 may cause the system to generate the user representation in step 1604 with a yellowish tint.

[0084] In another embodiment, as shown in FIG. 20A, a user 1301 with arms outstretched (2002) may be recognized by the system in step 1814 as assuming a superhero flying pose using the video stream 304 captured by the imaging device 102. In response, the system in step 1604 outputs a user representation 1312, for example, an actual image of the user wearing an augmented virtual superhero cape. Then, over successive iterations of the process loop, a first iteration 1312a, a second iteration 1312b, and a third iteration 1312c, the system in step 1606 will iteratively add virtual user representations, incrementally increasing upward and forward each pass of the loop (1312a 1312b 1312c) to move the virtual user representation 1312 as if flying through the three-dimensional scene 1302. The three-dimensional scene 1302 may include three-dimensional scene objects such as 1308 and 1310. This is an illustrative example of a user's poses / gestures and their detection by the present system which triggers or causes corresponding programmed or predefined actions.

[0085] As shown in FIG. 19, various optional steps may also be combined. In some embodiments, as part of updating the three-dimensional scene, step 1710, the system continuously or continuously monitors interactions or collisions between all virtual users and all objects in the three-dimensional scene, and also maintains / updates the state of each of these. For some objects, such as the spinning mirror ball from the example above, step 1710 may include the system continuously updating to keep it moving. For some passive objects, such as walls, step 1710 may include blocking virtual users from being able to pass through it. For interacting objects, step 1710 may include monitoring and detecting interactions between virtual users (or other objects) and the object to trigger and perform any necessary actions.

[0086] In some embodiments, the process step of updating the three-dimensional scene (step 1710) may be implemented as shown in FIG. 28. The system detects interactions between virtual users (or user representations) and interacting objects (step 2802), interactions between multiple virtual users (step 2808), interactions between objects (step 2810), and external triggers (step 2812), such as the passage of time, weather, button presses, game controller input, etc. When the system detects such interactions and / or triggers actions associated with interacting objects and then performs those triggered actions, the system updates the objects accordingly in step 2804 and updates the virtual users accordingly in step 2806. The virtual users (or user representations) are themselves also considered objects. Interactions between objects therefore include interactions between virtual users and objects. Similarly, it includes interactions between virtual users.

[0087] In some embodiments, step 1710 may include utilizing the position (including depth) and / or orientation data from step 1812 to update the three-dimensional scene. In step 1710, the system may do so by utilizing the physical user's position and / or the virtual user's virtual position to detect the triggering of an interaction object and then perform the associated action. In one embodiment, consider, for example, a situation where the virtual user representation is standing in front of a virtual automatic sliding door in a virtual three-dimensional scene. If the system detects in step 1812 that the real physical user is shifting forward in their physical space (i.e., a change in depth information), the system adds the virtual user to the three-dimensional scene in step 1606, one step forward from where they were previously. Upon detecting that the virtual user is now directly in front of the virtual automatic sliding door, the system in step 1710 determines the sliding door interaction object that is triggered and therefore performs the associated action for that door to slide open. Thus, the system in step 1710 updates the three-dimensional scene by sliding the virtual door open.

[0088] In other embodiments, the system may utilize the pose and / or gesture information provided by the system in step 1814 to update the three-dimensional scene in step 1710. For example, in one embodiment of a virtual whiteboard application, a long left-to-right swipe gesture with the palm may be recognized by the system (across consecutive video frames) in step 1814 as a "clear all" gesture. Once the system provides information of the occurrence of a "clear all" gesture, the system begins updating the three-dimensional scene in step 1710 by removing any virtual writing or drawing that may have been made. This removal is done by returning voxels involved in the writing ("voxels" are referred to as three-dimensional or "volumetric" pixels) to their original "unwritten" state.

[0089] Continuing with the virtual whiteboard example, consider the case where the system determines, at step 1814, that the physical user is making a pinching gesture with their fingers as if they were holding a pen. In this example scenario, the system also determines, at step 1812, the physical location of the physical user within its physical space. Once steps 1812 and 1814 have been performed by the system, the system maps that physical location, at step 1606, to a virtual location in the three-dimensional virtual space to which it adds a virtual user representation. The system's recognition of the pinching gesture (i.e., a pen gesture) at step 1814 causes the system, at step 1710, to update the virtual three-dimensional space to appear to write or draw (i.e., by changing the color of the appropriate voxels) at the virtual location of the virtual user's hand (or virtual pen being held in the hand) in the virtual three-dimensional space.

[0090] Figure 29 shows a process flow for one embodiment of a virtual whiteboard system 2912. At step 2902, the system recognizes a pen-holding gesture 2908 (such as pinched fingers) being made by a user. At step 2904, the system updates the scene by drawing, i.e., changing the color of voxels (or pixels in the 2D case) at the 3D position of the fingers 2910 of the user representation. At step 2906, the system displays the updated scene. The process of Figure 29 can be repeated.

[0091] As another example, if the virtual user's virtual position and pose indicate that the virtual user is in contact with a button within the three-dimensional virtual space, the system detects the button being pressed and then triggers and executes a corresponding action. Such a corresponding action may, for example, change the color of the button in response to it being pressed by the actor (i.e., by recognizing the user's position and pose that indicates contact with the virtual button). The virtual three-dimensional space or scene is updated to change the color of the button.

[0092] In some embodiments, the system may update the virtual user itself, step 1710. In one embodiment, a virtual user that strays off course and gets too close to a virtual mine (interaction object) may trigger a corresponding programmed action, which may be where interacting with it causes it to explode within the virtual three-dimensional world. Such an explosion may cause damage to other objects in its vicinity. To represent an explosion effect, in this example, the system, step 1710, updates the appearance and other characteristics / attributes (e.g., geometry, rotation, health, armor, speed, strength, etc.) of nearby objects in the three-dimensional scene. One such affected object may also be the virtual user representation itself. In such a case, the system, step 1710, may present an update to the virtual user representation 1312 by animating the virtual user representation hurling backwards from the explosion location, placing the virtual user in a reclining position (even though the real, physical user is still standing). Finally, in the next iteration of this presentation of a series of such three-dimensional scenes, the system may update the virtual user to stand up again, at step 1710, but the system may update its representation to appear bloody or injured, at step 1710. Moreover, perhaps one arm of the virtual user representation is no longer movable (whatever the pose of the real user). The system also changes the characteristics of the virtual user representation, at step 1710, to slow down the speed at which it can move, because it is now injured.

[0093] FIG. 23 shows an example embodiment of a user using a gesture to grab and move an interaction object 1208, in this example a candy as part of a game being played in a virtual scene. In a first snapshot 2301, the user approaches the yellow rectangular candy (interaction object 1208) with an open hand. The system captures a frame of a video feed of the real user in his physical room from an imaging device, such as a cell phone camera. The system then isolates or segments from that frame a user image that it begins to use as a user representation 1312. The system recognizes the user's hand pose or gesture as an open hand indicating the beginning of a "grab" gesture. The system adds the user representation 1312 (i.e., the user image with the open hand) to the scene. The system has not yet detected any object interaction, so it does not update the scene further. The system renders and displays the image 2301 as seen by the user on the screen of his cell phone. The process loop is then ready to begin its next iteration.

[0094] For the second snapshot 2302, the user sees on the display that his virtual self's open hand is behind the interaction object 1208, and the real user then begins to physically close his fist. In this example, the user can control his virtual representation 1312 by using the scene displayed on the display as overt feedback. The system captures a video frame of the real user in this pose. The system isolates the most recent image of the user's body (i.e., clenching the fist). The system recognizes the clenched fist (followed by the previous open hand) as indicating a "grab" gesture. The system adds the most recent user representation (i.e., the user image with the clenched fist) to the scene. Although the system detects the object interaction (i.e., grabbing), this embodiment does not require the candy to be redrawn as a result, so no further updates to the scene are made. The system renders and displays the image 2302 seen by the user on the display of his mobile phone. The process loop is ready to start its next iteration.

[0095] With respect to the third snapshot 2303, with feedback that his virtual self 1312 is grabbing the correct candy, the real user begins to move his fist to the right. The process proceeds as before. The system identifies the moving fist as a "drag" gesture. The system therefore updates the scene, this time to show that the interaction object 1208 is being dragged to the right. The system displays the scene to the user before the process loop begins the next iteration.

[0096] With respect to snapshot 2304, the real user opens his fist upon visual feedback that his virtual self is successfully dragging the virtual candy 1208 to its desired final location. Image 2304 is displayed showing a virtual user representation 1312 of the user releasing the candy (interaction object 1208) to its final desired location. This is an example of a user gesture / pose that triggers a programmed action of the interaction object 1208.

[0097] FIG. 24 shows another example of an embodiment in which a user can control and use one interaction object to trigger an action by another interaction object. In snapshot 2401, most of the body of the user representation is occluded by a handle (interaction object 1208) appearing in the center of the snapshot, but a raised arm can be seen in the action of throwing (or whipping) a virtual whip (interaction object 1208) held in its hand. The virtual whip is rendered and updated by the system with each iteration of the process loop. In snapshot 2402, the whip 1208 can be seen wrapped around the handle in the center of the snapshot. The system detected a contact interaction between the whip and the handle and updated the scene to show the whip wrapped around the handle. Then, in snapshot 2403, the user gestures to pull back on the whip 1208, which causes the virtual user 1312 to pull back on the virtual whip 1208, thereby pulling down the virtual handle. When the system recognized the gesture, the system updated the scene to reflect the whip pulling down the handle.

[0098] It should be noted that although the above description of Figures 16-19 discusses an embodiment involving a three-dimensional virtual space, some implementations implement the same or similar concepts utilizing a two-dimensional virtual space.

[0099] In some embodiments, the viewpoint of each virtual user is from a third-person perspective facing directly in front of the user. In some embodiments, the viewpoint (from which the three-dimensional scene is rendered) moves with the virtual user at a fixed distance in front of it. In other embodiments, the viewpoint is fixed. And in other embodiments, the distance and viewing angle of the viewpoint from the user varies depending on the situation.

[0100] In some embodiments, the perspective of each virtual user is from a third-person perspective from behind the user. In such cases, the user representation may use or display a rear (or "from behind") image or representation of the user. In some embodiments, such a representation may be extracted from a body cutoff and stored body representation similar to that described in the description of FIG. 15. In some embodiments, such as for applications or games that require a rear view of the user, some embodiments may utilize an explicit image capture step that involves asking the user to show their rear to an imaging device to capture a rear view to enable the system to have a rear view representation of the user. The rear view can then be used as the user representation seen by the user in a representation of his or her virtual self within the virtual world.

[0101] Multiple people and multiple users In some embodiments, multiple virtual users can be added to, participate in, and interact with the same virtual three-dimensional scene.

[0102] In some embodiments, multiple users may access the system remotely from different locations on a network. In such cases, some of the steps of the process loops of the example embodiments shown in Figures 16-19 may be performed on the cloud, on the user's local device, or a combination. Similarly, the steps of the process loops may be performed in a different order.

[0103] FIG. 25 shows an example of multiple users being added remotely to participate in a game in the same virtual space. In such an embodiment, the perspective of each virtual user is from a third-person perspective facing the virtual user. In such an embodiment, the virtual user representation is a mirror image of the real user, as people often find it more natural and intuitive to see a mirror image of themselves (e.g., if a real user moves their real left hand, they will see a virtual hand appearing to the left from their perspective move, even though it is technically the virtual user's right hand from the virtual user's perspective). With their own user representations incorporating their respective faces, each user can see the real expressions on each other's face, while also being able to communicate between nearby virtual users through voice via the system. This provides the users with a sense of co-presence. In addition, each user being able to see their own facial expressions also allows them to see how the other person sees them. Furthermore, such a sense of co-presence can be enhanced by displaying a virtual scene including both users on a display for each of the users.

[0104] In some embodiments, multiple users may access the system locally through the same imaging device, and in some embodiments, any user-specific steps described herein may be performed separately for each user.

[0105] To prevent irrelevant or unintended people (such as passersby or spectators appearing in the camera's field of view) from being treated as users of interest, the system may filter any people detected in the image. In some embodiments, a people culling method as shown in FIG. 26 is performed for each person detected in the video data (or video frame). The system detects person(s) appearing in the video frame at step 2602. For each person detected, the system compares the person to a list of users of interest at step 2604. In some embodiments, the list of users of interest includes at least a partial body image of the users of interest. The comparison may be made by well-known image comparison methods such as facial comparison, feature comparison, color histogram comparison, and the like. In some embodiments, the list of users of interest may be generated by an explicit user of interest registration process, such as may be prompted by a user interface, prior to the start of any use of the system. In other embodiments, the list of users of interest may be generated implicitly by taking images of the most recent and / or most frequently reoccurring user(s). Once the comparison is made in step 2604, the system determines in step 2606 whether the detected person is in fact the intended user. If so, the process proceeds to step 2608, whereby the detected person and his / her image are treated as a user, whereby a user representation of the person is allowed to be isolated and added to the virtual scene. If not, the process proceeds to step 2610, whereby the detected person and his / her image are treated as a user, whereby a user representation of the person is not allowed to be isolated and added to the virtual scene. After the culling process has been successfully performed for all detected people, no unintended people should be included in the virtual three-dimensional scene or appear in the display of the scene.

[0106] As shown in FIG. 27, some embodiments may apply person culling only if the number of detected people is greater than the number of expected people. The system detects some people in a video image in step 2702. The system compares that number to the number of expected users in step 2704. In some embodiments, the number of expected users may be predetermined by the application - for example, a one-player game would expect only one user. In other embodiments, the number of expected users may be explicitly received by a user interface presented to the user(s) at the start of the application. The system examines the results of step 2704 in step 2706 to determine whether the number of detected people is greater than the number of expected users. If yes, a person-by-person culling process is performed for each detected person as described in FIG. 26. If no, all detected people are allowed to be treated as users.

[0107] FIG. 4 shows example scenes 402, 404 from an output video that can be presented using a display device. In the scene 404, a subject layer is layered in front of the actor layer. An actor 406 is presented in the actor layer where a subject 408 of the subject layer is not occluded by the actor. A background layer 410 is presented after the actor layer. As shown, each of the scene layers can include transparent portions, so that the output video reproduces a rendered multi-layered scene displaying the actor 406 and the subject 408 of the subject layer on top of the background layer 410. In this example, the subject layer shows the subject 408, a graph positioned in front of the actor layer, which is shown with the actor 408 separated or extracted. The background layer 410 is positioned after all other layers. Scene 402 shows the scene before the adoption of the actor layer between the subject layer and the background layer 410. In this scene, the virtual actor 408 of the actor layer is not positioned between the subject layer and the background layer 410. In some examples, the background layer 410 and the subject layer are extracted from composite input data. In some examples, the background layer 410 and the subject layer are sourced from multi-layer input data.

[0108] In some examples, a three-dimensional effect (e.g., an output video having multiple layers presented with each layer having a different depth relative to one another) is achieved by identifying multiple layers from the composite input data. The composite input data can be analyzed to identify a background layer and a subject layer. Each frame of the output video is composed of at least one background layer behind the actor, at least one actor layer in front of the at least one background layer, and at least one subject layer in front of the at least one actor layer. In other words, the output video includes at least three separately layered scenes, i.e., multi-layered scenes. Similarly, the output video may include one or more scenes having three or more different layers (i.e., depth positions within the layers / layer stack) in which the layers are arranged or stacked to create a three-dimensional effect. Although the example embodiments of the present disclosure shown in the drawings primarily show a multi-layered scene with three layers (e.g., a background layer, an actor layer, and a subject layer) in the scene, multiple layers can comprise a multi-layered scene. For example, multiple layers can be placed between the background layer and the subject layer, thereby creating a three-dimensional effect or varying degrees of depth (i.e., different depth positions of the layers). In another example, a multi-layered scene can include multiple layers of background layers to create varying degrees of depth in the background, such as placing an image of a mountain (or a frame of a video) further after an image of a tree. Similarly, multiple layers can be introduced into the subject layer of a scene. In other words, a multi-layered scene can include multiple background layers, multiple actor layers, and multiple subject layers. The multiple layers that comprise a multi-layered scene enhances the visualization effect achieving a three-dimensional effect.

[0109] Slide Reconstruction Using Presentation Software. The layers of the multi-layer input data can be identified in a number of different ways. In one example, the presentation software (such as Powerpoint or Google Slides) can be configured so that a slide is a multiple of its typical size (corresponding to the number of layers it contains). As illustrated in FIG. 5, a slide is configured to be twice its typical size to include a subject layer 208 and a background layer 210 on the same slide. One half is used as the background layer 210 and the other half is used as the subject layer 208. Because this is done through the use of the presentation software itself, each half maintains all the functionality available through the presentation software (such as animation). The subject layer can utilize a particular color to indicate transparency within an area of ​​the layer, and the absence of a particular color within an area of ​​the layer indicates opacity or semi-transparency. The opaque or semi-transparent areas indicate the subject (i.e., the subject material of the subject) within the subject layer. In this example, a graph is shown as the subject material of the subject within the subject layer. And the background layer provides an aesthetic design element.

[0110] In some examples, using presentation software, a slide of typical size is used in conjunction with at least one pre-determined, automatically recognizable placeholder data location that indicates the placement of the actor layer. As shown in FIG. 6, a placeholder data location 606 is inserted into the slide and added before any content assigned to the background layer 610 and after any content assigned to the subject layer 608. The placeholder data location may also be sized and formed on the slide to indicate the area of ​​the actor layer to which the actor layer is to be added. Once the placeholder data location 606 is specified in the actor layer, the resulting output video has a multi-layered scene that includes the actor layer 612 positioned within the specified placeholder data location 606. On the display, the actor layer is displayed in place of the placeholder image. In some examples, the substitution of the actor layer in place of the placeholder image occurs during the positioning or stacking of layers prior to the rendering of the output video or image. In some examples, when an image or frame of the output video is rendered that includes a placeholder image and the system identifies a placeholder image, the visible portion of the placeholder image is then replaced with a corresponding portion of the actor layer 612.

[0111] Presentation software user interface. In some examples, the user interface of the presentation software (such as Powerpoint or Google Slides) is utilized, redesigned, or modified to include actor layers in the scene. Actor layers can be positioned on a slide and positioned in front of or behind subject layers or other subjects on other layers using options of the presentation software such as "Send backwards", "Send to back", "Bring forwards", and "Bring to front". In some examples, a user interface is implemented to allow a user to manipulate slides that have already been created using the presentation software (such as Powerpoint or Google Slides). The user interface displays subjects or objects on the slide in a way that makes it clear which are on which layer of the slide (i.e., which objects are in front of or behind others). The user interface then allows the user to select where between those layers they want to place at least one actor layer - i.e., in front of which layer, or after which layer, or between which layers. In some examples, automatic layering is used. Auto-layering involves using automated means to distinguish background layers from subject layers and insert actor layers between them.

[0112] Artificial intelligence and deep learning techniques can be employed for the system to identify the background layer and the subject layer. In some examples, artificial intelligence or deep learning can be employed to recognize the background layer and the subject layer. In some examples, this is accomplished by annotating a training set of data, such as images, videos, slides, screen shares, and the like (i.e., potential composite input data 302), to identify and extract which layers of data should be at the depth positions of which layers, such as the background layer and the subject layer. A neural net is then trained using the training set to create a classifier. The classifier is then applied to the composite input data to identify the scene layers, including the background layer and the subject layer. The actor layer can then be inserted between the two layers.

[0113] In another example, the system uses the presence or absence of features to determine which regions of the composite input data are assigned to which layers. For example, using composite input data of a balloon floating against a sky background and utilizing a feature classifier trained to identify balloons, regions of the composite input data where the balloon is present are assigned to the subject layer, while regions without the balloon (i.e., sky images) are assigned to the background layer. Similarly, other feature classifiers with non-binary outputs can be thresholded or otherwise utilized to assign regions of the input data (e.g., composite input data and multi-layer input data) to multiple layers.

[0114] An expert system approach can be employed by the system to identify the background layer and the subject layer. The expert system approach employs a set of rules to distinguish the background layer from the subject layer. In some examples, certain types of content may be prioritized to be assigned to the subject layer, which is further forward and therefore less likely to be occluded. As an example, text content may be prioritized to be assigned as the subject layer, so that it is positioned in front of the actor layer and is therefore not occluded by the actor layer. Other areas of the image can then be assigned to the background layer.

[0115] Similarly, multiple scene layers of the multi-layer input data can be ranked to determine an order or layer / depth position within the multi-layer scene based on the type of content. For example, certain text of certain keywords may be ranked or prioritized higher than others. Certain types of images, colors, shapes, data types, sizes, or other features may indicate a higher rank or priority than others.

[0116] In some examples, two different regions from the same layer input data (e.g., scenes, layers, images, video feeds, and the like) may be positioned at two different layer positions (i.e., depth positions when the layers are stacked). One of the two different regions may be positioned at the background layer, while the other region may be assigned to the subject layer. In some examples, application-specific knowledge is taken into account when assigning some regions of the input data (e.g., scenes, layers, images, video feeds, and the like) to the background layer, while other regions to the subject layer. For example, for a spreadsheet application (e.g., Excel or Google Sheets), the system may assign grid lines to the background layer, while numbers and text in spreadsheet cells are assigned to the subject layer. Thus, adding an actor layer between the background layer and the subject layer causes the actor to stand in front of the grid lines but not obscure the contents of the cell itself. As another example, the system deprioritizes regions of the input data that are specific to menu operations, such as toolbars at the top of the application window, so that they are assigned to the background layer. This allows the subject content in the subject layer to be presented without being obscured by areas that are typically not relevant for the presentation. Figure 4 shows an example where a graph (i.e., the subject content of the subject) is overlaid in front of an actor.

[0117] The location of a layer within a scene (depth location within a layer, e.g., subject layer and background layer) can be determined based on the location of content within or on the layer. As an example, content of data near the top of a scene can be de-prioritized and assigned to the background layer, while content near the middle or bottom of a scene can be prioritized to be in the subject layer. This allows an actor to stand without his head being occluded by content positioned above him in the scene, presenting content that is occluding him below, perhaps in the middle.

[0118] The system may employ user-assisted tiering, which involves providing a default "suggested" or "recommended" tiering utilizing the automatic tiering approach described above to give the user a starting point from which they can begin to further modify or customize the tiering using a user interface.

[0119] The layers of the multi-layered scene are presented positioned at various degrees of depth so that data feeds, such as video streams representing actors, coexist, are positioned, and / or are displayed on a display device without obscuring at least a portion or one or more of the layers of the multi-layered scene. In some examples, position information, such as depth information, is extracted from the input data and used to position the layers relative to each other. The depth information can be extracted from the input data or data feeds. Once the depth information associated with the scene is extracted, parts of the scene are positioned in different layers according to their distance (depth) from the camera. These positions are assigned to one of the defined layers in the scene. Both the background layer and the subject layer are instances of layers in the scene. An actor layer can be added between layers in the scene according to the distance of at least one actor from the camera (or similarly, an actor representation can be added at a particular depth in the three-dimensional space). Thus, the actors appear to move back and forth between the image parts displayed in the various layers of the scene or depths in the three-dimensional space. Actors can then interact with subjects or content located at any of these layers or depths.

[0120] In some examples, depth-sensing imaging devices (including, but not limited to, stereoscopic cameras, multiple cameras used together, 2D cameras supplemented with lidar, cameras on cell phones equipped with depth sensing technology, and the like) are used to extract depth information from a scene captured by the imaging device. In some examples, the imaging device is used together with recognition of depth cues in the scene to extract depth information from the scene. Such depth information can be identified from data feeds (e.g., video streams) capturing actors and multi-layered (three-dimensional) input data. Different portions or regions of a single scene can be assigned different depth levels that define the positioning of the portions or regions within layers or various depths of the three-dimensional scene.

[0121] The system extracts images of actors from video streams or data feeds using a variety of techniques, including but not limited to chromakey green screen, virtual green screen technology, skeletal / human pose recognition / estimation, and neural networks trained to isolate actors in images.

[0122] As an illustrative example of an embodiment utilizing a neural network, the neural network is trained using a training set of images containing annotated human (i.e., actor) faces and body parts. Once trained, the neural network is used as a classifier, whereby it can binary tag which regions of the image are most likely to be human faces or body parts. The identified regions are considered images of the actor that can be extracted from a video stream or data feed capturing the actor on a frame-by-frame basis.

[0123] The system described herein additionally employs input scaling and layering order definition. In some examples, two or more actors are displayed in the same scene with a shared space. The space can be a virtual space (such as a virtual dance hall) or an actual physical space such as one of the actors' living rooms captured by an imaging device. As an example, when displaying two or more actors in a shared space, scaling and normalization of the images capturing the actors results in the proportioning of the image sizes of the actors captured in the video stream or data feed so that they are the same size on the display. This can be done by applying feature recognition to the actors and normalizing the scale. If the shared space is the actor's actual physical space, that actor can be used as a reference for scaling and normalization of another actor presented on the display so that the two actors are sized proportionally to the same scale.

[0124] As mentioned above, in some embodiments, depth information is extracted from the data feed and used to position layers relative to one another according to their distance (depth) from the camera. In some examples, when combining two or more actors into a shared space, the distance between the imaging devices (e.g., cameras) of the actors can be used to adjust the scale of the actor as it is added into another actor's local space. More specifically, the actor is made larger as it moves closer to the camera and smaller as it moves away from the camera. Furthermore, the depth or layer position of the actor can be determined based on the dynamic distance between the imaging device and the actor. As the actor moves closer to the camera, the depth position of the layer capturing the actor can be assigned toward the front of the layer in the multi-layered scene (or, stated differently, the actor representation can be added further forward in the three-dimensional space or scene). Similarly, as the actor moves away from the camera, the depth or layer position capturing the actor can be assigned further back among the layers in the multi-layered scene (or, stated differently, the actor representation can be added further back in the three-dimensional space or scene).

[0125] Additional examples of layer sizing and dynamic positioning are presented below. In some examples, feature characteristics (such as size, shape, sharpness, brightness, orientation, presence / absence, or other characteristics) identified for objects (including objects on actors or other non-actor layers) by means such as feature recognition are used as depth cues to enable such dynamic positioning. In some examples, the relative size of the actors (as determined by a comparison of the sizes of features identified for the actors) is used as such depth cues. In one example, if actor A stands closer to a camera capturing actor A than actor B stands to a camera capturing actor B, actor A will be displayed at a relatively larger size and added at a more forward depth / layer position compared to the size and depth / layer position at which actor B is displayed on the display (e.g., output video). In another example, if actor A stands further away from the camera capturing actor A than actor B stands with respect to the camera capturing actor B, then actor A will appear relatively smaller and be added to a depth / layer position that is further away from the front compared to the size and layer position at which actor B appears on the display (e.g., output video). In some examples, such functionality allows two or more actors to take a virtual (i.e., two or more actors not present in the same physical space) selfie or photo together, where a first actor stands further away from the camera, allowing that actor to reach his arm behind the second actor's back, thereby occluding most of his arm, except for his hand, which reappears extending just past the second actor's shoulder.

[0126] FIG. 7 shows an example of depth-based layering / positioning for engagement between two or more actors. A background layer 708 is identified and positioned furthest away relative to the back of the layers. A first actor is extracted from a first data feed or a first video stream and identified by the system. A second actor is extracted from a second data feed or a second video stream and identified by the system. A depth / layer position between the first actor layer / representation and the second actor layer / representation is determined based on the distance between each of the actors and a camera (e.g., an image capture device). The scene 700 shows a first actor layer / representation 706 being added in front of a second actor layer / representation 704. In this case, the distance between the first actor 706 and the camera recording the first actor is less than the distance between the second actor 704 and the camera recording the second actor 704. The output video 702 shows a first actor layer 706 being added behind a second actor layer 704. In this case, the distance between the first actor 706 and the camera recording the first actor is greater than the distance between the second actor 704 and the camera recording the second actor. Thus, a portion of the first actor's arm is occluded by the second actor in this second scene 702. In another example, depth-based layering / positioning for an engagement between two or more actors can identify the distance each of the actors moves from its initial position to its next position. In other words, the depth-based layering / positioning can be based on the camera recording the distance traveled by each of the actors instead of the distance between the actors and their corresponding cameras.

[0127] As an alternative to automatic ordering of layers, a manual user interface may also be used. In some examples, through the use of triggers (including but not limited to remote control, voice or speech triggers, gestures, or key presses), an actor can move their own layer forward or backward to change the positioning of the layer within the scene. As an example, this approach would solve the selfie or photo layering problem mentioned above without having to move closer to or further away from the camera.

[0128] Just as the size or scale can change, so can the lateral position of an actor in the output video. In some instances, the lateral position of an actor in its actor layer is controlled by simply identifying the position of the actor in its own local space. As an actor moves laterally in its own local space in front of its own camera, the actor in the actor layer presented in the output video moves laterally as well. Alternatively, the position of an actor in its layer is not a direct representation of the actor's position in its own local space.

[0129] In some examples, the position of the actor in its actor layer of the multi-layered scene is fixed in place. While the actual actor depicted by the display device moves, the actor's position in the layer is fixed in place in the output. For example, the actor captured in the actor layer of the multi-layered scene in FIG. 6 may move around his camera's field of view, and the image of his face in each video frame is extracted using basic face tracking techniques and placed at a fixed position on the output video.

[0130] In some examples, a change in the lateral or vertical positioning of the actor within the actor layer is triggered by recognition of a gesture performed or performed by the actor. The gesture performed by the actor (i.e., a user or a person captured by the camera) is captured by an imaging device. As an example, such gesture recognition includes the system recognizing a running pose of the actor, thereby causing a change in the position of the actor within the actor layer. In other words, a gesture performed or performed by the actor, captured by the imaging device, is identified by the system. As a result, the lateral or vertical position of the actor depicted within the actor layer (i.e., the graphic representation of the actor recorded by the camera) changes. FIG. 8 illustrates another example of a gesture-triggered action (e.g., a movement, a shift, or a change in visual representation). In FIG. 8, this feature of the disclosure is provided in an application of a video game environment in which an image or video of the actor is depicted using an imaging device while the actor is playing the video game. In this scene 800, a running gesture of an actor captured by an imaging device is recognized by the system, which enables the actor in the actor layer to run, as evidenced by the initial position of the actor 802 and the gesture-elicited action depicted in the scene 800 showing the actor running (804). The scene 800 shows the position of the actor in the actor layer before the system recognizes the running gesture at 802, and after recognition at 804. Subjects in the subject layer (e.g., house 812) are displayed occluding the actor image due to the subject layer being positioned in front of the actor layer, and subjects in the background layer (e.g., castle 801, tree 816, tower 818) are displayed behind the actor layer. Similarly, an actor's mimicking a jump gesture, as recorded by the imaging device, may cause the position of the virtual actor in its actor layer to jump vertically on the screen from position 806 at 808, even though the physical actor in that physical space (i.e., non-virtual / real space) may not actually be lifted off the ground.Additionally, a subject (e.g., question mark box 814) is displayed in the layer at the top of the actor layer, thus occluding the jumping virtual actor represented in the actor layer. Scene 808 also illustrates how the system can be utilized to have more than two actors represented in a multi-layer scene at the same layer depth position in the scene. Two actors 808, 804 are represented in this scene 800. In this scene 800, question mark box 814 may be an interacting object, details of which are provided below.

[0131] In some examples, a partial actor image (such as face, head, or chest) may be extracted from an actor depicted as an actor in the actor layer. A non-limiting example of such a feature includes displaying the actor's chest extracted from a video feed in a fixed position in the actor layer during a presentation, as shown in scene 604 of FIG. 6. As the actor moves around his camera's field of view, as shown in FIG. 6, an image of his face in each video frame is extracted using basic face tracking techniques and placed in a fixed position on the multi-layered scene of the presentation. In another example, as shown in FIG. 9, the actor's face 904 extracted from the video stream is attached to a virtual character or avatar 902 added on the actor layer. This scene 900 includes a background layer 906, a subject layer (displaying the avatar 902), and an actor layer (displaying the actor 904). Such a virtual character may be controlled by an actor in a video game setting employing, among other things, gesture and action recognition techniques described herein. An image of the actor's face in each video frame is extracted using basic face tracking techniques and placed in place on the avatar's face. Although partial actor images are described elsewhere herein, it is understood that an image or graphical representation of an actor is not limited to capturing the entire actor with an imaging device, but may include any portion of the actor's body captured by the imaging device.

[0132] The present disclosure provides features whereby the system enables actors to interact with visible entities (also referred to herein as objects, e.g., graphic objects contained within an entity layer) using a display device. Such objects can be interacted with by actors via actors in the actor layer. Physical actors can enable such interactions using the gesture and action recognition of the system. As used herein, interaction refers to an actor image displayed within a multi-layered scene coming into contact with (also referred to as "colliding") with an object, controlling the object, or otherwise causing the object to be triggered by the presence of the actor image, gestures, or movement around or about the object image displayed within the scene.

[0133] An object can be placed in any scene layer (i.e., background layer, subject layer, or actor layer) of a multi-layered scene. Some objects can interact with at least one actor. These are called interaction objects. An interaction object is a virtual object depicted in a scene in at least one of the layers that make up the multi-layered scene. As used herein, an interaction object is a virtual object placed or created in a scene that is independent of (i.e., separable from) an actor, and an interaction with it triggers some action, such as a visual state change. With respect to interactions between subjects, objects, or actors depicted in a layer of a scene, an interaction object includes the whole or at least a part of the actor image itself. For example, when there is an interaction between two actors in a scene, such interaction is established using a similar method (i.e., one actor to the other actor(s)) as when the third-party actor(s) are treated as objects.

[0134] In some examples, recognition of the actor's movement or motion is used to trigger an interaction with an interaction object. Recognition of the actor's movement or motion can be done using video recognition approaches, such as YOLO (You Only Look Once), real-time object detection methods, human pose estimation (including those based on skeleton-based, contour-based, and / or volume-based models), and the like. Similarly, the actor's body parts (hands, feet, head, etc.) can be tracked by object recognition approaches, such as, for example, YOLO, human pose estimation (including those based on skeleton-based, contour-based, and / or volume-based models). As shown in FIG. 10, an action is triggered when the position of the actor's body part 1020 in the scene 1010 overlaps or contacts the interaction object 1022. An action in this context refers to a certain visible or audible change in the scene displayed in the display device. The triggered action can be the display of a certain animation, the provision of a certain sound, and the like. In some examples, the interaction causes such an action to occur where a change in the visual state of the interacting object (and / or the actor image itself) is observed from the scene. Examples of such changes include a certain prescribed animation, movement, color change, deformation, state change, and the like. Process 1012 demonstrates the process of actor interaction-induced actions. At step 1014, the system identifies an actor's body part that is assigned to interact with the interacting object. Once the actor's body part 1020 (hand in this example) interacts with the interacting object 1022 in the output video or scene therefrom by contacting the interacting object (i.e., the actor's body part 1020 overlaps, at least partially, with the interacting object 1022), an interaction between the body part and the interacting object 1022 is identified at step 1016. Thereupon, an action is triggered at 1018 by the system.In some examples, the interaction between the actor and the interaction object can be presented using a display device to present a video or image of the physical actor (e.g., an actual game player detected by an imaging device) in a composite video output. Such a graphical representation of the actor enhances the user's experience of this feature, for example, in a video game setting. The gesture-triggered action of FIG. 8 is also one type of triggered action. As illustrated in FIG. 8, a gesture change (in this example, a pose change) by the actor 802 from a standing pose to a running gesture results in the actor on the scene running (804) the actor's action in the scene. The imaging device 102 captures the actor, the game player, making the gesture. The data acquisition device 104 then captures such information, and in response, the control device 108 causes the actor in the scene to take the corresponding action. In the example illustrated in FIG. 8, the interaction object does not participate in the gesture-triggered action.

[0135] In some examples, such interaction features can be used by an actor to trigger animations in a presentation slide from software such as Powerpoint or Google Slides. In some examples, a student (i.e., an actor) in a virtual classroom can virtually touch a virtual flash card (i.e., an interaction object) to flip it over (i.e., a triggered or caused action). In some examples, the action is a change in some other state, such as an increase in a score in a video game. Obviously, many different types of objects and actions can be combined. An exemplary interaction is illustrated in FIG. 11. In a multi-layered scene, scene 1100, captured in FIG. 11, an actor 1106 is interacting with an interaction object 1108 in the scene. The actor 1106 is added on the actor layer between the subject layer and the background layer 1104 in this multi-layered scene 1100. In scene 1100, the actor 1106 can interact with the interaction objects in the subject layer and the background layer, such as 1108, 1114, 1112, etc. In this scene 1100, an interacting object (also called a subject of the subject layer) 1112 is located in the subject layer (i.e., a depth position with respect to the stacked layers in the scene), while interacting objects 1108 and 1114 are located in the background layer, where all objects in this scene can be interacted with by the actor. In one example, the interaction can be an action, such as pointing at the interacting object, thus triggering an action. This is an example of gesture-based interaction, where a body part of the actor (a finger in this example) is identified as having a certain gesture detected. Such gestures are detected by the system using an imaging device. Alternatively, similar to the example embodiment illustrated in FIG. 10, a body part of the actor that is in contact with (and therefore interacting with) or overlapping with the interacting object 1108 can trigger an action to be taken by the interacting object 1108. This is an example of contact-based interaction.Other examples include proximity-based interactions, where an actor approaching an interacting object beyond a certain threshold distance triggers an action manifested by the interacting object. In scene 1102, an action is triggered. In this example, a computation function is triggered and a graphical display of the computation result is the action. A change in the visual state of the interacting object is observed in scenes 1108 to 1110. As shown in FIG. 11, the system and method of the present disclosure can be applied to a learning environment to improve a student's learning ability by adding an interaction element to the learning process.

[0136] As an example, an actor can swipe his / her hand from left to right to advance the scene to the next scene in a presentation, or from right to left to change the scene to the previous slide. Similarly, in another example, an actor can gesture to mimic holding a writing implement with his / her hand. Recognition of this gesture then triggers writing on the virtual space of the slides in the scene as if the actor were holding the virtual writing implement. In a gaming example, the actor's gestures are recognized and used to trigger actions such as running, jumping, throwing a virtual basketball, swinging or casting a spell with a virtual golf club, stomping, kicking, pulling, punching interactive objects, and the like. These and other applications of the systems and methods of this disclosure can be realized to enhance the user's experience of the intended purpose of video-enabled interactive practice.

[0137] In some examples, more than one actor may be involved: one actor may interact with another actor; two or more actors may be capable of cooperatively interacting with interacting objects; and two or more actors may be capable of each interacting with various interacting objects. For example, in one embodiment (such as a video game), a first actor's hand may be used to virtually punch a second actor. In this interaction, the first actor's "punch" gesture is an interaction with the second actor, which in this case is treated as an interacting object. When the first actor makes virtual contact with the second actor, an action is triggered, such as a movement, deformation, or color change (i.e., a change in visual state) of the interacting object that is the second actor. For example, gestures that can be used to interact with an interaction object include punching, kicking, karate chopping, pushing, tapping, grasping, twisting, stomping, jumping, poking, pulling, pushing, lifting, dropping, throwing, eating, drinking, and the like.

[0138] Just as actors can interact with each other, objects controlled by actors can also interact with other objects (including other actors). As an example, an actor may have a toy sword that can interact with the interaction object on the output video. In some examples, an actor may have a virtual weapon (achieved through application of gesture recognition and augmented reality approaches) that can interact with interaction objects, including other actors or other actors' swords (whether the swords are virtual or physical, and whether the actors are physically in the same space or at a remote location). Non-limiting examples of actor-controlled objects include virtual weapons, such as close combat weapons, swords and lightsabers, and projectile weapons such as guns and snowballs. In educational or business applications, such actor-controlled objects include writing implements (with visually recognizable features on one or both ends for enhanced tracking), pointing devices, and the like.

[0139] In some examples, an actor uses an item such as a remote control to trigger an action within a scene. Such remote control devices include, but are not limited to, a remote slide advancer, a video game controller, a toy lightsaber, a toy gun, etc. In some examples, an actor uses speech or voice to trigger an action within a scene. Specifically, an actor can say "next slide" to trigger an action that changes the scene or triggers an action from one slide to the next in the presentation, and similarly, an actor can say "previous slide" to trigger an action that changes the scene from one slide to the previous slide in the presentation.

[0140] As shown in Fig. 12A, two or more actors may interact with the same interacting object in a virtual space. In this scene in Fig. 12A, two actors, a first actor 1204 and a second actor 1206, are visualized in front of a background layer 1202. The scene in Fig. 12A presents a virtual interaction between the two actors 1204, 1206 that is displayed in the output video or visualized in this multi-layered scene. The interaction is displayed in this example as both actors 1204, 1206 touching or overlapping the interacting object 1208. Actions triggered by such interactions are triggered only if the interaction is completed by the two actors simultaneously or by both actors 1204, 1206. In this example, the layers of the multi-layered scene include a background layer 1202 positioned behind all other layers, a second actor layer in front of the background layer, an interacting object layer positioned in front of the second actor layer, and then a first actor layer positioned in front of the interacting object layer. As is clear, Figure 12A shows a total of four different layers in the view, including two actor layers.

[0141] In some examples, multiple actors may be included within the same actor layer. In some examples, a single actor layer may depict multiple actors if two or more actors are physically in the same space in front of the same imaging device. When their actor images are separated, they are included within the same actor layer. In other examples, a single actor layer may depict multiple actors if they are physically separate in front of different imaging devices. When their actor images are separated, they are inserted within the same actor layer.

[0142] 12B is another visualization of the concept of an interacting object 1208. In this embodiment, the interacting object 1208 is placed in a three-dimensional space 1314, and multiple actor / user representations 1204, 1206 interact with the interacting object 1208. The multiple actors 1204, 1206 are caused to interact with the interacting object 1208 by real users captured by corresponding imaging devices of the system.

[0143] Body cutoff & extrapolation

[0144] Given the distance of the camera from the user / actor, it may occur that parts of the user's body may extend beyond the boundaries of the camera field of view. For these cases, the "missing" or "cut-off" body parts of the user will be artificially added to the representation of the user / actor contained within the actor layer and / or virtual world.

[0145] Similarly, there are other instances where it may be desirable to have only a portion of a physical user (as seen within the camera field of view) reflected in the virtual representation of the user, while artificially "filling out" or "simulating" the balance of the user's virtual representation.

[0146] For example, a user / actor may choose to operate the system while in one body posture (such as sitting or lying down), while it may be desired for the virtual representation of the user to assume a different posture (such as standing, walking or running). In such cases, portions of the user's virtual representation (e.g., the upper body) may more directly reflect the user's physical body image and movements, while other portions of the actor's virtual representation (e.g., the lower body) may be artificially rendered in a manner that does not reflect a similar portion of the actor's physical body.

[0147] Both the "body cutoff" case and the "relaxed posture" case, as well as other cases, may represent parts of the body differently than they appear in the camera image.

[0148] In one example, a user's body parts (including but not limited to hands, head, legs, feet) may extend beyond the boundaries of the camera field of view. The system detects when at least a portion of the body (including but not limited to hands, head, legs, feet, the entire lower body) extends beyond the boundaries of the camera field of view. One method for determining this includes determining that the shape or silhouette of the body (as determined by body segmentation) is in contact with the boundaries of the camera image.

[0149] In one embodiment, an image of the user, the entire user or individual body parts / portions thereof, is stored in memory. In some embodiments, the image of the user is captured as part of a body image capture process (such as the user standing in a "T-pose" in front of the camera and possibly being asked to turn around) to capture at least a portion of their body with the camera. In other embodiments, the image of the user's body is captured seamlessly during the user's process of using the system, without the need for a limited capture process.

[0150] Based on the captured images, the system creates and stores a representation of the body such that at least a portion of the representation can be utilized by the system as needed.

[0151] 14 and 15, the system detects or determines, at step 1502, the occurrence of a body cutoff and which body part is missing from the camera field of view 1512. The system then extracts a representation of the missing body part 1508 from the stored body representation, as described above, at step 1504. At step 1506, the extracted representation of the missing body part 1508 is added or supplemented to the user's representation so that the user's representation is shown as a whole along with the supplemented body part 1510 and displayed on the display device 110.

[0152] An illustrative example of how the aforementioned system may be utilized is as follows: In one embodiment, a user / actor may utilize the system using a mobile phone's selfie camera. Particularly in a landscape or horizontal orientation, the user / actor may feel that they need to stand far back from the phone to see their entire body, and therefore cannot see themselves well enough on the relatively small screen of the mobile phone. Instead, in such cases, the user may temporarily step back far enough from the phone so that the system can capture and learn what the user's entire body looks like. The user may then move closer to the camera (and may even choose to sit or lie down) so that the camera can see the user's upper body, but not their complete lower body. By utilizing pre-stored images and representations of the user / actor's entire body, the system can generate a full-body representation of the user / actor for the actor layer and / or virtual world. The image then displayed on the screen may then display a full-body representation of the user. Similarly, any other users viewing the representation of the virtual world will see a full-body representation of that user.

[0153] As discussed above and repeated below, the present disclosure includes, without limitation, the following example embodiments.

[0154] Item 1. A computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method including: receiving, using an imaging device, two-dimensional video stream data of a user, the two-dimensional video stream data at least partially capturing the user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separating a user representation of the user from the two-dimensional video stream data; identifying user position information from the two-dimensional video stream, the position information being identified based on a position of the user relative to the imaging device; using the position information to add the user representation to the three-dimensional scene by modifying data associated with voxels that comprise the three-dimensional scene; and displaying the three-dimensional scene and the added user representation on a display device, the added user representation displayed on the display device being controlled based on detection of a change to at least one of the user and the position information.

[0155] Clause 2. The method of clause 1, wherein identifying user position information includes extracting depth information from the two-dimensional video stream.

[0156] Clause 3. The method of clause 2, wherein the step of extracting depth information comprises recognizing depth cues within the two-dimensional video stream.

[0157] Clause 4. The method of clause 1, wherein the representation of the user includes an image of at least a portion of the user.

[0158] Addendum 5. The method of Addendum 1 further includes the steps of detecting a missing body part from the user representation, extracting a representation of the missing body part from the data storage device, and adding the extracted representation of the missing body part to the user representation.

[0159] Clause 6. The method of clause 1 further includes the steps of detecting a person within the two-dimensional video stream data, comparing the detected person to a target user, determining whether the detected person is the target user, and in response to determining that the detected person is the target user, identifying the detected person as a user.

[0160] Addendum 7. The method of Addendum 1 further includes a step of updating the added user representation in response to detecting a gesture or pose performed by the user, where updating the added user representation includes the added user representation performing a predefined action.

[0161] Clause 8. The method of clause 1 further comprises identifying user pose information from the two-dimensional video stream.

[0162] Clause 9. The method of clause 8 further includes updating the added user representation within the three-dimensional scene based on changes to at least one of the user, position information, and pose information.

[0163] Clause 10. The method of clause 9, wherein the modifying includes causing the user to cause the added user representation to interact with the virtual object.

[0164] Clause 11. The method of clause 10 further includes updating the virtual object in response to the user causing the added user representation to interact with the virtual object.

[0165] Clause 12. The method of clause 11, wherein updating the virtual object includes causing the virtual object to perform a predefined action.

[0166] Clause 13. The method of clause 1 further includes updating the virtual object in response to the user causing the added user representation to display a gesture.

[0167] Clause 14. The method of clause 13, wherein updating the virtual object includes causing the virtual object to perform a predefined action upon detecting the gesture.

[0168] Clause 15. A computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method including: receiving, using a first imaging device, first two-dimensional video stream data of a first user, the first two-dimensional video stream data at least partially capturing the first user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; isolating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, the first position information being identified based on a relative position of the first user with respect to the first imaging device; receiving, using a second imaging device, second two-dimensional video stream data of a second user, the second two-dimensional video stream data at least partially capturing the second user; and isolating a second user representation of the second user from the second two-dimensional video stream. identifying second position information of a second user from the second two-dimensional video stream, the second position information being identified based on a relative position of the second user with respect to the second imaging device; adding first and second user representations to the three-dimensional scene by modifying data associated with voxels comprising the three-dimensional scene, the data associated with the first user representation using the first position information and the data associated with the second user representation using the second position information; and displaying the three-dimensional scene, the added first user representation, and the added second user representation on the first display device and the second display device, the first added user representation displayed on the first display device being controlled based on detection of a change to at least one of the first user and the first position information and the second added user representation displayed on the second display device being controlled based on detection of a change to at least one of the second user and the second position information.

[0169] Clause 16. The method of clause 15 further includes updating a representation of the second added user within the three-dimensional scene based on changes to at least one of the second user and the second position information.

[0170] Clause 17. The method of clause 16, wherein the modifying includes having the second user interact with the added second user representation with the added first user representation.

[0171] Addendum 18. The method of Addendum 15, wherein the step of identifying position information of the first user and the second user includes extracting depth information from the first two-dimensional video stream and the second two-dimensional video stream, respectively.

[0172] Clause 19. The method of clause 18, wherein extracting depth information includes recognizing depth cues within one or more of the first two-dimensional video stream and the second two-dimensional video stream.

[0173] Clause 20. The method of clause 15, wherein the first user representation includes an image of at least a portion of the user.

[0174] Addendum 21. The method of Addendum 15 further includes steps of detecting a missing body part from the first or second user representation, extracting a representation of the missing body part from the data storage device, and adding the extracted representation of the missing body part to one or more of the first and second user representations.

[0175] Addendum 22. The method of Addendum 15 further includes the steps of detecting a person in the first two-dimensional video stream data, comparing the detected person to a target user, determining whether the detected person is the target user, and in response to determining that the detected person is the target user, identifying the detected person as the first user.

[0176] Addendum 23. The method of Addendum 15 further includes a step of updating the added first user representation in response to detecting a gesture or pose performed by the first user, where updating the added first user representation includes the added first user representation performing a predefined action.

[0177] Addendum 24. The method of Addendum 15 further includes a step of updating the added second user representation in response to detecting a gesture or pose performed by the second user, where updating the added second user representation includes the added second user representation performing a predefined action.

[0178] Clause 25. The method of clause 15 further includes identifying first pose information of a first user from the first two-dimensional video stream and second pose information of a second user from the second two-dimensional video stream.

[0179] Addendum 26. The method of Addendum 25 further includes the steps of updating the added first user representation within the three-dimensional scene based on changes to at least one of the first user, the first position information, and the first pose information, and updating the added second user representation within the three-dimensional scene based on changes to at least one of the second user, the second position information, and the second pose information.

[0180] Clause 27. The method of clause 26, wherein the modifying includes the first or second user causing the corresponding added first or second user representation to interact with the virtual object.

[0181] Clause 28. The method of clause 27 further includes updating the virtual object in response to the first and second users causing the added first and second user representations to interact with the virtual object.

[0182] Clause 29. The method of clause 28, wherein updating the virtual object includes causing the virtual object to perform a predefined action.

[0183] CLAIM 30. The method of claim 15 further includes updating the virtual object in response to the first and second users causing the corresponding added user representations to display gestures.

[0184] Clause 31. The method of clause 30, wherein updating the virtual object includes causing the virtual object to perform a predefined action upon detecting the gesture.

[0185] Clause 32. A system for controlling a virtual representation of a user within a three-dimensional scene, comprising: an imaging device that captures two-dimensional video stream data of the user; a storage device having computer-readable program instructions stored therein; and a processor configured in communication with the storage device to execute the computer-readable instructions to cause the system to at least: receive two-dimensional video stream data of the user from the imaging device, the two-dimensional video stream data at least partially capturing the user; receive a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separate a user representation of the user from the two-dimensional video stream data; identify user position information from the two-dimensional video stream, the position information being identified based on the user's position relative to the imaging device; add the user representation to the three-dimensional scene by modifying data associated with voxels that comprise the three-dimensional scene using the position information and pose information; and display the three-dimensional scene and the added user representation on a display device, the added user representation displayed on the display device being controlled based on detection of a change to at least one of the user and the position information.

[0186] Clause 33. The system of clause 32, wherein the user's position information includes depth information extracted from the two-dimensional video stream.

[0187] Clause 34. The system of clause 33, wherein extracting depth information includes recognizing depth cues within a two-dimensional video stream.

[0188] Clause 35. The system of clause 32, wherein the user representation includes an image of at least a portion of the user.

[0189] Addendum 36. The system of Addendum 32, wherein the processor is further configured to execute computer readable program instructions to cause the system to detect missing body parts from the user representation, extract a representation of the missing body parts from the data storage device, and add the extracted representation of the missing body parts to the user representation.

[0190] Clause 37. The system of clause 32, wherein the processor is further configured to execute computer readable program instructions to cause the system to detect persons within the two-dimensional video stream data, compare the detected persons to intended users, determine whether the detected persons are intended users, and in response to determining that the detected persons are intended users, identify the detected persons as users.

[0191] CLAIM 38. The system of claim 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the added user representation in response to detecting a gesture or pose performed by the user, wherein updating the added user representation includes the added user representation performing a predefined action.

[0192] Clause 39. The system of clause 32, wherein the processor is further configured to execute computer readable program instructions to cause the system to identify user pose information from the two-dimensional video stream.

[0193] Addendum 40. The system of Addendum 39, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the added user representation within the three-dimensional scene based on changes to at least one of the user, position information, and pose information.

[0194] Clause 41. The system of clause 40, wherein the transformation includes the user causing the added user representation to interact with the virtual object.

[0195] Clause 42. The system of clause 41, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the virtual object in response to a user causing the added user representation to interact with the virtual object.

[0196] Item 43. The system of item 42, wherein updating the virtual object includes causing the virtual object to perform a predefined action.

[0197] CLAIM 44. The system of claim 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the virtual object in response to a user causing the added user representation to display a gesture.

[0198] Addendum 45. The system of Addendum 44, wherein updating the virtual object includes causing the virtual object to perform a predefined action upon detecting the gesture.

[0199] Although some variations of the present disclosure have been described as examples in the specific description, it is clear that further examples can be developed within the spirit and scope of the present disclosure or its inventive concept. Those skilled in the art will readily recognize that numerous modifications, applications, and variations of the apparatus and method can be made while retaining the teachings of the present disclosure. It is therefore understood that the present disclosure is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it will be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, it is contemplated that combinations of elements and / or functions different from those explicitly described above may be recited in some of the appended claims. Although certain terms have been employed herein, they are used in a generic and descriptive sense only, and not for purposes of limitation. It will be understood that the variations disclosed above and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated substitutions, modifications, variations, or improvements therein will subsequently occur to those skilled in the art which are also intended to be encompassed by the following claims.

Claims

1. 1. A computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method comprising: receiving two-dimensional video stream data of a user using an imaging device, the two-dimensional video stream data at least partially capturing the user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separating a user representation of the user from the two-dimensional video stream data; identifying position information of the user from the two-dimensional video stream, the position information being identified based on a relative position of the user with respect to the imaging device; identifying pose information of the user from the two-dimensional video stream; adding the user representation to the three-dimensional scene by using the position information to modify data associated with voxels that comprise the three-dimensional scene; displaying the three-dimensional scene and the added user representation on a display device, wherein the added user representation displayed on the display device is controlled based on detection of a change to at least one of the user and the position information. A method comprising:

2. The method of claim 1 , wherein identifying the user's position information comprises extracting depth information from the two-dimensional video stream.

3. The method of claim 2 , wherein the step of extracting depth information comprises recognizing depth cues within the two-dimensional video stream.

4. The method of claim 1 , wherein the user representation includes an image of at least a portion of the user.

5. detecting missing body parts from the user representation; extracting a representation of the missing body part from a data store; adding the extracted representation of the missing body part to the user representation; The method of claim 1 , further comprising the step of:

6. detecting people within the two-dimensional video stream data; comparing the detected person with a target user; determining whether the detected person is the intended user; in response to determining that the detected person is the intended user, identifying the detected person as the user; The method of claim 1 , further comprising the step of:

7. 13. The method of claim 1, further comprising updating the added user representation in response to detecting a gesture or pose performed by the user, wherein the updating of the added user representation comprises the added user representation performing a predefined action.

8. The method of claim 1 , further comprising updating the added user representation within the three-dimensional scene based on changes to at least one of the user, the position information, and the pose information.

9. The method of claim 8 , wherein the change comprises the user causing the added user representation to interact with the virtual object.

10. 10. The method of claim 9, further comprising updating the virtual object in response to the user causing the added user representation to interact with the virtual object, wherein the updating of the virtual object comprises causing the virtual object to perform a predefined action.

11. 10. The method of claim 1, further comprising: updating the virtual object in response to the user causing the added user representation to display a gesture, the updating of the virtual object comprising causing the virtual object to perform a predefined action upon detecting the gesture.

12. 1. A computer-implemented method for controlling a virtual representation of a user within a three-dimensional scene, the method comprising: receiving, using a first imaging device, first two-dimensional video stream data of a first user, the first two-dimensional video stream data at least partially capturing the first user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, the first position information being identified based on a relative position of the first user with respect to the first imaging device; receiving, using a second imaging device, second two-dimensional video stream data of a second user, the second two-dimensional video stream data at least partially capturing the second user; separating a second user representation of the second user from the second two-dimensional video stream data; identifying second position information of the second user from the second two-dimensional video stream, the second position information being identified based on a relative position of the second user with respect to the second imaging device; adding the first and second user representations to the three-dimensional scene by modifying data associated with voxels comprising the three-dimensional scene, the data associated with the first user representation using the first position information and the data associated with the second user representation using the second position information; displaying the three-dimensional scene, the added first user representation, and the added second user representation on a first display device and a second display device, wherein the first added user representation displayed on the first display device is controlled based on detection of a change to at least one of the first user and the first position information, and the second added user representation displayed on the second display device is controlled based on detection of a change to at least one of the second user and the second position information. A method comprising:

13. The method of claim 15 , further comprising updating the representation of the second added user within the three-dimensional scene based on changes to at least one of the second user and the second position information.

14. The method of claim 16 , wherein the change includes the second user causing the added second user representation to interact with the added first user representation.

15. 1. A system for controlling a virtual representation of a user within a three-dimensional scene, comprising: an imaging device for capturing two-dimensional video stream data of a user; a storage device having computer readable program instructions stored therein; a processor in communication with the storage device to execute the computer readable instructions to provide the system with at least: receiving the two-dimensional video stream data of the user from the imaging device, the two-dimensional video stream data at least partially capturing the user; receiving a three-dimensional scene, the three-dimensional scene including data associated with at least one virtual object and its position within the three-dimensional scene; separating a user representation of the user from the two-dimensional video stream data, the user representation including an image of at least a portion of the user; identifying position information of the user from the two-dimensional video stream, the position information being identified based on a relative position of the user with respect to the imaging device, the position information of the user including depth information extracted from the two-dimensional video stream; identifying pose information of the user from the two-dimensional video stream; adding the user representation to the three-dimensional scene by modifying data associated with voxels that comprise the three-dimensional scene using the position information and the pose information; and displaying the three-dimensional scene and the added user representation on a display device, wherein the added user representation displayed on the display device is controlled based on detecting changes to at least one of the user, the position information, and the pose information. A processor configured to execute A system comprising:

Images