User-controlled three-dimensional scene

By receiving and processing the user's two-dimensional video stream data, identifying location information and modifying the voxel data of the three-dimensional scene, the problem of user virtual representation occluding background in virtual reality is solved, and real-time control and interaction of users in the virtual world is realized.

CN118922865APending Publication Date: 2024-11-08M·库恩都
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380018411.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-01-23
Filing Date
2023-01-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art methods of users controlling their virtual representations in virtual reality have occlusion problems, especially in remote presentations, where the user's video streams are usually displayed separately, making it difficult to effectively interact with the audience, and green screen technology causes virtual characters to obstruct background content.

Method used

By using the imaging unit to receive the user's two-dimensional video stream data, identify the user's location information, and modify the voxel data of the three-dimensional scene, add the user's virtual representation to the scene, so that it can interact in real time and display on the display unit , control based on user's actions and position changes.

Benefits of technology

It realizes the user's real-time control of the movement and interaction of their virtual representations in the virtual world, avoids the problem of virtual characters blocking the background content, and improves the interaction effect between users and audiences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922865A_ABST
    Figure CN118922865A_ABST
Patent Text Reader

Abstract

The present disclosure generally relates to systems and methods for a user to control its own virtual representation in a three-dimensional virtual world. The system and method can position a user itself in a three-dimensional scene using two-dimensional image or video data of the user with extracted depth information. It also provides a control system and method for a user to control its own virtual representation using an output video as a visual feedback mechanism in a three-dimensional space including its own virtual representation. The user interacts with other virtual objects or items in the scene, even with other users visualized in the scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for a user to control a virtual representation of themselves in a three-dimensional virtual world. Background Art

[0002] Interacting with a virtual reality world currently requires using a virtual reality (VR) headset, leveraging specialized 3D sensors, typing on a keyboard and computer, or swiping on a phone to control an avatar to move and interact in the virtual environment.

[0003] The present disclosure may be used to insert an image or representation of a user / actor into a three-dimensional virtual world. The present invention may then allow the user to see themselves in the virtual world and move around in their physical world, thereby controlling the movement of their image or representation and interaction with objects in the virtual world they are viewing.

[0004] The present disclosure provides systems and methods that can be used in many situations where a participant needs to appear on screen in real time with other content, but not overlap or be obscured by certain objects in the virtual world. This is achieved by inserting a representation of the participant into a "scene". Such a scene is composed of multiple different levels or depth layers of content, some of which are behind the participant and some of which are in front of the participant from the viewer's perspective and are therefore not obscured by the participant.

[0005] The scene may be composed of voxels (three-dimensional pixels) in xyz three-dimensional space, and the representation of the user is inserted as a set of voxels contained in the scene. The present disclosure provides systems and methods that enable (one or more) users to interact in real time with other objects or items in the scene or even with each other in the case of multiple users.

[0006] Currently, green screen and virtual green screen technology is commonly used to place a human image over a background image or video. Like a meteorologist on the evening news weather segment, the "weatherman" effect blocks the view of the subject material so the human must move left and right to point out parts of the background without obscuring parts of the background.

[0007] This is particularly problematic for remote presentations. Today, the participant’s video stream is often displayed in a completely separate window from the content, making it more difficult for the participant to point to the content (using anything other than a mouse pointer) and communicate more effectively with the audience. If the participant is placed in front of the content (such as by using a chroma key green screen), the participant will obscure parts of the content they are trying to present. Summary of the invention

[0008] In some cases, the subject matter of the present application may involve interrelated products, alternative solutions to a particular problem, and / or multiple different uses of a single system or article. Accordingly, the present disclosure includes, but is not limited to, the following example embodiments.

[0009] The present disclosure generally relates to systems and methods for users to control their own virtual representation in a three-dimensional virtual world. The systems and methods can utilize two-dimensional image or video data of the user with extracted depth information to position themselves in a three-dimensional scene.

[0010] Some example embodiments provide a computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: receiving two-dimensional video stream data of a user using an imaging unit, wherein the two-dimensional video stream data at least partially captures the user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a user representation of the user from the two-dimensional video stream data; identifying position information of the user from the two-dimensional video stream, wherein the position information is identified based on a relative position of the user relative to the imaging unit; adding the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information; and displaying the three-dimensional scene and the added user representation on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting a change to at least one of the user and the position information.

[0011] Some example embodiments provide a computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: receiving, using a first imaging unit, first two-dimensional video stream data of a first user, wherein the first two-dimensional video stream data at least partially captures the first user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, wherein the first position information is identified based on a relative position of the first user relative to the first imaging unit; receiving, using a second imaging unit, second two-dimensional video stream data of a second user, wherein the second two-dimensional video stream data at least partially captures the second user; isolating a second user representation of the second user from the second two-dimensional video stream data; Identifying second position information of the second user, wherein the second position information is identified based on the relative position of the second user relative to the second imaging unit; adding the first user representation and the second user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene, wherein the data associated with the first user representation uses the first position information, and the data associated with the second user representation uses the second position information; and displaying the three-dimensional scene, the added first user representation, and the added second user representation on a first display unit and a second display unit, wherein the added first user representation displayed on the first display unit is controlled based on detecting a change to at least one of the first user and the first position information, and the added second user representation displayed on the second display unit is controlled based on detecting a change to at least one of the second user and the second position information.

[0012] Some example embodiments provide a system for controlling a virtual representation of a user in a three-dimensional scene, comprising: an imaging unit that captures two-dimensional video stream data of a user; a storage unit having computer-readable program instructions stored therein; and a processor that communicates with the storage unit, the processor being configured to execute the computer-readable program instructions so that the system at least: receives the two-dimensional video stream data of the user from the imaging unit, wherein the two-dimensional video stream data at least partially captures the user; receives a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolates a user representation of the user from the two-dimensional video stream data; identifies position information of the user from the two-dimensional video stream, wherein the position information is identified based on a relative position of the user relative to the imaging unit; adds the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information and posture information; and displays the three-dimensional scene and the added user representation on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting a change to at least one of the user and the position information.

[0013] These and other features, aspects and advantages of the present disclosure will become apparent by reading the following detailed description and the accompanying drawings (described briefly below). The present disclosure includes any combination of two, three, four or more features or elements set forth in the present disclosure, whether or not these features or elements are explicitly combined or otherwise described in the specific example embodiments described herein. The present disclosure is intended to be read as a whole so that any separable features or elements of the present disclosure should be considered combinable in any of its aspects and example embodiments, unless the context of the present disclosure clearly provides otherwise.

[0014] Therefore, it should be understood that the present disclosure is provided only to summarize some example embodiments in order to provide a basic understanding of certain aspects of the present disclosure. Therefore, it should be understood that the above-mentioned example embodiments are merely examples and should not be interpreted as narrowing the scope or spirit of the present disclosure in any way. Other example embodiments, aspects and advantages will be apparent from the following detailed description, in conjunction with the accompanying drawings, which illustrate the principles of some of the described example embodiments by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various aspects of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] Figure 1 A system according to some example implementations is shown.

[0017] Figure 2 and Figure 3 is a flowchart illustrating various steps in a method of generating a multi-layer scene according to various example embodiments of the present disclosure.

[0018] Figures 4 to 9 Example implementations of systems and methods according to the present disclosure are shown.

[0019] Fig.10 and Fig.11 An example implementation of a participant interacting with an interactive object according to the systems and methods of the present disclosure is shown.

[0020] Fig. 12A and Fig. 12B An example implementation of multiple participants interacting with an interactive object according to the systems and methods of the present disclosure is shown.

[0021] Figures 13A to 29 Various example implementations of systems and methods according to the present disclosure are shown.

[0022] It should be noted that some details in the figures have been simplified and drawn to facilitate understanding rather than maintaining strict structural accuracy, detail, and scale. DETAILED DESCRIPTION

[0023] The detailed description set forth below in conjunction with the accompanying drawings or diagrams is intended to describe the currently preferred embodiments of the present disclosure and does not represent the only form in which the present disclosure may be constructed and / or utilized. The description sets forth the functions and step sequence of constructing and operating the present disclosure in conjunction with the illustrated examples.

[0024] When referring to the description, specific details are set forth in order to provide a thorough understanding of the disclosed examples. In other instances, well-known methods, procedures, components, and materials have not been described in detail to avoid unnecessarily lengthening the present disclosure.

[0025] By reading the following detailed description and the accompanying drawings (described briefly below), these and other features, aspects and advantages of the present disclosure will be apparent. The present disclosure includes any combination of two, three, four or more features or elements set forth in the present disclosure, whether or not these features or elements are clearly combined or otherwise narrated in the specific example embodiments described herein. The present disclosure is intended to be read as a whole so that any separable features or elements of the present disclosure should be considered combinable in any of its aspects and example embodiments, unless the context of the present disclosure clearly specifies otherwise.

[0026] It should be understood that if an element or portion is referred to herein as being "on," "relative to," "in communication with," "connected to," "attached to," or "coupled to" another element or portion, it can be directly on, relative to, in communication with, connected to, attached to, or coupled to another element or portion, or intervening elements or portions may be present. When used, the term "and / or" includes any and all combinations of one or more of the associated listed items, if provided.

[0027] The terms used herein are only used to describe specific embodiments and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly dictates otherwise. It should also be understood that the terms "include" and / or "comprise" when used in this specification specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof that are not explicitly stated.

[0028] Furthermore, it should be understood that the terms "data," "content," "digital content," "information" and similar terms may sometimes be used interchangeably unless otherwise specified.

[0029] Furthermore, it should be understood that, unless otherwise specified, the term "real-time" means processing, displaying, capturing, detecting, recognizing or analyzing data without intentional delay, within the processing limitations of the system.

[0030] Various operations may be described as multiple discrete operations in turn, which may aid in understanding the embodiments; however, the order of description should not be construed as to imply that these operations are order dependent.

[0031] For ease of description and / or illustration, spatial relative terms such as "below", "below", "lower", "above", "upper", "proximal", "distal", "behind", "front", etc. may be used herein to describe the relationship of one element or feature to another element or feature, as shown in the various figures. However, it should be understood that spatial relative terms are intended to cover different orientations of the device in use or operation other than the orientation shown in the figures. For example, if the device in the figure is turned over, the elements described as "below" or "below" other elements or features will be positioned "above" other elements or features. Therefore, relative spatial terms such as "below" can cover orientations above and below. The device can be oriented in other ways (rotated 90 degrees or other orientations), and the spatial relative descriptors used herein should be interpreted accordingly. Similarly, the relative spatial terms "proximal" and "distal" can also be interchangeable where applicable. Such descriptions are merely for the convenience of discussion and are not intended to limit the application of the disclosed embodiments.

[0032] The term first, second, third, etc. can be used herein to describe various elements, components, regions, parts and / or parts. It should be understood that these elements, components, regions, parts and / or parts should not be limited by these terms. These terms are only used to distinguish an element, component, region, part or part from another region, part or part. Therefore, the first element, component, region, part or part discussed below can be referred to as the second element, component, region, part or part, without departing from the teaching of this article.

[0033] Some examples of the present disclosure may be implemented on a computer system that typically includes one or more processors for processing information and instructions, RAM for storing information and instructions, ROM for storing static information and instructions, data storage units such as magnetic or optical disks and disk drives for storing information and instructions, modules as software units executed on the processors, optional user output devices such as display screen devices (e.g., monitors) for displaying filtered information to a computer user, and optional user input devices.

[0034] Those skilled in the art will appreciate that the present embodiment may be at least partially embodied as a computer program product embodied in any tangible medium of expression in which a computer-usable program code is stored. For example, some of the embodiments described below with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products may be implemented by computer program instructions. Computer program instructions may be stored in a computer-readable medium that may instruct a computer, a controller, or other programmable data processing device to operate in a particular manner so that the instructions stored in the computer-readable medium constitute a manufactured product, including instructions and processes for implementing the functions / actions / steps specified in the flowcharts and / or block diagrams. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that instructions executed by a processor of a computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0035] In the following description, reference is made to the accompanying drawings, which are illustrations of embodiments in which the disclosed system may be practiced. However, it should be appreciated that other structural and functional modifications may be developed by those skilled in the art without departing from the novelty and scope of the present disclosure.

[0036] The system may include one or more computers or computerized elements that communicate with each other and together perform different functions of the system. The systems and methods contemplated herein may also include a storage unit, such as a non-transitory computer-readable medium, which is configured to instruct one or more computers to perform the steps and functions of the systems and methods described herein. In some embodiments, communication between one or more computers or similarly one or more processors may support a variety of encryption / decryption methods and mechanisms for various types of data.

[0037] The computerized user interface can include one or more computing devices that communicate with each other through the Internet. One or more computers of the computerized user interface envisioned herein can include a memory, a processor, and an input / output system. In some embodiments, the computer can also include a network connection and / or a display screen. These computerized elements can work together within the network to provide function to the computerized user interface. The computerized user interface can be any type of computerized interface known in the art, can allow a user to input data and receive feedback therefrom. The computerized user interface can also provide the output performed by the system as described herein.

[0038] The storage unit or data described in this article can be in a format including but not limited to XML, JSON, CSV, binary, through any connection type: serial, Ethernet, etc., through any protocol: UDP, TCP, etc.

[0039] The computers or computing devices contemplated herein may include, but are not limited to, virtual systems, cloud / remote systems, desktop computers, laptop computers, tablet computers, handheld computers, smart phones and other cellular phones and similar Internet mobile devices, digital cameras, custom computing devices configured specifically to perform the methods contemplated in the present disclosure, and the like.

[0040] The networks contemplated herein may include, for example, one or more of the Internet, a wide area network (WAN), a local area network (LAN), analog or digital wired and wireless telephone networks (e.g., PSTN, integrated services digital network (ISDN), cellular networks, and digital subscriber lines (xDSL)), radio, television, cable, satellite, and / or any other transport or tunneling mechanism for carrying data. A network may include multiple networks or subnetworks, each of which may include, for example, a wired or wireless data path. A network may include a circuit-switched voice network, a packet-switched data network, or any other network capable of carrying electronic communications. Examples include, but are not limited to, Picture Transfer Protocol (PTP) over Internet Protocol (IP), IP over Bluetooth, IP over WiFi, and PTP over IP networks (PTP / IP).

[0041] The imaging units or video capture devices contemplated herein may include, but are not limited to, various types of two-dimensional and three-dimensional cameras, such as webcams, mobile phone cameras, depth sensing imaging units (including, but not limited to, stereo cameras, multiple cameras used together, 2D cameras complementary to LiDAR, cameras on mobile phones with depth sensing technology, etc.), DSLRs, non-SLR digital cameras (such as, but not limited to, compact digital cameras and SLR-like bridge digital cameras (also known as advanced digital cameras) and SLR-like interchangeable lens digital cameras), and video recorders (such as, but not limited to, portable cameras, 2D analog cameras and 2D IP cameras, 3D cameras, etc.; devices that can provide video feeds of any duration, such as DVRs; portable computing devices with cameras, such as tablets, laptops, smartphones); etc. The position or location of the imaging unit may also vary. The imaging unit may face to capture the front of a participant, the face of a participant, a side view of a participant, a top view of a participant, a rear view of a participant, a perspective view of a participant, etc. Similarly, the imaging unit may zoom in and out to change the area of ​​the participant or object captured.

[0042] In some embodiments, the system can utilize a display positioned away from the user so that the user's movements when using the system are not hindered by any wearable device. Similarly, the system can include an imaging unit positioned away from the user so that the user's movements are not restricted by any such wearable device.

[0043] The image or video file contemplated herein can be any digital image format that can be interpreted by a computer or computing device. Examples of image files contemplated herein include, but are not limited to, JPEG, GIF, TIFF, PNG, Bitmap, RAW, PNM, WEBP, etc.

[0044] The present disclosure provides a system and method: the system and method are used to enable a user to be represented in a three-dimensional scene by using an imaging unit (including a two-dimensional imaging unit), and to control the movement and interaction of the representation in the three-dimensional scene. More specifically, the present disclosure provides such a system and method: the system and method are used to add a representation or image of a user (e.g., separated from a two-dimensional video stream captured by an imaging unit) to a three-dimensional scene by modifying voxels of the scene, and then enable the representation to interact with virtual objects (including representations of other users) in the three-dimensional scene in response to changes in the actual user's location, posture, and / or gesture. The resulting three-dimensional scene can be viewed using a display unit.

[0045] The present disclosure also generally provides systems and methods for generating multi-layer scenes. More specifically, the present disclosure provides systems and methods for injecting or inserting data feeds (e.g., one or more images, video streams, real-time video streams) (such as video capturing a participant (a person digitally imaged or captured by an imaging unit)) into a multi-layer scene to produce a video (e.g., a composite video) that approximates a three-dimensional space with depth by displaying the participant in front of one or more layers of the multi-layer scene and behind other layers of the multi-layer scene. The present disclosure describes such systems and methods: the systems and methods can be used to receive multiple media inputs (e.g., images, videos, etc.) to create a displayable scene, wherein the displayable scene includes multiple layers of graphics or visual data. The final output video of the multi-layer scene can be viewed using a display unit.

[0046] Among other things, the system displays, injects and / or positions data feeds at varying degrees of depth (i.e., each layer of a multi-layer scene, including in front of, behind or in between one or more layers of the multi-layer scene) so that the data feeds (e.g., data feeds representing participants) coexist, are placed or displayed on the display without obstructing at least a portion of one or more layers of the multi-layer scene. In addition, the system enables the injected data feeds to interact with at least a portion of one or more layers. For example, a participant or user can be captured as a data feed, and the participant's movement can result in a virtual interaction with an object presented in or by another layer of the multi-layer scene into which the participant image or video feed is injected. This merging of the multi-layer scene and the data feed is displayed on the display as an output video or output image. As a non-limiting embodiment, an object can appear as a solid object, an opaque object, a semi-transparent object, a transparent object, or an object displayed by a display with any varying degrees of transparency or semi-transparency.

[0047] With reference to the detailed description of the present disclosure, embodiments of the system and method mainly present the display of participant data feeds (i.e., data feeds representing participants such as users captured by imaging units, such as video streams of users) into multi-layer scenes. It should be noted that some details in the figures have been simplified and drawn for ease of understanding, rather than maintaining strict structural accuracy, details, and proportions. Specifically, video streams of people (i.e., users, participants, etc.) are described with simplified drawings or representations, rather than actual photographs, videos, or other types of data records of recorded subjects or objects. This is to easily and clearly demonstrate the functionality of the present disclosure. The visual representations contemplated herein are primarily directed to actual records of people, objects, or other visible things. The simplified descriptions of these in the figures are not literal descriptions of the present disclosure in actual implementation, but are representative.

[0048] In this specification and claims, the terms "user" and "participant" are used interchangeably.

[0049] The term “actual” or “physical” means “non-virtual” or “tangible” or “real life” as opposed to “virtual”.

[0050] In this specification, the terms "add" and "position" are used interchangeably with respect to users, participants, or participant layers. With respect to "adding to" or "positioning at" a voxel-based 3D scene, "adding" or "positioning" should be understood to include modifying data or values ​​of a set of voxels (3D pixels in xyz space) within the 3D scene.

[0051] The term "multi-layer" is also understood to mean "multi-depth" and "three-dimensional".

[0052] "Posture" and "gesture" are complementary features of the human body and are closely related. "Posture estimation or recognition" is the recognition or determination of the shape or posture of a human body, which can be understood to include the position and orientation of large body parts such as arms and legs, as well as finer details such as fingers, facial points, and eyes. Gesture recognition involves further interpretation of the posture information. For example, gesture recognition can interpret the posture information of a hand indicating an extended index finger and curled other fingers as a classic pointing gesture. When "posture estimation" or "posture recognition" etc. are mentioned, it can be understood that the step of "gesture recognition" may be followed. Similarly, when "gesture recognition" is mentioned, it can be understood that there is usually a step of "posture estimation" preceding it. A common posture estimation technique uses convolutional neural networks and heat maps to estimate the user's posture.

[0053] Reference now Figure 1 , a system for displaying data in a multi-layer scene, hereinafter also referred to as a data processing system 100), includes an imaging unit 102, a data acquisition unit 104, a storage unit 106, a control unit 108, and a display unit 110. Such system component units represent software units executed on a processor or processing circuit of a computer system, one or more hardware components that perform the methods described herein, or any combination of the two. Although Figure 1 The system component units are shown grouped into a single system, but such association is not limited to forming a single physical hardware unit (e.g., a single unit formed on a single computing device), but also represents a functional association of system units that together perform the methods described herein. The imaging unit 102 captures data feeds, such as images and videos. In some embodiments, the imaging unit 102 records or captures images or videos of participants (or users). The data acquisition unit 104 controls and receives the data captured by the imaging unit 102. The control unit 108 processes and practices the methods described herein using the storage unit 106. The display unit 110 outputs the processed result data as an output video.

[0054] Figure 2 and Figure 3 The present system is described in terms of a method for processing various data types (e.g., data feeds 204 and multi-layer (three-dimensional) input data 202) to display an output video 218 including a multi-layer scene. Although the process is described in a sequential manner, different input data can be processed separately or in parallel, and one input data does not depend on the processing or progress of another data feed. Figure 2As shown, the system (also referred to as data processing system 100) receives data feed 204 and multi-layer (three-dimensional) input data 202 through data acquisition unit 104 of data processing system 100. Data feed 204 (e.g., video stream 304) is acquired using imaging unit 102. During data acquisition phase 220, data acquisition unit 104 acquires input data (e.g., multi-layer (three-dimensional) input data 202 and composite input data 302). Data control unit 108 identifies various types of input data and data captured by imaging unit 102 and its scene layers 202, and extracts and merges them during data control phase 222. Finally, display unit 110 presents multi-layer scene 214 rendered as output video 218. In some embodiments, input data is three-dimensional input data 202 contained in a memory or data storage unit of a computer. In some embodiments, input data is multi-layer input data 202, which includes at least one set of independent but coordinated input scene layers 202. Multi-layer input data 202 with coordinated input scene layers can be time-synchronized input data with multiple layers (i.e., multi-layer input data). For example, the multi-layer input data includes a time-synchronized background layer 210 video feed and a subject layer 208 video feed. In some embodiments, the scene layers of the multi-layer input data 202 can be coordinated by associating various objects captured in the input data. For example, the background layer 210 can include an object that is associated or related to another object captured by the subject layer 208, and vice versa.

[0055] In some embodiments, the input data is composite input data 302 including various data (e.g., fixed images, slides, videos, etc.). Figure 3 As shown, composite input data 302 is processed to extract background and subject layers (ie, extracting layers 306). Thus, extracting layers 306 may be required when the input data does not contain layered data (eg, multi-layer input data 202).

[0056] In some embodiments, such as Figure 2 As shown, the system can also receive multi-layer input data 202 including multiple scene layers (e.g., background layer 210 and theme layer 208). In this embodiment, the multi-layer input data 202 includes two layers. The multi-layer input data 202 can be a collection of the background layer 210 and the theme layer 208. Multiple such layers enhance the three-dimensional effect and details of the multi-layer (three-dimensional) input data 202 scene. Similarly, the multi-layer input data 202 can be two separate input data of the background layer 210 and the theme layer 208.

[0057] From the data feed 204 (e.g., video stream 304), the system extracts an image (or set of images, such as a video clip) that captures one or more participants (i.e., extracts the participant 206). The data feed 204 (e.g., video stream 304) captured by these imaging units includes at least a portion of the participant, and the system processes it to isolate the portion of the data feed that includes at least a portion of the participant. In some embodiments, the extraction or isolation or segmentation of the participant may use a chroma key green screen, virtual green screen technology, skeletal / human pose recognition / estimation, a neural network trained to isolate the participant in the image or video stream, etc. Various known image processing and video processing techniques may be applied to extract or isolate the participant from the data feed 204 or video stream 304. The extracted participant representation is then added to a participant layer 212, which is composed of a non-transparent area that includes the isolated participant and a transparent area that does not include the isolated participant. In some embodiments, the extracted participant representation is added to the three-dimensional virtual scene by modifying the data associated with the appropriate three-dimensional position or voxel (three-dimensional or "volumetric" pixel) within the three-dimensional virtual scene. For example, see Fig.13A and Fig. 13B A three-dimensional scene 1302 in FIG.

[0058] Once a participant layer 212 is identified from a data feed 204 (e.g., a video stream 304), and multiple layers 202 are identified from the multi-layer input data 202 or the composite input data 302, the participant layer 212 including the isolated participant 206 is added in front of the background layer 210 and behind the subject layer 208, thereby generating a scene 214 containing three layers (i.e., a multi-layer scene). In other words, the participant layer 212 is added between the background layer 210 and the subject layer 208. This collection of at least three layers (background layer 210, participant layer 212, and subject layer 208) depicting visual or displayable data forms the multi-layer scene 214. A multi-layer scene refers to one or more layered visual representations of data, such as images and videos. The background layer 210, the participant layer 212, and the subject layer 208 are rendered / merged 216 (e.g., flattened) to render an output video 218, which is then displayed 224 using the display unit 110. In some embodiments, the layers in the scene 214 can be flattened so that any area of ​​the resulting output video 218 or image is composed of the video or image of the frontmost layer from the same area, where the frontmost layer of the area is opaque. If the frontmost layer of the area is translucent, the area of ​​the output video 218 or image is composed of a visual combination of the frontmost layer and the flattened layers recursively behind the frontmost layer (e.g., the colors of the layers are averaged, mixed, or otherwise visually combined). The display unit 110 can be a monitor, a projector mobile phone display, a three-dimensional display, augmented reality glasses, a virtual reality headset, a display presented by other wearable devices, or any other device that displays images or videos for visual presentation.

[0059] In some embodiments, such as Fig.13A and Fig. 13B As described in , the system can receive three-dimensional input data 1302 (also referred to as a "three-dimensional scene"), which includes objects at multiple locations in the three-dimensional space (including interactive or passive and movable or fixed, including fixed passive features of the environment, such as slopes or hills), such as objects 1310 further back and objects 1308 further forward. In this embodiment, the three-dimensional input data 1302 contains objects at two different locations in the three-dimensional space. The three-dimensional input data 1302 can be a collection of objects 1308 and 1310 located in the three-dimensional space. Such objects located at multiple locations enhance the perceived richness and details of the three-dimensional input data 1302. Similarly, the three-dimensional input data 1302 can also be sparsely populated with very few objects. Among other properties, the three-dimensional objects 1308 and 1310 can have different transparency and opacity levels-which will be taken into account during rendering 1316. The three-dimensional input data 1302 can be stored in the storage unit 106. The components of data processing system 100 communicate directly or indirectly through various communication protocols.

[0060] From the data feed 204 (e.g., video stream 304), the system extracts 206 an image (or set of images, such as a video clip) capturing one or more physical participants 1301. The data feed 204 (e.g., video stream 304) captured by these imaging units includes at least a portion of the participant, which the system processes to isolate the portion of the data feed that includes at least a portion of the participant. In some embodiments, the extraction or isolation or segmentation of the participant can use chroma key green screen, virtual green screen technology, skeleton / human pose recognition / estimation, neural networks trained to isolate participants in images or video streams, etc. Various known image processing and video processing techniques can be applied to extract or isolate participants from the data feed 204 or video stream 304.

[0061] Once the extracted participant representation 1312 is identified from the data feed 204 (e.g., video stream 304), and the three-dimensional scene 1302 is available, the extracted participant representation 1312 is positioned 1313 within the three-dimensional scene 1302. This positioning is accomplished by modifying the data or values ​​of the appropriate voxels (or three-dimensional pixels) within the three-dimensional space / scene, thereby generating a three-dimensional virtual scene / space 1314 that includes the various objects plus the participant representation. In other words, the participant representation 1312 is located within the three-dimensional scene objects (e.g., 1308 and 1310).

[0062] Positioning and updating step 1313 may optionally (as shown by dashed lines) also involve taking as input additional participant information identified from video stream 304, including but not limited to position / depth information 1309 (information about the physical position of the participant relative to the camera, including depth or distance and lateral and vertical position, in other words, x, y, z coordinates relative to the camera) and posture and / or gesture information 1311 about the physical posture and / or gesture of the participant. Position / depth information 1309 may be determined 1303 based on the user captured in video stream 304. Posture and gesture information 1311 may be identified 1304 based on the user captured in video stream 304.

[0063] As part of the update, in step 1313, the system may also take additional participant information as input and utilize the additional participant information to update the scene 1302 based on the position, posture, or gesture of the participant 1301. For example, by utilizing the position / depth information 1309 and / or the posture and gesture information 1311, the positioning and updating step 1313 involves determining whether and how the virtual participant is in contact with an interactive object in the three-dimensional scene to trigger a corresponding action. After the action is triggered, in step 1313, the system updates the three-dimensional scene accordingly to reflect the action (e.g., visual state changes, etc.). For example, in one embodiment, if the virtual position and posture of the virtual participant indicates that the virtual participant is in contact with a button in the three-dimensional virtual space that changes color when pressed, then in step 1313, the system considers that the button is pressed, triggers and performs a corresponding action, and updates the virtual three-dimensional space or scene to reflect that the button has changed color. As another example, in another embodiment, if it is determined that the physical participant is making a pinch gesture with his fingers, as if holding a pen, then in step 1313, the system maps the physical position of the physical participant to the corresponding virtual position of the virtual participant in the virtual three-dimensional space, wherein the recognition of the pinch gesture (i.e., the pen gesture) causes the system to update the virtual three-dimensional space in step 1313 to display the writing or drawing of the virtual participant in the virtual three-dimensional space in the virtual position of the hand (or the virtual pen held by the hand). Therefore, the system uses certain visually recognizable changes of the user 1301 (such as the additional participant information described above) to cause the positioning or updating, as described in step 1313. In other words, the user can control the participant representation 1312 by introducing visually recognizable changes to the system through the video stream 304 data input using the output video 218 provided by the system.

[0064] In some embodiments, the imaging unit or camera is positioned to fully capture the user to provide a complete view of the user included in the video stream.

[0065] In some embodiments, the imaging unit or camera is positioned to capture at least a portion of the user himself to provide a partial view of the user included in the video stream.

[0066] A three-dimensional scene refers to a three-dimensional space with one or more objects at multiple depths or distances from a viewpoint. Objects 1310 further back, participant representations 1312, and objects 1308 further forward are rendered (step 1316), such as by rendering output video 218 via 2D projection (e.g., orthogonal projection, oblique projection, perspective projection, etc.), and then displayed 224 using display unit 110. Display unit 110 may be a monitor, a projector, augmented reality glasses, a virtual reality headset, a display presented by other wearable devices, or any other device that displays images or videos for visual presentation. In some embodiments, such as when display device 110 is capable of receiving three-dimensional content, rendering 1316 may enable the generated three-dimensional scene 1316 to be directly transferred to display device 110 for display substantially unchanged.

[0067] Computer program instructions can be stored in a computer-readable medium, which can instruct the system, its components or other programmable data processing devices to operate in a specific manner, so that the instructions stored in the computer-readable medium constitute a manufactured product, including instructions and processes for implementing the functions / actions / steps specified in the flowcharts and / or block diagrams provided herein. These computer program instructions can be provided to the processor of the system so that the instructions create a device for implementing the functions / actions specified in one or more boxes of the flowcharts and / or block diagrams. In the following description, reference is made to the accompanying drawings, which are illustrations of embodiments of the disclosed system that can be practiced. The system of the present disclosure is configured to specifically perform the methods envisioned in the present disclosure. However, it should be understood that those skilled in the art can develop other structural and functional modifications without departing from the novelty and scope of the present disclosure.

[0068] In some embodiments, such as Fig.16 As shown, the system acts as a user control mechanism that enables a physical user to control their virtual self (i.e., a virtual user or user representation generated on a display) in a virtual three-dimensional world and interact with objects in the virtual world by displaying to the physical user a real-time representation of the virtual user as it is seen in the virtual world. Where the user sees an object they want to interact with, the user can move their body (or make gestures to a similar effect) in the physical world, causing their virtual user representation to likewise move its body to take such action in the virtual world.

[0069] like Fig.16 As shown, in some embodiments, the user control mechanism operates as follows:

[0070] In step 1602, the system receives video data, such as video frames (containing images of actual users), from an imaging device. The imaging device may be a two-dimensional or three-dimensional camera or multiple cameras. Types of cameras include, but are not limited to, webcams on laptop computers and cameras on mobile phones (including "front-facing" or "selfie" cameras on mobile phones).

[0071] In step 1604, the system isolates the user representation from the video data received in step 1602. In some embodiments, the isolation is accomplished through human segmentation or "selfie segmentation," which is commonly used by video conferencing software to isolate or segment the image of the user from the image of their actual surroundings so that it can be removed and replaced with a more desirable background.

[0072] Once the image of the user is isolated, some embodiments of the system can use all or part of the image as a user representation (also referred to as a "virtual user" or "virtual user representation"). In some embodiments, the virtual user representation can be entirely a "real life" image of the physical user. In other embodiments, the virtual user representation may be partially a "real life" image (e.g., only the face), while other parts of the virtual user may be virtual (e.g., the body of an avatar) or enhanced (e.g., wearing virtual clothes or holding a virtual object). And, in some embodiments, the virtual user can be completely virtual, such as a virtual avatar (including but not limited to an avatar representation of a physical user). In some embodiments, the user representation can be flat (like a paper cut) or have a fixed thickness. In other embodiments, it can be round or more similar to an actual human body. In some embodiments, such as an avatar user representation, the user representation can be based on a large number of images of the captured user, whether in an explicit training step (e.g., requiring the user to stand in front of the camera in a "T pose" and possibly rotate) or by capturing a large number of images during the normal process of the user using the system.

[0073] After receiving the user representation in step 1604, in step 1606, the system adds the user representation or virtual user to the three-dimensional scene. A three-dimensional scene, also referred to as a "virtual world / scene / space" or "three-dimensional space", is a three-dimensional space that may also contain various features, such as objects, environmental features (e.g., hills, mountains, trees), other virtual users, etc. The three-dimensional scene / space may be a three-dimensional representation composed of voxels (i.e., three-dimensional or "volume" pixels) and / or vectors with xyz coordinate positions and stored in computer memory. The three-dimensional scene may be static or dynamic. In some embodiments, it may be generated by a virtual reality application (e.g., a game).

[0074] In some embodiments, step 1606 may involve adding the virtual user representation to the three-dimensional scene on a "frame-by-frame" basis, thereby allowing the physical user to control the virtual user through the actual physical user's own frame-by-frame movements or actions. Fig. 20B and Fig. 20C As shown, in some embodiments, rather than adding a virtual user representation for each frame and re-adding the virtual user representation again, step 1606 may include step 1606a, i.e., initially instantiating and adding the virtual user representation once as an object in the three-dimensional scene, and then continuing to update the virtual user representation object in the three-dimensional scene in subsequent iterations (step 1606b), which may correspond to subsequent "frames". In this way, the virtual user representation object is only added and instantiated once (or less frequently than each frame) and updated (as frequently as each frame), rather than being re-instantiated and re-added for each frame.

[0075] After receiving the three-dimensional scene that now includes the virtual user or user representation (e.g., via a pointer to a computer memory storing the scene), the system proceeds to display the three-dimensional scene back to the physical or actual user in step 1608. This real-time feedback to the user enables them to continue to control their virtual self in the virtual three-dimensional world. In some embodiments, the display is performed via a two-dimensional display (e.g., a 2D monitor). In such a display, the user is able to see their image in the virtual world. (When displayed on a 2d monitor, the three-dimensional scene (including the user representation) is first rendered using a typical two-dimensional projection method (e.g., orthogonal projection, oblique projection, perspective projection, etc.).) In other embodiments, the display can be performed via a three-dimensional display device.

[0076] After step 1608, the process loop repeats, starting a new iteration from step 1602 to continue providing the user with control and visual feedback to control their virtual self in the three-dimensional virtual world.

[0077] Fig.21 Shows the application Fig.16 The user physically stands in front of the camera in her actual room. In step 1602, the system receives a first frame of video from the camera. After receiving the video, in step 1604, the system separates the user's image from the video through human body segmentation and continues to use the image as part of the user representation. More specifically, in Fig.21In the embodiment shown, in step 1604, the system uses three copies of the user's body image (with fixed positions and scale offsets) as user representations. Given an "empty" (i.e., no virtual user representation) three-dimensional scene or space of a virtual ballroom, the system continues to add user representations (e.g., for this particular embodiment, three copies of the user's current body image with the latest video frame from the camera. For clarity, any number of copies of the user's current body image (including a single copy) can be used as user representations) to the three-dimensional scene of the virtual ballroom in step 1606. In step 1608, the system renders the three-dimensional scene into a 2D image, which is then displayed on a display unit (e.g., a monitor screen) for the user to view. Seeing what her current virtual user representation looks like in the virtual world, the user will slightly adjust her body position and posture. When she does this, the process loop repeats starting from step 1608, which receives the next frame of video with the user's adjusted posture and position (i.e., a visually recognizable change in the user). Based on the latest video frame as input, the process proceeds again to: step 1604, separating the user image and generating a user representation (three images with the user's body), step 1606, adding the latest user representation to the empty virtual scene, and step 1608, displaying the latest 2D rendering to the user, reflecting what the virtual user now looks like in the virtual three-dimensional space.

[0078] like Fig.17 As shown, some embodiments may further include step 1710 for updating the three-dimensional scene. In some embodiments, step 1710 involves updating the three-dimensional scene in addition to the virtual user representation. In some embodiments, step 1710 involves updating the three-dimensional scene, including updating or modifying the virtual user representation.

[0079] In step 1710, the system makes the three-dimensional scene dynamic. In some embodiments, in step 1710, the system updates the scene based on the passage of time or randomly over time. For example, if there is no step 1710 in the process, virtual users can be added to the virtual space of a static dance hall that never changes. If step 1710 is part of the process, the virtual dance hall can have a rotating disco ball and a floor composed of tiles that pulsate in different colors. In some embodiments, the three-dimensional scene can be updated based on external inputs (such as external weather, stock market, or signals sent from a keyboard or game controller).

[0080] like Fig.18AAs shown, some embodiments may include a step 1812 of identifying the user's physical location. The location may include, but is not limited to, one or more of a lateral or "x" position, a vertical or "y" position, a depth or "z" position, and a rotation or orientation relative to all three axes. In some embodiments, the location is determined relative to a camera or imaging unit. In other embodiments, the location is determined relative to a room or physical space in which the user is located. The location may be determined by techniques such as body tracking, depth estimation, and the like.

[0081] With respect to depth estimation, in some embodiments, a depth-sensing imaging unit (including but not limited to a time-of-flight (TOF) camera / sensor, a stereo camera, multiple cameras used together, a 2D camera complementary to a lidar, a camera on a mobile phone with depth sensing technology, etc.) may be used to extract depth information from a scene captured by the imaging unit. In other embodiments, an imaging unit (e.g., a two-dimensional camera) may be used in conjunction with the recognition of depth cues in a scene to extract depth information from the scene. Examples of depth cues include the distance between detectable features on the human body, such as the distance between the eyes or various joints of the body. These features appear larger when they are close to the camera and smaller when they are far away from the camera, which can help estimate the depth or "z" distance from the user.

[0082] After generating the position information in step 1812, the system may utilize the information to determine where to position the user representation or virtual user when adding the user representation or virtual user to the three-dimensional scene in step 1606. In one embodiment, a physical participant moving one meter to the left (as determined in step 1812) may cause the system to add the virtual user to the position moved to the left in the virtual space in step 1606. However, in another embodiment, the same movement to the left by the physical user may cause the system to add the virtual user representation to the right (rather than the left) in step 1606, indicating a "mirror" effect, or in another embodiment to add the user to an entirely different area within the three-dimensional space, which may indicate a "wormhole" or "teleportation" effect.

[0083] The changes in position and orientation provided by the system in step 1812 may be referred to as "shifts". In some embodiments, step 1606 may involve "mapping" certain types of shifts of the actual physical user to specific corresponding types of shifts represented by the virtual user. In some embodiments, the left-to-right shift of the user may cause the virtual user representation to shift from left to right, while in other embodiments it may be the opposite. In some embodiments, the user's forward movement toward the camera or imaging device may cause the virtual user representation to shift forward, and vice versa. Alternatively, in other embodiments, the user's shift to the camera (i.e., the change in depth or "z" distance) may be completely ignored (i.e., not mapped to the corresponding virtual shift), so that the virtual user representation can only shift left and right but not forward and backward. Changes in orientation or rotation can also be mapped or completely ignored in various ways. It can be seen that the position and / or shift mapping between the physical world of the actual user and the virtual world of the user representation can vary with the specific implementation scheme implemented.

[0084] Fig. 22 An example embodiment of a user playing a virtual reality game is shown. Fig. 22 In the following description, Fig.19 The steps presented in Fig. 22 For example, on the display screen of her laptop or mobile phone, the user sees her virtual self in the virtual world. Fig. 22 As can be seen in the four snapshots shown in , after seeing her virtual self on the screen, the user calibrates and controls the movement of her virtual self frame by frame to carefully walk from a virtual stone on the left to another virtual stone on the right without taking a wrong step and falling into the water.

[0085] Regarding the first snapshot 2201: The user sees her virtual self on the display screen needing to step over an adjacent stone. The actual user physically extends her first foot. In step 1602, the system receives the first frame of video (captured by the front-facing "selfie" camera of the user's laptop or mobile phone) that shows the actual user in her actual room. In step 1604, the system isolates or segments the user image from the image of her in her room and continues to use the user image as the user representation. In step 1812, the system determines the actual user's physical location (including depth) in her actual room. After mapping that physical location to a corresponding virtual location in the virtual scene, in step 1606, the system adds her image (i.e., the user representation) to the virtual three-dimensional scene just above the left stone. Then, in step 1608, the system renders the three-dimensional scene from the 3D representation into a 2D image for display on the user's laptop or mobile phone screen. And, the next iteration of the process loop is ready to start again at step 1602.

[0086] With respect to the second snapshot 2202, the user has seen in the 2D image through the display unit that the virtual user has extended her feet to the correct direction and position (as shown in the first snapshot 2201), thereby receiving positive feedback, and the real user continues to put her physical feet down so that her virtual user has now firmly placed her feet on the two stones (as shown in the second snapshot 2202). This is achieved by the real user putting down her physical feet, the system capturing the image in step 1602, isolating and segmenting the image from the rest of the room in step 1604, determining her actual position (including depth) in step 1812, adding the virtual user representation (her image) in the appropriate position (just above the gap between the two stones) in step 1606, and rendering and displaying the updated three-dimensional scene as the second snapshot 2202 in step 1608. And the next iteration of the process loop is ready to start again in step 1602. This is an embodiment of the user controlling his virtual self by using the displayed scene as feedback. As the user moves, the system causes the corresponding movements to be reflected in the virtual self displayed on the display.

[0087] With the positive feedback that she appears to have successfully crossed from one step to the next, during the next two snapshots 2203 and 2204, the process loop (e.g. Figure 16-19 2204) can continue to repeat and iterate, we see the virtual user (controlled by the physical user) first shift her weight to the right stone 2203, and then move her feet to the right stone 2204. This is an example of a user controlling their virtual self by using the displayed scene as feedback. As the user moves, the system causes the corresponding movement to be reflected in the virtual self displayed on the display.

[0088] As shown in Figure 18b, some embodiments may include step 1814 of identifying a posture or gesture of the user from the image of the user in the video data. Posture estimation may be performed by various well-documented techniques. Gesture recognition (also well-documented) may be used to interpret posture information to identify various gestures of interest.

[0089] After recognizing the posture and / or gesture information in step 1814, the system may use the information to determine where to position the user representation or virtual user when adding it to the three-dimensional scene in step 1606. For example, the system may recognize the user alternately bending each knee as making a walking gesture with his legs in step 1814. In response, in some embodiments, for each consecutive video data (i.e., video frame) processed by the repeated loop of the process, the system may add the virtual user moving forward from its previous virtual position for each iteration of the loop (i.e., for each video frame) in step 1606, thereby moving the position forward as if the virtual user is walking through the virtual three-dimensional scene. In another embodiment, the user system recognizes in step 1814 as making a jumping gesture may be added upward from its previous position for the next few consecutive iterations of the loop to show the virtual user jumping upward in the virtual three-dimensional scene. In another embodiment, the system's recognition of the user's facial gesture or expression as a frown in step 1814 may cause the system to generate a user representation with a blue hue in step 1604, while the system's recognition of a smile in step 1814 may cause the system to generate a user representation with a yellow hue in step 1604.

[0090] In another embodiment, Fig. 20A As shown, in step 1814, using the video stream 304 captured by the imaging unit 102, the user 1301 with the extended arm 2002 can be identified by the system as having a superhero flying posture. In response, the system outputs a user representation 1312 in step 1604, in which the actual image of the user is, for example, wearing an enhanced virtual superhero cape. Then, in successive iterations of the process loop (first iteration 1312a, second iteration 1312b, and third iteration 1312c), the system in step 1606 will add virtual user representations in each pass of the loop (1312a, 1312b, 1312c) in an incremental manner upward and forward iteration to move the virtual user representation 1312 as if flying in the three-dimensional scene 1302. The three-dimensional scene 1302 may include three-dimensional scene objects, such as 1308 and 1310. This is an illustrative embodiment of a user posture / gesture and the system detecting such posture / gesture to trigger or cause a programmed corresponding action or predefined action.

[0091] like Fig.19As shown, various optional steps may also be combined. In some embodiments, as part of updating the three-dimensional scene, step 1710, the system continuously or continuously monitors the interactions or collisions between all virtual users and all objects in the three-dimensional scene, and also maintains / updates the state of each object. For certain objects, such as the rotating disco ball in the aforementioned embodiment, step 1710 may include the system continuously updating it to keep it moving. For certain passive objects, such as walls, step 1710 may include steps to prevent virtual users from being able to walk through it. For interactive objects, step 1710 may include steps to monitor and detect interactions between virtual users (or other objects) and objects to trigger and perform any necessary actions.

[0092] In some embodiments, the process step of updating the three-dimensional scene (step 1710) can be as follows Fig.28 The system detects interactions between a virtual user (or user representation) and an interactive object (step 2802), interactions between multiple virtual users (step 2808), interactions between objects (step 2810), and external triggers (step 2812), such as the passage of time, weather, button presses, game controller input, etc. When the system detects such interactions and / or triggering of actions associated with the interactive object and then performs those triggered actions, the system updates the object accordingly in step 2804 and updates the virtual user accordingly in step 2806. The virtual user (or user representation) itself is also considered an object. Therefore, interactions between objects include interactions between virtual users and objects. Similarly, interactions include interactions between virtual users.

[0093] In some embodiments, step 1710 may include a step of updating the three-dimensional scene using the position (including depth) and / or orientation data from step 1812. In step 1710, the system may do this by detecting the triggering of an interactive object using the position of the physical user and / or the virtual position of the virtual user representation, and then performing the associated action. In one embodiment, for example, consider a situation where the virtual user representation is standing in front of a virtual automatic sliding door in a virtual three-dimensional scene. When the system detects in step 1812 that the actual physical user has moved forward in their physical space (i.e., the depth information has changed), the system adds the virtual user to the three-dimensional scene in step 1606, one step forward from their previous position. When it is detected that the virtual user is now directly in front of the virtual automatic sliding door, the system determines in step 1710 that the sliding door interactive object is to be triggered, and therefore performs the associated action, i.e., the sliding door opens. Therefore, the system updates the three-dimensional scene in step 1710 by sliding the virtual door open.

[0094] In other embodiments, the system in step 1710 may update the three-dimensional scene using the posture and / or gesture information provided by the system in step 1814. For example, in one embodiment of a virtual whiteboard application, the system may recognize a long left-to-right swipe gesture of an open palm (over the course of consecutive video frames) as an "erase all" gesture in step 1814. When the system provides information that the "erase all" gesture has occurred, the system continues to update the virtual three-dimensional scene in step 1710 by removing any virtual writing or drawing that may have been done. This removal is accomplished by returning the voxels involved in the writing (a "voxel" refers to a three-dimensional or "volume" pixel) to their original "unwritten" state.

[0095] Continuing with the virtual whiteboard example, consider the case where the system determines in step 1814 that the physical user is making a pinch gesture with his fingers, as if holding a pen. In this embodiment scenario, the system also determines the physical location of the physical user within his physical space in step 1812. Through the system executing steps 1812 and 1814, the system maps the physical location to a virtual location within the three-dimensional virtual space in step 1606 and adds a virtual user representation therein. In step 1814, the system's recognition of the pinch gesture (i.e., pen gesture) causes the system to update the virtual three-dimensional space in step 1710 to display the writing or drawing of the virtual user's hand (or hand holding the virtual pen) at the virtual location in the virtual three-dimensional space (i.e., by changing the color of the appropriate voxels).

[0096] Fig.29 A process flow of one embodiment of a virtual whiteboard system 2912 is shown. In step 2902, the system recognizes a pen gesture 2908 (e.g., pinching fingers) made by a user. In step 2904, the system updates the scene by drawing or modifying the color of voxels (or pixels in the 2D case) at the three-dimensional location of the user's represented fingers 2910. In step 2906, the system displays the updated scene. Fig.29 The process can be repeated.

[0097] As another example, if the virtual position and posture of the virtual user indicate that the virtual user is in contact with a button in the three-dimensional virtual space, the system detects that the button is pressed, and then triggers and performs a corresponding action. For example, such a corresponding action can be to change the color of the button in response to the participant pressing the button (i.e., by identifying the position and posture of the user indicating contact with the virtual button). The virtual three-dimensional space or scene is updated to change the color of the button.

[0098] In some embodiments, the system may update the virtual user itself in step 1710. In one embodiment, a virtual user who is too close to a virtual mine (interactive object) may trigger a programmed corresponding action, which may be an interaction with the virtual mine causing it to explode in the virtual three-dimensional world. Such an explosion may cause damage to other objects in its vicinity. To represent the explosion effect, in this embodiment, the system updates the appearance and other characteristics / attributes (e.g., geometry, rotation, health, armor, speed, strength, etc.) of nearby objects in the three-dimensional scene in step 1710. One such affected object may also be the virtual user representation itself. In this case, the system may present an update to the virtual user representation 1312 in step 1710 by animating the virtual user representation to be thrown back from the explosion location and placing the virtual user in a lying position (even if the actual physical user is still standing). Finally, in the next iteration of presenting a series of such three-dimensional scenes, the system may update the virtual user in step 1710 to make it stand up again, but the system may update the representation in step 1710 to make it look bloody or injured. In addition, one arm of the virtual user representation may no longer be movable (regardless of the posture of the actual user). The system may also change the characteristics of the virtual user representation in step 1710, slowing down its movement because it is now injured.

[0099] Fig.23 An example implementation is shown in which a user utilizes gestures to grab and move an interactive object 1208 (a piece of candy in this example) as part of a game played in a virtual scene. In the first snapshot 2301, the user approaches a yellow rectangular candy (interactive object 1208) with an open hand. The system captures a frame of a video feed of an actual user in his physical room from an imaging unit (e.g., a mobile phone's camera). The system then isolates or segments the user image from the frame and proceeds to use it as a user representation 1312. The system recognizes the user's hand pose or gesture as an open hand, indicating the start of a "grab" gesture. The system adds the user representation 1312 (i.e., the user image with the open hand) to the scene. The system has not detected any object interactions yet, so the scene will not be updated further. The system renders and displays the image 2301 for the user to view on his phone screen. The process loop is then ready to start the next iteration.

[0100] Regarding the second snapshot 2302, the user sees the open hand of his virtual self on the display behind the interactive object 1208, and then the real user continues to physically clench his fist. In this embodiment, the user is able to control his own virtual representation 1312 by using the scene displayed on the display as positive feedback. The system captures a video frame of the real user in this posture. The system isolates the most recent user body image (i.e., clenched fist). The system recognizes the clenched fist (which was an open hand before) as indicating a "grab" gesture. The system adds the most recent user representation (i.e., the user image of the clenched fist) to the scene. Although the system detects object interaction (i.e., grabbing), this embodiment may not need to redraw the candy, so there is no need to update the scene further. The system renders and displays the image 2302 for the user to see on the display of his mobile phone. And the process loop is ready to start its next iteration.

[0101] Regarding the third snapshot 2303, with the feedback that his virtual self 1312 has grabbed the correct piece of candy, the real user continues to move his fist to the right. The process proceeds the same as before. The system recognizes the moving fist as a "drag" gesture. Therefore, the system updates the scene this time to show that the interactive object 1208 is being dragged to the right. The system displays this scene to the user before the process loop continues to start the next iteration.

[0102] With respect to snapshot 2304, the actual user opens his fist with visual feedback that his virtual self has successfully dragged the virtual candy 1208 to its desired final location. Display image 2304 shows that the user's virtual user representation 1312 has released the candy (interactive object 1208) to its final desired location. This is an example of a user's gesture / posture triggering a programmed action of the interactive object 1208.

[0103] Fig.24 Another example of an embodiment is shown in which a user is able to control and use an interactive object to trigger the action of another interactive object. In snapshot 2401, although most of the body represented by the user is obscured by the handle (interactive object 1208) that appears in the center of the snapshot, the raised arm can be seen throwing (or whipping) a virtual whip (interactive object 1208) held in its hand. The system draws and updates the virtual whip in each iteration of the process loop. In snapshot 2402, it can be seen that the whip 1208 has been wrapped around the handle in the center of the snapshot. The system detects the contact interaction between the whip and the handle and updates the scene to show that the whip is wrapped around the handle. Then, in snapshot 2403, when the user makes a gesture to pull the whip 1208 back, the virtual user 1312 pulls the virtual whip 1208 back, thereby pulling down the virtual handle. After the system recognizes the gesture, the system updates the scene to reflect the whip pulling down the handle.

[0104] Although the above Figure 16-19 The description discusses embodiments involving three-dimensional virtual spaces, but it should be noted that some embodiments implement the same or similar concepts using two-dimensional virtual spaces.

[0105] In some embodiments, the viewpoint of each virtual user is from a third-person perspective facing forward of the user. In some embodiments, the viewpoint (from which the rendering of the three-dimensional scene is done) moves as the virtual user maintains a fixed distance in front of it. In other embodiments, the viewpoint is stationary. And, in other embodiments, the distance of the viewpoint from the user and the viewing angle vary from case to case.

[0106] In some embodiments, the view of each virtual user is from a third-person perspective from behind the user. In this case, the user representation may use or display a backside (or "from behind") image or representation of the user. In some embodiments, this representation may be extracted from a stored body representation, similar to the body truncation discussion and Fig.15 In some embodiments, such as for applications or games that require a rear view of the user, some embodiments may utilize an explicit image capture step, including requiring the user to show their back to an imaging device to capture a rear view, so that the system can have a rear view representation of the user. The rear view can then be used as the representation of the user that the user sees in their virtual self display in the virtual world.

[0107] Multiple people and multiple users

[0108] In some embodiments, multiple virtual users may be added to the same virtual three-dimensional scene, and the multiple virtual users may participate in and interact with the same virtual three-dimensional scene.

[0109] In some embodiments, multiple users can remotely access the system from different locations on the network. In this case, Figure 16-19 Some steps in the process loop example embodiment shown in can be performed on the cloud, the user's local device, or a combination. Likewise, the steps of the process loop can be performed in a different order.

[0110] Fig.25An embodiment in which multiple users are remotely added to the same virtual space and participate in a game is shown. In embodiments such as this, the view of each virtual user is from a third-person perspective facing the front of the virtual user. In embodiments such as this, the virtual user representation is a mirror image of the actual user, because it usually feels more natural and intuitive for people to see their mirror images (for example, when the actual user moves their actual left hand, they will see the virtual hand that appears on the left side from their perspective move, even if from the perspective of the virtual user, this is technically the right hand of the virtual user). Through their user representation combined with their respective faces, each user can see the actual expressions on each other's faces, while also communicating with them through the audio that the system transmits between nearby virtual users. This gives users an enhanced sense of co-presence. In addition, each user is able to see their own facial expressions, which allows them to see how others will see them. In addition, this sense of co-presence can be enhanced by displaying a virtual scene containing two users to each user on a display.

[0111] In some embodiments, multiple users can access the system locally through the same imaging device.In some embodiments, any user-specific steps described herein can be performed separately for each user.

[0112] To prevent unrelated or unexpected persons (e.g., passers-by or bystanders who appear in the camera view) from being considered as intended users, the system can filter any persons detected in the image. In some embodiments, the following is performed on each person detected in the video data (or video frame). Fig.26The person filtering method shown. The system detects people appearing in the video frame in step 2602. For each person detected, the system compares the person with the expected user list in step 2604. In some embodiments, the expected user list contains at least a partial body image of the expected user. Well-known image comparison methods can be used for comparison, such as facial comparison, feature comparison, color histogram comparison, etc. In some embodiments, the expected user list can be filled by an explicit expected user registration process, such as the expected user registration process that may be prompted by the user interface before starting to use the system. In other embodiments, the expected user list can be implicitly filled by taking images of the most recent and / or most common recurring users. Once the comparison is completed in step 2604, the system determines in step 2606 whether the detected person is actually the expected user. If so, the process proceeds to step 2608, thereby allowing the detected person and his image to continue as a user, thereby isolating and adding his user representation to the virtual scene. If not, the process proceeds to step 2610, thereby not allowing the detected person and his image to continue as a user and not allowing his user representation to be isolated or added to the virtual scene. After the filtering process has been successfully performed on all detected persons, any unintended persons should not be included in the virtual three-dimensional scene or appear in the display of the scene.

[0113] like Fig. 27 As shown, some embodiments may apply the people filter only when the number of detected people is greater than the expected number of people. In step 2702, the system detects the number of people in the video image. In step 2704, the system compares the number of people to the expected number of users. In some embodiments, the expected number of users may be predetermined by the application - for example, a single-player game only expects one user. In other embodiments, the expected number of users may be explicitly received by a user interface presented to the user when the application starts. In step 2706, the system considers the results of step 2704 to determine whether the number of detected people exceeds the expected number of users. If so, the following is run for each detected person. Fig.26 If not, all detected persons are allowed to be considered users.

[0114] Figure 4Embodiment scenes 402-404 from output videos that can be presented using a display unit are shown. In scene 404, a subject layer is overlaid in front of a participant layer. Participants 406 are presented in the participant layer, wherein the subject 408 of the subject layer is not obscured by the participants. Background layer 410 is presented behind the participant layer. As shown, each scene layer can include transparent portions so that the output video plays a rendered multi-layer scene depicting participants 406 and subject 408 of the subject layer on background layer 410. In this embodiment, the subject layer is displaying subject 408, i.e., a graphic located in front of the participant layer, wherein the participant layer displays isolated or extracted participants 408. Background layer 410 is located behind all other layers. Scene 402 shows a scene before a participant layer is employed between the subject layer and background layer 410. In this scene, a virtual participant 408 of the participant layer is not located between the subject layer and background layer 410. In some embodiments, background layer 410 and subject layer are extracted from composite input data. In some embodiments, background layer 410 and subject layer are derived from multi-layer input data.

[0115] In some embodiments, a three-dimensional effect is achieved by identifying multiple layers from the composite input data (e.g., presenting an output video of multiple layers, wherein the depth of each layer is different from each other). The composite input data may be analyzed to identify a background layer and a subject layer. Each frame of the output video is composed of at least one background layer behind a participant, at least one participant layer in front of at least one background layer, and at least one subject layer in front of at least one participant layer. In other words, the output video includes at least three different levels of scenes, i.e., a multi-layer scene. Similarly, the output video may include one or more scenes with three or more different layers, wherein the layers are positioned or stacked (i.e., the depth position within the layer / layer stack) to create a three-dimensional effect. Although the example embodiments of the present disclosure shown in the figures mainly show a multi-layer scene with three layers in the scene (e.g., a background layer, a participant layer, and a subject layer), multiple layers may constitute a multi-layer scene. For example, multiple layers may be located between a background layer and a subject layer, thereby creating a three-dimensional effect or different degrees of depth (i.e., different depth positions of the layers). In another embodiment, a multi-layer scene may include multiple layers of background layers to create different degrees of depth in the background, such as placing an image of a mountain (or a frame of a video) further behind an image of a tree. Similarly, multiple layers can be introduced into the scene theme layer. In other words, the multi-layer scene can include multiple background layers, multiple participant layers, and multiple theme layers. By composing a multi-layer scene with multiple layers, the visual effect of achieving a three-dimensional effect can be enhanced.

[0116] Reconfigure the slide using presentation software. The layers of the multi-layer input data can be identified in several different ways. In one embodiment, the presentation software (such as Powerpoint or Google Slides) can be configured so that the slide is a multiple of its typical size (corresponding to the number of layers it contains). Figure 5 As described, the slide is configured to be twice its typical size to include a theme layer 208 and a background layer 210 on the same slide. One half is used as the background layer 210 and the other half is used as the theme layer 208. Since this is achieved using the presentation software itself, each half retains all the features (e.g., animation, etc.) provided by the presentation software. The theme layer can utilize specific colors to indicate transparency in the layer area, while the lack of a specific color in the layer area indicates opacity or translucency. The opaque or translucent area indicates the theme in the theme layer (i.e., the subject material of interest). In the present embodiment, the graphics are displayed as the subject material of interest in the theme layer. And the background layer provides an aesthetic design element.

[0117] In some embodiments, using presentation software, a slide of typical size is used in conjunction with at least one predetermined automatically identifiable placeholder data location that indicates the location of the participant layer. Figure 6 As shown, a placeholder data location 606 is inserted into the slide and added in front of any content assigned to the background layer 610 and behind any content assigned to the subject layer 608. The placeholder data location can also be resized and reshaped on the slide to indicate the area of ​​the participant layer to which the participant layer is to be added. Once the placeholder data location 606 is specified within the participant layer, the resulting output video has a multi-layer scene, including a participant layer 612 located within the specified placeholder data location 606. When displayed, the participant layer is displayed in place of the placeholder image. In some embodiments, the placeholder image is replaced with the participant layer during positioning or stacking of layers before the output video or image is rendered. In some embodiments, an image or frame of the output video containing the placeholder image is rendered, and after the system recognizes the placeholder image, the visible portion of the placeholder image is replaced with the corresponding portion of the participant layer 612.

[0118] User interface of presentation software. In some embodiments, the user interface of presentation software (e.g., Powerpoint or Google Slides) is utilized, redesigned, or modified to include a participant layer in the scene. The participant layer can be positioned on the slide and positioned in front of or behind other topics in the subject layer or other layers using presentation software options (e.g., "Send backwards", "Send to back", "Bringforwards", and "Bring to front"). In some embodiments, a user interface is implemented to enable a user to manipulate a slide that has been made using presentation software (e.g., Powerpoint or Google Slides). The user interface displays the topics or objects on the slide in a way that clarifies which topics or objects are on which layers of the slide (i.e., which objects are in front of or behind other objects). The user interface then allows the user to select where they want to place at least one participant layer in these layers - that is, before which layer, after which layer, or between which layers. In some embodiments, automatic layering is used. Automatic layering involves using an automated method to distinguish between background layers and subject layers and inserting a participant layer between them.

[0119] The system can use artificial intelligence and deep learning techniques to identify background layers and theme layers. In some embodiments, artificial intelligence or deep learning can be used to identify background layers and theme layers. In some embodiments, this is achieved by annotating training data sets (e.g., images, videos, slides, screen shares, etc.) (i.e., potential composite input data 302) to identify and extract which layers of the data should be located at which layer depth position, such as background layers and theme layers. The training set is then used to train a neural network to create a classifier. The classifier is then applied to the composite input data to identify a scene layer that includes a background layer and a theme layer. The participant layer can then be inserted between the two layers.

[0120] In another embodiment, the system uses the presence or absence of features to determine which areas of the composite input data are assigned to which layers. For example, using composite input data of balloons floating in the sky, and using a feature classifier trained to identify balloons, areas of the composite input data where balloons are present are assigned to the subject layer, while areas where balloons are not present (i.e., sky images) are assigned to the background layer. Similarly, other feature classifiers with non-binary outputs can be thresholded or otherwise used to assign areas of input data (e.g., composite input data and multi-layer input data) to multiple layers.

[0121] The system can employ an expert system approach to identify background layers and subject layers. The expert system approach employs a set of rules to distinguish between background layers and subject layers. In some embodiments, a particular type of content can be preferentially assigned to a subject layer that is more forward and therefore less likely to be obscured. For example, text content can be preferentially assigned to a subject layer so that it is placed in front of a participant layer and is not obscured by the participant layer. Other areas of the image can then be assigned to the background layer.

[0122] Similarly, multiple scene layers of multi-layer input data can be sorted to determine the order or layer / depth position in the multi-layer scene based on the type of content. For example, text with certain keywords can be ranked higher or prioritized over other text. Certain types of images, colors, shapes, data types, sizes, or other features may indicate a higher ranking or higher priority than other types of images, colors, shapes, data types, sizes, or other features.

[0123] In some embodiments, two different regions from the same layer of input data (e.g., scene, layer, image, video feed, etc.) can be positioned to two different layer positions (i.e., depth positions when the layers are stacked). One of the two different regions can be positioned to the background layer, while the other region is assigned to the subject layer. In some embodiments, application-specific knowledge is taken into account when assigning certain regions of the input data (e.g., scene, layer, image, video feed, etc.) to the background layer and other regions to the subject layer. For example, for a spreadsheet application (e.g., Excel or Google Sheets), the system can assign grid lines to the background layer, while numbers and text in spreadsheet cells are assigned to the subject layer. Thus, by adding a participant layer between the background layer and the subject layer, the participant will stand in front of the grid lines, but will not obscure the contents of the cells themselves. As another embodiment, the system can reduce the priority of input data regions specific to menu operations, such as the toolbar at the top of the application window, so that they are assigned to the background layer. This enables interesting content within the subject layer to be presented without being obscured by areas that are not usually relevant to the presentation. Figure 4 An embodiment is shown in which a graphic (ie, subject matter of interest) is overlaid in front of a participant.

[0124] The location of the layer in the scene (i.e., the depth position within the layer, e.g., the subject layer and the background layer) can be determined based on the location of the content within or on the layer. As an example, data content near the top of the scene is deprioritized and assigned to the background layer, while content near the middle or bottom of the scene is prioritized to be in the subject layer. This enables a participant to stand and present content that obscures the lower and possibly middle of the participant without having the participant's head obscured by content higher up in the scene.

[0125] The system may employ user-assisted stratification. This involves utilizing the automatic stratification methods discussed above to provide a default "suggested" or "recommended" stratification order, thereby providing the user with a starting point from which they can further change or customize the stratification using a user interface.

[0126] The layers of the multi-layer scene are positioned and presented at different degrees of depth so that data feeds (e.g., video streams representing participants) coexist, are positioned and / or displayed on a display unit without obstructing at least some portions or one or more layers of the multi-layer scene. In some embodiments, position information (e.g., depth information) is extracted from the input data and used to position the layers relative to each other. Depth information can be extracted from the input data or data feed. Once the depth information associated with the scene is extracted, the various parts of the scene are positioned in different layers according to their distance (depth) from the camera. These parts are assigned to one of the layers defined in the scene. Background layers and subject layers are both examples of layers within the scene. Depending on the distance of at least one participant from the camera, participant layers can be added between layers in the scene (or, similarly, participant representations can be added to specific depths within the three-dimensional space). In this way, participants can be seen moving back and forth between image portions displayed in the various layers of the scene or depths of the three-dimensional space. And participants can interact with themes or content located in any of these layers or depths.

[0127] In some embodiments, a depth sensing imaging unit (including but not limited to a stereo camera, multiple cameras used together, a 2D camera complementary to a LiDAR, a camera on a mobile phone with depth sensing technology, etc.) is used to extract depth information from a scene captured by the imaging unit. In some embodiments, the imaging unit is used in conjunction with the identification of depth cues in the scene in order to extract depth information from the scene. This depth information can be identified from a data feed (e.g., a video stream) capturing the participants and multiple layers of (three-dimensional) input data. Different depth levels can be assigned to different parts or areas of a single scene, which defines the positioning of the parts or areas within the layers or various depths of the three-dimensional scene.

[0128] The system extracts images of participants from a video stream or data feed using a variety of techniques including, but not limited to, using chroma key green screen, virtual green screen techniques, skeletal / human pose recognition / estimation, neural networks trained to isolate participants in the image.

[0129] As an illustrative example of an embodiment utilizing a neural network, a training set of images containing annotated faces and body parts of people (i.e., participants) is used to train the neural network. Once trained, the neural network is used as a classifier by which it can label in a binary manner which regions of the image are most likely to be part of a person's face or body part. The identified regions are considered to be images of the participant, which can be extracted from a video stream or data feed that captures the participant frame by frame.

[0130] The system described herein also employs scaling of inputs and defines a hierarchical order. In some embodiments, more than one participant is displayed in the same scene with a shared space. The space can be a virtual space (e.g., a virtual dance floor) or an actual physical space, such as one of the participant's living rooms captured by an imaging unit. As an embodiment, when multiple participants are displayed in a shared space, scaling and normalizing the images of the captured participants can proportionally adjust the image sizes of the participants captured in the video stream or data feed so that they are similar in size on the display. This can be achieved by applying feature recognition to the participants and normalizing the scale. When the shared space is the actual physical space of the participant, the participant can be used as a reference for scaling and normalizing another participant presented on the display so that the sizes of the two participants are proportional in the same proportion.

[0131] As described above, in some embodiments, depth information is extracted from the data feed and the layers are placed relative to each other based on their distance (depth) from the camera. In some embodiments, when more than one participant is combined into a shared space, the distance between the imaging unit (e.g., camera) and the participant can be used to adjust the scale of the participant when added to the local space of another participant. More specifically, the participant becomes larger when it is close to the camera and becomes smaller when it moves away from the camera. In addition, the depth or layer position of the participant can be determined based on the dynamic distance between the imaging unit and the participant. When the participant is close to the camera, the depth position of the layer capturing the participant can be assigned to the front of the layer in the multi-layer scene (or, in other words, the participant representation can be added further forward in the three-dimensional space or scene). Similarly, when the participant moves away from the camera, the depth or layer position of the layer capturing the participant can be assigned to a position further back in the layer in the multi-layer scene (or, in other words, the participant representation can be added further back in the three-dimensional space or scene).

[0132] More examples of adjusting the size and dynamic positioning of layers are given below. In some embodiments, the characteristics (e.g., size, shape, sharpness, brightness, orientation, presence / absence, or other characteristics) of features identified on objects (including participants or other non-participant layers) by means such as feature recognition are used as depth cues to achieve such dynamic positioning. In some embodiments, the relative size of the participants (determined by comparing the size of the features identified on the participants) is used as such depth cues. In one embodiment, when participant A stands closer to the camera that captures participant A than participant B stands closer to the camera that captures participant B, the size of participant A is displayed proportionally larger than the displayed size and depth / layer position of participant B on the display (e.g., output video), and participant A is added to a depth / layer that is closer to the front. In another embodiment, when participant A stands in front of a camera that captures participant A than participant B stands in front of a camera that captures participant B, participant A is displayed proportionally smaller than the displayed size and layer position of participant B on the display (e.g., output video), and is added to a depth / layer that is farther from the front. In some embodiments, such functionality will allow two or more participants to take a virtual (i.e., the two or more participants are not present in the same physical space) selfie or group photo in which a first participant can step further away from the camera, allowing that participant to extend their arm behind the back of a second participant such that the second participant obscures most of the arm except for the hand, which reappears just beyond the second participant's shoulder.

[0133] Figure 7An embodiment of depth-based layering / positioning for interaction between two or more participants is shown. A background layer 708 is identified and positioned at the back of the layer. A first participant is extracted from a first data feed or a first video stream and identified by the system. A second participant is extracted from a second data feed of a second video stream and identified by the system. Based on each distance between a participant and a camera (e.g., an imaging unit), a depth / layer position between a first participant layer / representation and a second participant layer / representation is determined. Scene 700 shows that a first participant layer / representation 706 is added before a second participant layer / representation 704. In this case, the distance between the first participant 706 and the camera recording the first participant is less than the distance between the second participant 704 and the camera recording the second participant 704. Output video 702 shows that the first participant layer 706 is added after the second participant layer 704. In this case, the distance between the first participant 706 and the camera recording the first participant is greater than the distance between the second participant 704 and the camera recording the second participant. Therefore, in this scene 702, a portion of the arm of the first participant is occluded by the second participant. In another embodiment, depth-based stratification / localization of an interaction between two or more participants may identify the distance traveled by each participant from its initial location to its next location. In other words, depth-based stratification / localization may be based on the distance traveled by each participant as recorded by the camera, rather than the distance between the participant and its corresponding camera.

[0134] As an alternative to the automatic dynamic ordering of layers, a manual user interface may also be used. In some embodiments, by using a trigger (including but not limited to a remote control, voice or speech trigger, gesture or key), the participant can move their own layer forward or backward to change the position of the layer within the scene. As an example, this approach would solve the selfie or photo layering problem described above without having to move closer or further away from the camera.

[0135] Just as the size or scale can be modified, the lateral position of a participant in the output video can also be modified. In some embodiments, the lateral position of a participant within its participant layer is controlled simply by identifying the location of the participant in the participant's own local space. As the participant moves sideways (e.g., left or right) within the participant's own local space in front of their own camera, so will the participant within the participant layer presented in the output video. Alternatively, the location of the participant within its layer is not a direct representation of the location of the participant within the participant's own local space.

[0136] In some embodiments, the position of a participant within a participant layer of a multi-layered scene is fixed in place. Although the actual participant depicted by the display unit is in motion, the position of the participant within the layer is positionally fixed within the output. For example, Figure 6The actor captured in the actor layer of the mid-layer scene may move within the field of view of his camera, and his facial image in each video frame is extracted and located at a fixed position on the output video using a basic face tracking method.

[0137] In some embodiments, a change in the lateral or vertical positioning of a participant within their participant layer is triggered by recognition of a gesture performed or exercised by the participant. The gesture performed by the participant (i.e., a user or person captured by the camera) is captured by the imaging unit. For example, such gesture recognition includes the system recognizing a simulated running posture of the participant, which in turn triggers a change in the position of the participant in their participant layer. In other words, the system recognizes a gesture performed or exercised by the participant captured by the imaging unit. In turn, the lateral or vertical position depicted by the participant in their participant layer (i.e., the graphical representation of the participant recorded by the camera) changes. Figure 8 Another embodiment of a gesture-induced action (e.g., motion, movement, or change in a visual representation) is shown. Figure 8 , this feature of the disclosure is provided in an application of a video game environment, where an imaging unit is used to depict an image or video of a participant while the participant plays a video game. In this scene 800, a running gesture of a participant captured by an imaging system is recognized by the system, which enables the participant's position in its participant layer to run, as demonstrated by the participant's initial position 802 and the gesture-induced action showing the participant running 804 depicted in the scene 800. Scene 800 shows the position of the participant in its participant layer before (at 802) and after (at 804) the system recognizes the running gesture. Because the subject layer is located in front of the participant layer, the subject of the subject layer (e.g., house 812) is displayed as occluding the participant image, while the subject of the background layer (e.g., castle 801, tree 816, tower 818) is displayed behind the participant layer. Similarly, a simulated jumping gesture of a participant recorded by the imaging unit may cause the virtual participant's position in its participant layer to jump vertically from position 806 to screen 808, even though the physical participant may not actually leave the ground in its physical space (i.e., non-virtual / real space). Likewise, the subject (e.g., question mark box 814) is displayed in a layer above the participant layer, thereby obscuring the jumping virtual participant represented in the participant layer. Scene 808 also depicts how the system can be used to represent more than two participants in a multi-layer scene at a depth position in the same layer within the scene. Two participants 808804 are represented in the scene 800. In the scene 800, the question mark box 814 can be an interactive object, details of which will be provided below.

[0138] In some embodiments, a portion of a participant image (e.g., face, head, or half-body) may be extracted from a participant to be depicted as a participant in the participant layer. Non-limiting examples of such features include, for example, Figure 6As shown in scene 604 of FIG. 5 , a half-body of a participant extracted from a video source is displayed at a fixed position within the participant layer during the presentation. Figure 6 As shown, as the participant moves within the field of view of his camera, a basic face tracking method is used to extract the participant's face image in each video frame and locate it at a fixed position on the multi-layer scene of the demonstration. In another embodiment, as shown in FIG. Fig. 9 As shown, the participant face 904 extracted from the video stream is fixed to the virtual character or avatar 902 added to the participant layer. The scene 900 includes a background layer 906, a theme layer (displaying the avatar 902), and a participant layer (displaying the participant 904). Such virtual characters can be controlled by the participant in a video game setting using gesture and action recognition techniques described herein, etc. Basic facial tracking methods are used to extract the participant's facial image in each video frame and position it to replace the avatar face. It should be understood that although partial participant images are discussed separately herein, the image or graphical representation of the participant is not limited to capturing the entire participant using an imaging unit, but includes any part of the participant's body captured by the imaging unit.

[0139] The present disclosure provides features in which a system enables a participant to interact with a subject (also referred to herein as an object, e.g., a graphical object contained in a subject layer) viewable using a display unit. A participant can interact with such an object through a participant within a participant layer. A physical participant can use gesture recognition and motion recognition features of the system to enable such interaction. As described herein, interaction refers to a participant image displayed in a multi-layer scene contacting an object (also referred to as a "collision"), controlling an object, or otherwise causing an object to be triggered by the presence of a participant image, a gesture, or movement near or around an object image displayed in the scene.

[0140] Objects can be located in any scene layer of a multi-layer scene (i.e., background layer, subject layer, or participant layer). Some objects can interact with at least one participant. These are called interactive objects. An interactive object is a virtual object depicted in a scene within at least one of the multiple layers that make up a multi-layer scene. As described herein, an interactive object is a virtual object placed or created in a scene that is independent of a participant (i.e., separate from the participant), and interaction with the virtual object triggers certain actions, such as a change in visual state. With respect to interactions between subjects, objects, or participants depicted in a scene layer, an interactive object includes all or at least a portion of the participant image itself. For example, when there is a virtual interaction between two participants in a scene, this interaction is established using a similar approach, in which a third-party participant (i.e., other participant relative to one participant) is treated as an object.

[0141] In some embodiments, recognition of the participant's movement or action is used to induce interaction with an interactive object. Recognition of the participant's action or movement can be accomplished using video recognition methods, such as You Only Look Once (YOLO), real-time object detection methods, human pose estimation (including skeleton-based, contour-based and / or volume-based models), etc. Similarly, the participant's body parts (hands, feet, head, etc.) can be tracked by object recognition methods (e.g., YOLO, human pose estimation (including skeleton-based, contour-based and / or volume-based models), etc.). Fig.10 As shown, when the position of the participant's body part 1020 in the scene 1010 overlaps or contacts the interactive object 1022, an action will be triggered. The action in this context refers to some visible or audible changes in the scene displayed in the display unit. The triggered action can be to display a certain animation, provide a certain sound, etc. In some embodiments, the interaction causes such an action to occur, wherein the visual state of the interactive object (and / or the participant image itself) observed from the scene changes. Embodiments of such changes include certain prescribed animations, movements, color changes, deformations, state changes, etc. Process 1012 demonstrates the process of participant interaction leading to an action. In step 1014, the system identifies the body part of a participant assigned to interact with the interactive object. Once the participant's body part 1020 (in this embodiment, a hand) interacts with the interactive object 1022 by contacting the interactive object 1020 in the output video or the scene from it (i.e., the participant's body part 1020 at least partially overlaps with the interactive object 1022), the interaction between the body part and the interactive object 1022 is identified at step 1016. In response, the system triggers an action at 1018. In some embodiments, the interaction between the participant and the interactive object can be presented using a display unit, where a video or image of a physical participant (e.g., an actual game player sensed by an imaging unit) is presented in a composite video output. This graphical representation of the participant enhances the user's experience of this feature, such as in a video game setting. Figure 8 The gesture-induced action in is also a trigger action. Figure 8 As shown, the gesture change (in this embodiment, posture change) of the participant 802 changes from the participant's standing posture to a running posture, causing the participant in the scene to cause the participant in the scene to run 804. The imaging unit 102 captures the participant (game player) making a gesture. Then the data acquisition unit 104 acquires such information, and the control unit 108 responds to the information to make the participant in the scene take corresponding actions. Figure 8 In the illustrated embodiment, the interactive object does not participate in gesture-induced actions.

[0142] In some embodiments, participants can use this interactive feature to trigger animations in presentation slides from software such as Powerpoint or Google Slides. In some embodiments, students (i.e., participants) in a virtual classroom (i.e., scene) can virtually touch a virtual flashcard (i.e., interactive object) to flip it (i.e., trigger or cause an action). In some embodiments, the action is a change in other states, such as an increase in score in a video game. Obviously, many different types of objects and actions can be combined. Fig.11 An example interaction is shown in Fig.11 In the scene 1100 (multi-layer scene) captured in FIG. 1 , a participant 1106 is interacting with an interactive object 1108 within the scene. In the multi-layer scene 1100, the participant 1106 is added to a participant layer between a subject layer and a background layer 1104. In the scene 1100, the participant 1106 can interact with interactive objects of the subject layer and the background layer, such as 1108 1114 1112. In the scene 1100, the interactive object (or also referred to as the subject of the subject layer) 1112 is positioned in the subject layer (i.e., the depth position relative to the stacked layers in the scene), while the interactive objects 1108 and 1114 are positioned in the background layer, wherein all objects in the scene can be interacted with by the participant. In one embodiment, the interaction can be an action, such as pointing a finger at an interactive object, thereby triggering an action. This is an embodiment of a gesture-based interaction, in which a body part of the participant (a finger in this embodiment) is recognized to detect a certain gesture. The system detects such a gesture using an imaging unit. Alternatively, similar to Fig.10 In the example embodiment shown in , a body part of a participant contacting (and therefore interacting with) or overlapping an interactive object 1108 can trigger an action taken by the interactive object 1108. This is an embodiment of a contact-based interaction. Other embodiments include proximity-based interactions, where a participant's proximity to an interactive object beyond a certain threshold distance causes the interactive object to demonstrate an action. At scene 1102, an action is triggered. In this embodiment, a computation function is triggered, and a graphical presentation of the computation result is the action. A change in the visual state of the interactive object from 1108 to 1110 is observed in the scene. As shown in FIG. Fig.11 As shown, the system and method of the present disclosure can be applied in an educational environment to enhance students' learning ability by adding interactive elements to the learning process.

[0143] For example, a participant may make a sliding motion with their hand from left to right to advance the scene to the next scene in the presentation, or a sliding motion from right to left to change the scene to the previous slide. Similarly, in another embodiment, a participant may form a gesture with their hand that mimics holding a writing instrument. Recognition of the gesture then triggers drawing on the virtual space of the slide in the scene, as if the participant was holding a virtual writing instrument. In a gaming embodiment, a participant's gesture may be recognized and used to trigger actions such as running, jumping, throwing a virtual basketball, swinging a virtual golf club or casting magic, stepping on an interactive object, kicking, pulling, hitting, and the like. These and other applications of the presently disclosed systems and methods may be implemented to enhance the user's experience of the intended purpose of video-enabled interactive exercises.

[0144] In some embodiments, more than one participant may be involved. One participant may be able to interact with another participant; two or more participants may be able to interact with an interactive object collaboratively; and two or more participants may be able to interact with various interactive objects each. For example, in one embodiment (e.g., a video game), a first participant's hand may be used to virtually hit a second participant. In this interaction, the first participant's "hitting" gesture is an interaction with a second participant, where the second participant is considered an interactive object. When the first participant makes virtual contact with the second participant, an action is triggered, such as moving an interactive object, deforming an interactive object, or changing the color of the interactive object as the second participant (i.e., a change in visual state). For example, gestures that can be used to interact with interactive objects include punching, kicking, karate chopping, pressing, tapping, grasping, twisting, stomping, jumping, lunging, pulling, pushing, lifting, putting down, throwing, eating, drinking, etc.

[0145] Just as participants can interact with each other, objects controlled by participants can also interact with other objects (including other participants). For example, a participant holds a toy sword (or a toy lightsaber) that can interact with interactive objects on the output video. In some embodiments, a participant holds a virtual weapon (implemented by applying gesture recognition and augmented reality methods) that can interact with interactive objects, including other participants or other participants' swords (whether the sword is virtual or physical, and whether the participants are physically located in the same space or remotely located). Non-limiting embodiments of participant-controlled objects include virtual weapons, such as hand-to-hand combat weapons, swords and lightsabers, and projectile weapons, such as guns and snowballs. In educational or commercial applications, such participant-controlled objects include writing tools (with visually identifiable features at one or both ends to enhance tracking), pointing devices, etc.

[0146] In some embodiments, participants use items such as remote control devices to trigger actions in the scene. Such remote control devices include, but are not limited to, remote slide advancers, video game controllers, toy lightsabers, toy guns, etc. In some embodiments, participants use voice or sound to trigger actions in the scene. Specifically, participants can say "next slide" to trigger an action to change the scene or to trigger an action from one slide to the next slide in the presentation, and similarly, say "previous slide" to trigger an action to change the scene from one slide to the previous slide in the presentation.

[0147] like Fig. 12A As shown, two or more participants can interact with the same interactive object in a virtual space. Fig. 12A In the scene of FIG. 1204 , two participants, a first participant 1204 and a second participant 1206 , are visualized in front of a background layer 1202 . Fig. 12A The scene in presents a virtual interaction between two participants 1204-1206 displayed in the output video or visualized in the multi-layer scene. In this embodiment, the interaction is shown as the two participants 1204-1206 touching or overlapping the interactive object 1208. The action triggered by such an interaction will only be triggered when both participants complete the interaction at the same time or both participants 1204-1206 complete the interaction. In this embodiment, the layers of the multi-layer scene include a background layer 1202 located behind all other layers, a second participant layer located in front of the background layer, an interactive object layer located in front of the second participant layer, and then a first participant layer located in front of the interactive object layer. As shown in the figure, Fig. 12A A total of four different display layers are shown, including two participant layers.

[0148] In some embodiments, multiple participants may be included in the same participant layer. In some embodiments, when more than one participant is physically located in the same space in front of the same imaging unit, a single participant layer may depict multiple participants. After isolating their participant images, they are included in the same participant layer. In other embodiments, when multiple participants are physically separated and located in front of different imaging units, a single participant layer may depict multiple participants. After isolating their participant images, the participant images are inserted into the same participant layer.

[0149] Fig. 12B is another visualization of the concept of an interactive object 1208. In this embodiment, the interactive object 1208 is placed in a three-dimensional scene 1314, and multiple participants / user representations 1204 1206 are interacting with the interactive object 1208. The multiple users 1204 1206 interact with the interactive object 1208 as captured by the system's respective imaging units.

[0150] Body truncation and extrapolation

[0151] Given the distance of the camera from the user / participant, some parts of the user's body may extend beyond the bounds of the camera's view. For these cases, the "missing" or "truncated" body parts of the user will be artificially added to the participant layer and / or the representation of the user / participant included in the virtual world.

[0152] Likewise, in other cases, it may be desirable to reflect only a portion of the physical user (as seen in the camera view) into the user's virtual representation, while artificially "filling in" or "faking" the balance of the user's virtual representation.

[0153] For example, a user / participant may choose to operate the system in one physical posture (e.g., a relaxed position such as sitting or lying down), while the user's virtual representation may wish to be in a different posture (e.g., standing, walking, or running). In this case, certain portions of the user's virtual representation (e.g., upper body) may more directly reflect the user's body image and movement, while other portions of the participant's virtual representation (e.g., lower body) may be artificially represented as not reflecting the state of similar portions of the participant's body.

[0154] The "body cut off" case and the "relaxed position" case, as well as other cases, may indicate that parts of the body are different from those shown in the camera image.

[0155] In one embodiment, a user's body parts (including but not limited to hands, head, legs, feet) may extend beyond the boundaries of the camera view. The system detects when at least a part of the body (including but not limited to hands, head, legs, feet, entire lower body, etc.) extends beyond the boundaries of the camera view. One way to determine this includes determining that the shape or outline of the body (determined by body segmentation) is contacting the boundaries of the camera image.

[0156] In one embodiment, images of the user, the entire user, or individual body parts / portions thereof are stored in memory. In some embodiments, images of the user are captured as part of a body image capture process (e.g., requiring the user to stand in front of the camera in a "T-pose" and possibly rotate) to capture at least a portion of their body using the camera. In other embodiments, images of the user's body are captured seamlessly during the user's use of the system without the need for a dedicated capture process.

[0157] Based on the captured images, the system creates and stores a representation of the body so that the system can use at least a portion of the representation as needed.

[0158] like Fig.14 and Fig.15As shown, the system detects or determines the occurrence of body truncation and which body part is missing from the camera view 1512 in step 1502. The system then extracts a representation of the missing body part 1508 from the stored body representation in step 1504, as described above. In step 1506, the extracted representation of the missing body part 1508 is added or augmented to the representation of the user, so that the user representation is displayed as a whole with the augmented body part 1510 and displayed on the display unit 110.

[0159] An illustrative example of how the above system can be utilized is as follows: In one embodiment, a user / participant can utilize the system using a mobile phone's selfie camera. Particularly in landscape or horizontal orientation, the user / participant may feel that they must stand so far away from the phone to see their full body that they cannot see themselves clearly on the relatively small screen of the mobile phone. Instead, in this situation, the user can temporarily move away from the phone so that the system can capture and understand what the user's full body looks like. The user then moves closer to the phone (and may even choose to sit or lie down) so that the camera can see their upper body, but not their entire lower body. By utilizing pre-stored images and representations of the user's / participant's full body, the system is able to generate a full body representation of the user / participant for the participant layer and / or virtual world. The image displayed on the screen can then show the full body representation of the user. Likewise, any other user who sees the representation of the virtual world will see the full body representation of the user.

[0160] As stated above and reiterated below, the present disclosure includes, but is not limited to, the following example embodiments.

[0161] Item 1. A computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: receiving two-dimensional video stream data of a user using an imaging unit, wherein the two-dimensional video stream data at least partially captures the user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a user representation of the user from the two-dimensional video stream data; identifying position information of the user from the two-dimensional video stream, wherein the position information is identified based on the relative position of the user relative to the imaging unit; adding the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information; and displaying the three-dimensional scene and the added user representation on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting changes to at least one of the user and the position information.

[0162] Clause 2. The method as described in Clause 1, wherein the step of identifying the user's location information includes: extracting depth information from the two-dimensional video stream.

[0163] Clause 3. The method of clause 2, wherein the step of extracting the depth information comprises: identifying depth cues in the two-dimensional video stream.

[0164] Clause 4. The method of Clause 1, wherein the user representation comprises an image of at least a portion of the user.

[0165] Clause 5. The method as described in Clause 1 also includes the following steps: detecting missing body parts in the user representation; extracting the representation of the missing body parts from the data storage unit; and adding the extracted representation of the missing body parts to the user representation.

[0166] Clause 6. The method described in Clause 1 further includes the following steps: detecting a person in the two-dimensional video stream data; comparing the detected person with an expected user; determining whether the detected person is the expected user; and in response to determining that the detected person is the expected user, identifying the detected person as the user.

[0167] Clause 7. The method as described in Clause 1 also includes: a step of updating the added user representation in response to detecting a gesture or posture performed by a user, wherein the updating of the added user representation includes: the added user representation performing a predefined action.

[0168] Clause 8. The method as described in Clause 1 further includes: a step of identifying user posture information from a two-dimensional video stream.

[0169] Clause 9. The method of clause 8, further comprising: a step of updating the added user representation in the three-dimensional scene based on a change to at least one of the user, the position information, and the posture information.

[0170] Clause 10. The method of Clause 9, wherein the changing comprises: the user causing the added user representation to interact with the virtual object.

[0171] Clause 11. The method of Clause 10, further comprising: updating the virtual object in response to the user causing the added user representation to interact with the virtual object.

[0172] Clause 12. The method of clause 11, wherein the updating of the virtual object comprises: causing the virtual object to perform a predefined action.

[0173] Clause 13. The method of clause 1, further comprising: updating the virtual object in response to the user making a gesture to display the added user representation.

[0174] Clause 14. The method of clause 13, wherein the updating of the virtual object comprises causing the virtual object to perform a predefined action when a gesture is detected.

[0175] Item 15. A computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: using a first imaging unit to receive first two-dimensional video stream data of a first user, wherein the first two-dimensional video stream data at least partially captures the first user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, wherein the first position information is identified based on the relative position of the first user relative to the first imaging unit; using a second imaging unit to receive second two-dimensional video stream data of a second user, wherein the second two-dimensional video stream data at least partially captures the second user; isolating a second user representation of the second user from the second two-dimensional video stream data; identifying the first user from the second two-dimensional video stream; The method comprises the steps of: detecting, based on the detection of a change in at least one of the first user and the first position information, a controllable first user representation displayed on the first display unit and a controllable second user representation displayed on the second display unit; detecting, based on the detection of a change in at least one of the first user and the first position information, a controllable second user representation displayed on the first display unit and a controllable second user representation displayed on the second display unit; and controlling, based on the detection of a change in at least one of the second user and the second position information, a controllable second user representation displayed on the second display unit.

[0176] Clause 16. The method of clause 15, further comprising: a step of updating the added second user representation in the three-dimensional scene based on a change to at least one of the second user and the second position information.

[0177] Clause 17. The method of Clause 16, wherein the changing comprises the second user causing the added second user representation to interact with the added first user representation.

[0178] Clause 18. The method of clause 15, wherein the step of identifying the location information of the first user and the second user comprises: extracting depth information from the first two-dimensional video stream and the second two-dimensional video stream, respectively.

[0179] Clause 19. The method of Clause 18, wherein extracting depth information comprises identifying depth cues in one or more of the first two-dimensional video stream and the second two-dimensional video stream.

[0180] Clause 20. The method of Clause 15, wherein the first user representation comprises an image of at least a portion of the user.

[0181] Clause 21. The method as described in Clause 15 also includes the following steps: detecting missing body parts in the first user representation or the second user representation; extracting representations of the missing body parts from a data storage unit; and adding the extracted representations of the missing body parts to one or more of the first user representation and the second user representation.

[0182] Clause 22. The method as described in Clause 15 also includes the following steps: detecting a person in the first two-dimensional video stream data; comparing the detected person with an expected user; determining whether the detected person is the expected user; and in response to determining that the detected person is the expected user, identifying the detected person as the first user.

[0183] Clause 23. The method of clause 15 further comprising: a step of updating the added first user representation in response to detecting a gesture or posture performed by the first user, wherein the updating of the added first user representation comprises the added first user representation performing a predefined action.

[0184] Clause 24. The method of clause 15, further comprising: a step of updating the added second user representation in response to detecting a gesture or posture performed by the second user, wherein the updating of the added second user representation comprises the added second user representation performing a predefined action.

[0185] Clause 25. The method as described in Clause 15 also includes: the steps of identifying first posture information of the first user from the first two-dimensional video stream, and identifying second posture information of the second user from the second two-dimensional video stream.

[0186] Clause 26. The method as described in Clause 25 also includes: updating the added first user representation in the three-dimensional scene based on changes to at least one of the first user, the first position information, and the first posture information, and updating the added second user representation in the three-dimensional scene based on changes to at least one of the second user, the second position information, and the second posture information.

[0187] Clause 27. The method of Clause 26, wherein the changing comprises: the first user or the second user causing the corresponding added first user representation or second user representation to interact with the virtual object.

[0188] Clause 28. The method of Clause 27, further comprising: updating the virtual object in response to the first user and the second user causing the added first user representation and the added second user representation to interact with the virtual object.

[0189] Clause 29. The method of Clause 28, wherein the updating of the virtual object comprises causing the virtual object to perform a predefined action.

[0190] Clause 30. The method of clause 15, further comprising: updating the virtual object in response to the first user and the second user making the respective added user representation display gestures.

[0191] Clause 31. The method of Clause 30, wherein the updating of the virtual object comprises causing the virtual object to perform a predefined action when a gesture is detected.

[0192] Item 32. A system for controlling a virtual representation of a user in a three-dimensional scene, comprising: an imaging unit for capturing two-dimensional video stream data of a user; a storage unit having computer-readable program instructions stored therein; and a processor in communication with the storage unit and configured to execute the computer-readable program instructions so that the system at least: receives two-dimensional video stream data of a user from the imaging unit, wherein the two-dimensional video stream data at least partially captures the user; receives a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolates a user representation of the user from the two-dimensional video stream data; identifies position information of the user from the two-dimensional video stream, wherein the position information is identified based on the relative position of the user relative to the imaging unit; adds the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information and posture information; and displays the three-dimensional scene and the added user representation on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting a change to at least one of the user and the position information.

[0193] Clause 33. The system of clause 32, wherein the user's position information comprises depth information extracted from a two-dimensional video stream.

[0194] Clause 34. The system of clause 33, wherein the extraction of depth information comprises: identifying depth cues in a two-dimensional video stream.

[0195] Clause 35. The system of Clause 32, wherein the user representation comprises an image of at least a portion of the user.

[0196] Clause 36. A system as described in Clause 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to: detect missing body parts in the user representation; extract representations of the missing body parts from the data storage unit; and add the extracted representations of the missing body parts to the user representation.

[0197] Clause 37. A system as described in Clause 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to: detect a person in two-dimensional video stream data; compare the detected person with an expected user; determine whether the detected person is the expected user; and in response to determining that the detected person is the expected user, identify the detected person as a user.

[0198] Clause 38. A system as described in Clause 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the added user representation in response to detecting a gesture or posture performed by the user, wherein the updating of the added user representation includes the added user representation performing a predefined action.

[0199] Clause 39. The system of clause 32, wherein the processor is further configured to execute computer-readable program instructions to enable the system to recognize gesture information of the user from the two-dimensional video stream.

[0200] Clause 40. The system of clause 39, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the added user representation in the three-dimensional scene based on a change in at least one of the user, the position information, and the gesture information.

[0201] Clause 41. The system of Clause 40, wherein the change comprises the user causing the added user representation to interact with the virtual object.

[0202] Clause 42. The system of clause 41, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the virtual object in response to the user causing the added user representation to interact with the virtual object.

[0203] Clause 43. The system of Clause 42, wherein updating of the virtual object comprises causing the virtual object to perform a predefined action.

[0204] Clause 44. The system of clause 32, wherein the processor is further configured to execute computer-readable program instructions to cause the system to update the virtual object in response to the user making an added user representation display gesture.

[0205] Clause 45. The system of clause 44, wherein updating of the virtual object comprises causing the virtual object to perform a predefined action upon detecting a gesture.

[0206] Although several variations of the present disclosure are described in the specific description by way of example, it is clear that further embodiments can be developed within the spirit and scope of the present disclosure or within its inventive concept. Those skilled in the art will readily observe that many modifications, applications, changes and alterations can be made to the apparatus and method while retaining the teachings of the present disclosure. Therefore, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. In addition, although the foregoing description and the associated drawings describe example embodiments in the context of certain embodiment elements and / or functional combinations, it should be understood that different elements and / or functional combinations can be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, it is also envisioned that elements and / or functional combinations different from those explicitly described above are also envisioned, as may be set forth in certain appended claims. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes. It should be understood that the above disclosed variations and other features and functions or their alternatives can be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.

Claims

1. A computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: receiving, using an imaging unit, two-dimensional video stream data of a user, wherein the two-dimensional video stream data at least partially captures the user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a user representation of the user from the two-dimensional video stream data; Identifying position information of the user from the two-dimensional video stream, wherein the position information is identified based on a relative position of the user with respect to the imaging unit; recognizing posture information of the user from the two-dimensional video stream; adding the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information; as well as The three-dimensional scene and the added user representation are displayed on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting a change to at least one of the user and the position information.

2. The method according to claim 1, wherein: The step of identifying the user's position information includes: extracting depth information from the two-dimensional video stream.

3. The method according to claim 2, wherein: The step of extracting the depth information includes identifying depth cues in the two-dimensional video stream.

4. The method according to claim 1, wherein: The user representation includes an image of at least a portion of the user.

5. The method according to claim 1, further comprising the steps of: detecting missing body parts in the user representation; extracting the missing representation of the body part from the data storage unit; as well as The extracted missing representation of the body part is added to the user representation.

6. The method according to claim 1, further comprising the steps of: Detecting a person in the two-dimensional video stream data; comparing the detected person to the expected user; determining whether the detected person is the intended user; as well as In response to determining that the detected person is the intended user, the detected person is identified as the user.

7. The method according to claim 1, further comprising: The step of updating the added user representation in response to detecting a gesture or posture performed by a user, wherein the updating of the added user representation comprises the added user representation performing a predefined action.

8. The method according to claim 1, further comprising: The step of updating the added user representation in the three-dimensional scene based on changes to at least one of the user, the position information, and the posture information.

9. The method according to claim 8, wherein: The changing includes: the user causing the added user representation to interact with the virtual object.

10. The method according to claim 9, further comprising: The step of updating the virtual object in response to the user causing the added user representation to interact with the virtual object, wherein the updating of the virtual object comprises causing the virtual object to perform a predefined action.

11. The method according to claim 1, further comprising: The step of updating the virtual object in response to the user making the added user representation display gesture, wherein the updating of the virtual object comprises: causing the virtual object to perform a predefined action when the gesture is detected.

12. A computer-implemented method for controlling a virtual representation of a user in a three-dimensional scene, the method comprising: receiving, using a first imaging unit, first two-dimensional video stream data of a first user, wherein the first two-dimensional video stream data at least partially captures the first user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a first user representation of the first user from the first two-dimensional video stream data; identifying first position information of the first user from the first two-dimensional video stream, wherein the first position information is identified based on a relative position of the first user relative to the first imaging unit; receiving, using a second imaging unit, second two-dimensional video stream data of a second user, wherein the second two-dimensional video stream data at least partially captures the second user; isolating a second user representation of the second user from the second two-dimensional video stream data; identifying second position information of the second user from the second two-dimensional video stream, wherein the second position information is identified based on a relative position of the second user relative to the second imaging unit; adding the first user representation and a second user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene, wherein the data associated with the first user representation uses the first position information and the data associated with the second user representation uses the second position information; as well as The three-dimensional scene, the added first user representation, and the added second user representation are displayed on a first display unit and a second display unit, wherein the added first user representation displayed on the first display unit is controlled based on detecting a change to at least one of the first user and the first position information, and the added second user representation displayed on the second display unit is controlled based on detecting a change to at least one of the second user and the second position information.

13. The method according to claim 15, further comprising: The step of updating the added second user representation in the three-dimensional scene based on a change to at least one of the second user and the second position information.

14. The method according to claim 16, wherein: The changing includes the second user causing the added second user representation to interact with the added first user representation.

15. A system for controlling a virtual representation of a user in a three-dimensional scene, comprising: An imaging unit, wherein the imaging unit captures two-dimensional video stream data of a user; a storage unit having computer-readable program instructions stored therein; as well as a processor in communication with the memory unit, the processor being configured to execute the computer-readable program instructions to cause the system to at least: receiving the two-dimensional video stream data of the user from the imaging unit, wherein the two-dimensional video stream data at least partially captures the user; receiving a three-dimensional scene, wherein the three-dimensional scene includes data associated with at least one virtual object and its position in the three-dimensional scene; isolating a user representation of the user from the two-dimensional video stream data, wherein the user representation includes an image of at least a portion of the user; identifying position information of the user from the two-dimensional video stream, wherein the position information is identified based on a relative position of the user relative to the imaging unit, and the position information of the user includes depth information extracted from the two-dimensional video stream; recognizing posture information of the user from the two-dimensional video stream; adding the user representation to the three-dimensional scene by modifying data associated with voxels constituting the three-dimensional scene using the position information and the pose information; as well as The three-dimensional scene and the added user representation are displayed on a display unit, wherein the added user representation displayed on the display unit is controlled based on detecting a change to at least one of the user, the position information, and the gesture information.