Representing two-dimensional representation as three-dimensional avatar

Abstract

Systems and methods for representing a two-dimensional representation as a three-dimensional avatar are provided herein. In some examples, one or more input video streams are received. A first subject within one or more input video streams is identified. A first view of a first subject is identified based on one or more input video streams. Based on the one or more input video streams, a second view of the first subject is identified. The first body is divided into a plurality of planar objects. The plurality of planar objects are transformed relative to each other. The plurality of planar objects are based on a first view and a second view of the first subject. A plurality of planar objects are output in an output video stream. The plurality of planar objects provide a viewing angle of the first subject to one or more viewers.

Description

Background Art

[0001] Video conferencing can be used to allow one or more conference participants to join a conference remotely. A hybrid conference can include multiple participants joining a conference through multiple different mediums, such as in person, remotely, and / or through an avatar.

[0002] With respect to these and other general considerations, embodiments have been described.Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the invention

[0003] Aspects of the present disclosure relate to methods, systems, and media for representing a two-dimensional representation as an avatar in a three-dimensional space.

[0004] The mechanisms described herein can be used to generate three-dimensional avatars for hybrid conferences, such as conferences that incorporate metaverse technology (e.g., virtual reality and / or augmented reality technology). In some examples, one or more input video streams can be received. One or more subjects (such as people, animals, or objects) can be identified within the one or more input video streams. The mechanisms provided herein can segment each of the one or more subjects into multiple planar objects (e.g., billboards) to improve the perspective of one or more subjects relative to one or more participants in the hybrid conference. Additionally or alternatively, different views of one or more subjects can be identified within one or more input video streams. Different views of one or more subjects can be presented based on relative positions in a room. The relative positions in the room can be associated with participants in a hybrid conference, where the hybrid conference is associated with a room.

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0007] Figure 1 An overview of an example system according to some aspects described herein is shown.

[0008] Figure 2 An example hybrid conference is shown according to some aspects described herein.

[0009] Figure 3 An example of generating one or more views of a subject according to some aspects described herein is shown.

[0010] Figure 4 An overview of an example method according to some aspects described herein is shown.

[0011] Figure 5 An example of generating one or more views of a subject according to some aspects described herein is shown.

[0012] Figure 6 An overview of an example method according to some aspects described herein is shown.

[0013] Figure 7 is a block diagram illustrating example physical components of a computing device with which aspects of the present disclosure may be practiced.

[0014] Fig. 8A and Figure 8B is a simplified block diagram of a mobile computing device with which aspects of the present disclosure may be practiced.

[0015] Fig. 9 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0016] In the following detailed description, reference is made to the accompanying drawings which are part of the present invention, and in which specific embodiments or examples are shown by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be practiced as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementation, complete software implementation, or a combination of software and hardware implementations. Therefore, the following specific embodiments should not be considered restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0017] As described above, video conferencing can be used to allow one or more conference participants to join a conference remotely. Hybrid conferences can include multiple participants, with multiple participants joining the conference through multiple different mediums, such as in person, remotely (e.g., via video and / or audio communication devices), and / or via avatars. In this regard, hybrid conferences can allow participants to join the conference using a variety of metaverse technologies (e.g., virtual reality headsets, augmented reality headsets, audio communication technologies, video communication technologies, etc.).

[0018] In recent years, the rise of remote work has spurred the development of technologies that allow multiple individuals (e.g., teammates, colleagues, board members, family members, etc.) to meet or collaborate without having to physically be in the same room. However, existing technologies for remote work (such as conventional video conferencing software) may provide limited options for immersing remote participants into a hybrid meeting environment. In addition, some existing technologies for remote work may be computationally intensive.

[0019] In some examples disclosed herein, a virtual subject (e.g., a person, an animal, or an object) can be segmented or otherwise decomposed into multiple planar objects (e.g., a billboard) to provide a perspective view of the subject to one or more viewers of the virtual subject. Specifically, one or more input video streams can be received from one or more cameras. A first object, such as a person, can be identified within the one or more input video streams. Based on the one or more input video streams, a first view of the first subject and a second view of the first subject can be identified. The first subject can be segmented into multiple planar objects (e.g., planes that transform relative to each other). Multiple planar objects can be based on the first view and the second view of the first subject. In addition, multiple planar objects can be output in an output video stream.

[0020] Some people may be accustomed to accepting planar objects (such as billboards) as real objects, such as the everyday experience of viewing a flat display of a photo, television, and / or movie from an angle. When an object is divided into several billboards, the perspective distortion on each part of the divided object can be reduced. In addition, the transition between billboards may produce artifacts. In some examples, a continuous transition through a curve proxy can be used between billboards. Additionally or alternatively, in some examples, two overlapping billboards can be blended together. A person of ordinary skill in the art can recognize additional and / or alternative mechanisms for transitioning between billboards, for example to reduce perspective distortion between billboards.

[0021] The billboard may also be represented by multiple planes and / or thin volume representations to improve plenoptic functionality and / or view-dependent effects. For example, the billboard may be represented using a plenoptic voxel (e.g., plenoptic volume element) representation. The plenoptic voxel representation may be used to improve fidelity without the need for a neural network. Fidelity may be improved by using a standard mean square error (MSE) reconstruction loss relative to training images and a total variation regularizer. Additionally or alternatively, the billboard may be represented using a neural radiance field (NERF) representation. Using the NERF representation, novel views of complex scenes may be synthesized by improving the underlying continuous volumetric scene function using a set of sparse input views.

[0022] In some other examples disclosed herein, a virtual subject can be observed from multiple views based on the corresponding positions of multiple hybrid conference participants. Specifically, multiple input video streams can be received as part of a hybrid conference. The geometry of a room corresponding to the hybrid conference can be identified. A first subject at a first position in the room and a second subject at a second position in the room can be identified. A first view of the first subject can be determined based on a first input video stream in the multiple input video streams. A second view of the first subject can be determined based on a second input video stream in the multiple input video streams. In addition, the first view of the first subject can be output based on the second position in the room.

[0023] Thus, aspects of the present disclosure relate to representing a two-dimensional representation (e.g., a two-dimensional representation of a user) as an avatar in a three-dimensional space to improve user engagement and provide other benefits. Specifically, the mechanisms disclosed herein provide improved user engagement by creating a perspective view of a user based on one or more two-dimensional views received from one or more input video streams, with relatively low computational cost compared to conventional systems that may convert a two-dimensional representation into a three-dimensional avatar (e.g., by using a three-dimensional point cloud). Other advantages may be recognized by one of ordinary skill in the art in view of at least the teachings disclosed herein.

[0024] The billboards or planar objects described herein may be rendered using a variety of techniques. An example technique for rendering billboards described herein is through Neural Radiance Fields (NERF). NERF technology may combine multi-view images into a volume representation to generate a better view (e.g., a view with higher fidelity) than the original view (e.g., the original billboard view). The volume representation enables the generation of new views of objects without reference geometry by rendering rays (e.g., camera rays) to project output colors and densities (e.g., pixels) into an image. In some examples, rendering of a generic human model may be performed by identifying rays along each limb of the person and querying the NERF rays based on the motion bending of one or more limbs of the person.

[0025] Figure 1 An example of a system 100 according to some aspects of the disclosed subject matter is shown. The example system 100 may be a system for representing a two-dimensional image as an avatar in a three-dimensional space, such as a hybrid conference space. Figure 1 As shown, system 100 includes computing device 102, server 104, video data source 106, and communication network or network 108. Computing device 102 may receive video stream data 110 from video data source 106, which may be, for example, a webcam, a video camera, a video file, etc. Additionally or alternatively, network 108 may receive video stream data 110 from video data source 106, which may be, for example, a webcam, a video camera, a video file, etc.

[0026] The computing device 102 may include a communication system 112, a real subject identification engine or component 114, and a virtual subject generation engine or component 116. In some examples, the computing device 102 may execute at least a portion of the real subject identification component 114 to identify, locate, and / or track a subject (e.g., a person, animal, or object) from the video stream data 110. In addition, in some examples, the computing device 102 may execute at least a portion of the virtual subject generation component 116 to determine a view, a planar object (e.g., a billboard) segmentation, or a planar object configuration of a subject (such as a subject identified by the real subject identification component 114) within the video stream data 110. The view of the subject may be determined based on a single set of video stream data 110 and / or multiple sets of video stream data 110. In addition, the planar object may be segmented based on one or more pivot points of the subject. Additionally or alternatively, the planar object may be generated or otherwise configured based on a trained machine learning model that is trained to reduce feature errors between the altered image and the true value image based on a view relative to a reference point within the hybrid meeting space.

[0027] In some examples, the trained classifier can be used to detect the posture of the skeleton of a person and each person based on a two-dimensional video (e.g., video stream data 110). The two-dimensional video can be received from an RGBD (red, green, blue and depth) video source (e.g., video data source 106). The RGBD video can be received from a line motion sensing input device (e.g., for a gaming system or video system) including one or more RGB cameras and an infrared projection and detector, which maps depth by structured light or time-of-flight calculation. Additionally or alternatively, RGBD videos can be received from multiple video streams of a person (e.g., a stereo camera). The skeleton can be a guide for a simple geometric agent. A general human model can be used to project pixels from one or more video sources to a new view. An example agent described herein is an articulated model in which each limb of a person is modeled by a different plane. In addition, the depth data obtained from the video stream data 110 can be used to segment a person from a video, generating a construction of a proxy geometry or volume representation that is better than a planar object.

[0028] The server 104 may include a communication system 112, a real subject identification engine or component 114, and a virtual subject generation engine or component 116. In some examples, the server 104 may execute at least a portion of the real subject identification component 114 to identify, locate, and / or track a subject (e.g., a person, animal, or object) from the video stream data 110. Additionally, in some examples, the server 104 may execute at least a portion of the virtual subject generation component 116 to determine a view, a planar object segmentation, or a planar object configuration of a subject (such as a subject identified by the real subject identification component 114) within the video stream data 110. The view of the subject may be determined based on a single set of video stream data 110 and / or multiple sets of video stream data 110. Additionally, the planar object may be segmented based on one or more pivot points of the subject. Additionally or alternatively, the planar object may be generated or otherwise configured based on a trained machine learning model that is trained to reduce feature errors between the altered image and the true value image based on viewpoint relative to a reference point within the hybrid conference space.

[0029] Additionally or alternatively, in some examples, computing device 102 may transmit data received from video data source 106 to server 104 via communication network 108, which may execute at least a portion of real subject identification component 114 and / or virtual subject generation component 116. In some examples, real subject identification component 114 may execute the following in conjunction with Figure 4 and Figure 6 In addition, in some examples, the virtual subject generation component 116 can perform the following combined Figure 4 and Figure 6 One or more portions of method / process 400 and / or 600 are described.

[0030] In some examples, computing device 102 and / or server 104 may be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smart phone, a tablet computer, a wearable computer, a server computer, a virtual machine executed by a physical computing device, a head-mounted display (HMD) (e.g., a virtual reality headset and / or an augmented reality headset), etc.

[0031] In some examples, video data source 106 may be any suitable source of video streaming data (e.g., data generated from a computing device, data generated from a webcam, data generated from a video camera, etc.). In a more specific example, video data source 106 may include a memory storing video streaming data (e.g., local memory of computing device 102, local memory of server 104, portable memory connected to computing device 102, etc.). In another more specific example, video data source 106 may include an application configured to generate video streaming data (e.g., a teleconferencing application with video streaming capabilities, a subject identification application, and / or a virtual subject generation application executed by computing device 102, server 104, and / or any other suitable computing device).

[0032] In some examples, video data source 106 may be local to computing device 102. For example, video data source 106 may be a camera coupled to computing device 102. Additionally or alternatively, video data source 106 may be remote from computing device 102 and may transmit video stream data 110 to computing device 102 (and / or server 104) via a communication network (e.g., communication network 108).

[0033] In some examples, communication network 108 can be any suitable communication network or combination of communication networks. For example, communication network 108 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc., in accordance with any suitable standard), a wired network, etc. In some examples, communication network 108 can be a local area network (LAN), a wide area network (WAN), a public network (e.g., the Internet), a private or semi-private network (e.g., a company or university intranet), any other suitable type of network, or any suitable combination of networks. Figure 1 The communication links (arrows) shown in the figure can each be any suitable communication link or combination of communication links, such as a wired link, a fiber optic link, a Wi-Fi link, a Bluetooth link, a cellular link, etc.

[0034] Although the example system 100 includes a single computing device 102, a single server 104, and a single network 108, one of ordinary skill in the art will recognize that there may be multiple computing devices 102, multiple servers 104, and / or multiple networks 108. Furthermore, there may be multiple video data sources 106 in communication with one or more computing devices (e.g., computing device 102). Additionally or alternatively, there may be multiple video data sources 106 in communication with one or more networks (e.g., network 108).

[0035] Figure 2 An overview of an example hybrid conference 200 according to some aspects described herein is shown. The hybrid conference 200 can occur within a room 204. In some examples, the room 204 can be a physical room. Alternatively, in some examples, the room 204 can be a virtual room. Multiple participants 208 can participate in the hybrid conference 200. A first subset 208A of the multiple participants 208 can participate in the hybrid conference 200 via video conferencing. A second subset 208B of the multiple participants 208 can be physically present at the hybrid conference 200. A third subset 208C of the multiple participants 208 can wear a head mounted display (HMD) 216 to view one or more generated avatars 210 corresponding to remote participants (e.g., those in the first subset 208A or in the live subset 208B).

[0036] A first subset 208A of the plurality of participants 208 can each be remote from the room 204. The first subset 208A can each have a computing device (e.g., computing device 102) and / or a video data source (e.g., video data source 106) that is local to each participant in the first subset 208A and allows each participant in the first subset 208A to participate in the hybrid conference 200. The first subset 208A can participate in the hybrid conference 200 by being presented on a display screen 212, projected onto a surface in the room 204, projected onto an object in the room 204 (e.g., a chair, a table, etc.), or otherwise displayed in the room 204.

[0037] The second subset 208B of the plurality of participants 208 can each be physically located within the room 204. The third subset 208C can be physically located in the same room as the second subset 208B or in a remote location. The room 204 can include one or more cameras 220 disposed therein to generate a video stream from the second subset 208B to one or more participants to one or more remote users (e.g., the first subset 208A of the plurality of participants 208 and / or the third subset 208C of the plurality of participants 208).

[0038] The avatars 210 generated corresponding to one or more participants within the third subset 208C may be visible to at least some of the participants 208 remote from the hybrid conference 200. Additionally and / or alternatively, at least some of the plurality of participants 208 may be able to view the one or more generated avatars 210 by wearing a head mounted display (HMD) 216. The HMD 216 may be a virtual reality headset. Alternatively, the HMD 216 may be an augmented reality headset. In further aspects, the one or more generated avatars 210 may be displayed on a display screen 212. Additionally or alternatively, the one or more generated avatars 210 may be displayed on one or more computing devices (e.g., computing device 102) remote and / or local to the room 204. In this regard, the one or more generated avatars 210 may be viewed by any heterogeneous endpoint (e.g., HMD, laptop, smart phone, display screen in a physical room, projector, wearable computing device, etc.) of the plurality of participants 208 interacting with the hybrid conference 200.

[0039] One or more cameras 220 (e.g., 2D and / or 2.5D RGBD cameras) may be similar to those previously described herein. Figure 1 The video data source 106 described above. One or more cameras 220 can collect a batch of still images taken at consecutive time intervals. Additionally or alternatively, one or more cameras 220 can collect a single instance of a still image taken at a certain moment. Additionally or alternatively, one or more cameras 220 can collect a real-time feed of video data. One or more cameras 220 can be configured at specific locations within the room 204 based on user preferences. Additionally, there can be any number of cameras 220 within the room 204, depending on different advantages or disadvantages that can be recognized by ordinary technicians in the field (e.g., multiple desired camera angles, desired overhead for processing video / image data received from the cameras 220, etc.).

[0040] Figure 3 An example 300 of generating one or more views of a subject 302 according to some aspects described herein is shown. The subject 302 may be a person, an animal, and / or an object generated based on video stream data (e.g., video stream data 110). Additionally or alternatively, in some examples, the subject 302 may be generated based on one or more still images (e.g., two-dimensional images) and / or one or more animations corresponding to a physical version of the subject 302. The subject 302 shown in example 300 is a virtual subject, which may be similar to the subject previously described with respect to Figure 2 The generated avatar 210 is described.

[0041] The subject 302 may be located within a hybrid meeting space (e.g., a space in which the hybrid meeting 200 occurs). In addition, the subject 302 may correspond to a physical subject (not shown) that is remote from the hybrid meeting space. The subject 302 may be segmented into a plurality of planar objects 304. The plurality of planar objects 304 may be a plurality of billboards. The plurality of billboards 304 may be planes that are transformed relative to each other. The planes may be two-dimensional planes and / or curved surfaces. In addition, the planes may be rotated at an angle relative to each other (e.g., not coplanar).

[0042] The body 302 may include one or more pivot points around which two or more parts of the body 302 are configured to rotate. For example, the body 302 is a person, where the one or more pivot points are joints around which the body parts are configured to rotate (e.g., wrists, elbows, knees, etc.). One or more panels may be split at the pivot points. One or more panels may be connected via virtual joints, and the virtual joints may be joint points of rotation.

[0043] One or more boards 304 may be bent toward one another, stretched to overlap or intersect one another, blurred toward one another, blended toward one another, discretely abutted toward one another, or otherwise visually manipulated to form a virtual subject (e.g., object 302) corresponding to a physical subject, such as a subject that may remotely join a hybrid conference. Each of the one or more boards 304 may be translated and / or rotated within a three-dimensional space (e.g., including an x-plane, a y-plane, and a z-plane) to form a perspective view of a portion of the subject 302 contained within the translated and / or rotated board 304.

[0044] In example 300, one or more billboards 304 corresponding to the body of subject 302 may be intentionally rotated so that the geometry of subject 302 is visible to the viewer. Determining which aspects of subject 302 (e.g., contained on or within one or more billboards 304) may be more visually appropriate to face the viewer may be configured by the user. Additionally, determining which aspects of subject 302 may be more visually appropriate to face the viewer may be learned by a machine learning model. For example, participants in a hybrid conference may interact (e.g., watch, listen, etc.) with a subject (e.g., subject 302) that is arranged in a particular manner more frequently (e.g., for a longer measurable duration). Such examples may include a face facing the viewer, a palm facing the user's hand, a chest facing the user, and the like. A person of ordinary skill in the art may use the mechanisms disclosed herein to identify additional and / or alternative configurations that may help increase user interaction with subject 302.

[0045] In some examples, a skeleton of subject 302 may be generated or detected based on video stream data (e.g., video stream data 110). The skeleton may be rendered to a new view with each limb of subject 302 aligned along its respective axis and facing a reference point corresponding to the new view. The reference point may be a camera of a computing device (e.g., computing device 102). In some examples (e.g., where there are no constraints), the skeleton may be retained in the same arrangement as the original video, and a single billboard (e.g., billboard 304) may be generated for the entire body of subject 302.

[0046] Alternatively, in some examples, such as if the subject's 302 hand is touching a whiteboard, a separate lookboard may be generated for each bone in the subject's 302 hand to satisfy the constraints for the new view. One of ordinary skill in the art may recognize additional and / or alternative examples of visual constraints that may affect the subject's visual rendering when generating a new view.

[0047] The sign 304 may be used to generate additional environmental effects that may enhance the realism of the composite scene—such as rendering reflections of the subject 302 onto the conference table, rendering shadows of the subject's 302 hands and body onto the table and / or walls, or even projecting shadows onto the physical conference table in the physical room to better represent the remote participants (e.g., as an alternative or in addition to viewing the remote participants on a display screen).

[0048] The segmented billboards 304 can be conceived as an origami configuration, where each segmented billboard 304 is folded relative to each other to provide a three-dimensional avatar of a user (e.g., subject 302) including perspective of one or more views. Such an approach can improve upon existing methods of generating three-dimensional avatars with relatively little computational cost, while still providing perspective information to the viewer that improves the user experience.

[0049] Additionally or alternatively, a machine learning model may be used to segment the subject 302 into the plurality of fascinations 304. The machine learning model may be trained to generate the segmented fascinations 304 based on a plurality of constraints or factors. For example, the plurality of constraints may include a computational cost (e.g., for the computing device 102 and / or the server 104) of generating the segmented fascinations 304, one or more input video streams, an error between the physical subject and an avatar (e.g., the subject 302) generated from the segmented fascinations 304. The physical subject to which the generated avatar corresponds may be found within the one or more input video streams. One of ordinary skill in the art may recognize additional and / or alternative constraints.

[0050] Furthermore, it should be appreciated that while generating a relatively large number of kanbans (e.g., kanban 304) may reduce the error between subject 302 and the corresponding physical subject, generating a relatively large number of kanbans may also increase computational overhead. Thus, the training of the machine learning model may also depend on a policy factor that weights variables from the plurality of variables based on user preferences and / or economic factors.

[0051] Furthermore, it should be appreciated that a board (e.g., board 304) itself may not be photo-realistic, but rather a representation of what we are accustomed to seeing on a flat screen viewed from an angle. Whenever a board is split into multiple boards, objectionable artifacts (e.g., anomalies that are noticeable during the visual representation) may be generated. Therefore, in some examples, it may be beneficial to use as few boards as possible, such as when the mapping of a subject (e.g., subject 302) via a single board (e.g., one of boards 304) does not achieve a certain desired constraint (e.g., an improved perspective view). Therefore, the cost of increasing the number of boards may not only be more overhead, but also an increased number of artifacts. Such factors may be considered when determining how many segments a subject (e.g., subject 302) should be split into, with each segment represented by a corresponding board.

[0052] The plurality of panels 304 may be output in the output video stream. For example, the plurality of panels 304 may form the subject 302, and the subject 302 may be output in the output video stream. The output video stream may be output to a computing device (e.g., computing device 102), which may be, for example, a display screen and / or a head mounted display (e.g., HMD 216) of a computer on which one or more users are video conferencing. Thus, after the subject 302 is generated in the output video stream via the plurality of panels 304, the user may interact with the subject 302.

[0053] Figure 4 An example method 400 is shown according to some aspects described herein. The method 400 can be a method of representing a two-dimensional image of a user as an avatar or subject (e.g., avatar 210 or subject 302) in a three-dimensional space, and / or a method of generating a three-dimensional avatar or subject. In an example, aspects of the method 400 are as described above with respect to Figure 1 The computing device 102 and / or the server 104 discussed above may be executed.

[0054] Method 400 begins at operation 402, where one or more input video streams are received. For example, the video data source (e.g., Figure 1The video data source 106 discussed above receives an input video stream. As described above, the video data source can be, for example, a webcam, a video camera, a video file, etc. In some examples, operation 402 can include obtaining an input video stream. For example, the input video stream can be obtained by executing a command via a processor, which causes the input video stream to be received by, for example, a real subject identification component (such as real subject identification component 114). In addition, in some examples, operation 402 can include receiving multiple input video streams.

[0055] In some examples, the input video stream may include a three-dimensional scene. Therefore, the various aspects of the disclosure described below (e.g., identification, segmentation, tracking, etc.) can be applied to a three-dimensional scene in a similar manner as applied to a two-dimensional scene. For example, the system described herein can identify a three-dimensional sub-scene (e.g., containing a person or object or animal of interest) of an input video stream and segment or segment subjects within the three-dimensional sub-scene.

[0056] At operation 404, it is determined whether the input video stream contains a subject (e.g., one or more people, animals, objects, etc.). In addition, in some examples, it can be determined whether multiple input video streams contain a subject. The subject can be identified by one or more computing devices (e.g., computing device 102 and / or server 104). Specifically, one or more computing devices can receive visual data from a visual data source (e.g., video data source 106) to identify a person. The mechanisms described herein can be used to process visual data to identify the presence of one or more people of interest, one or more animals of interest, and / or one or more objects of interest. One or more people, one or more animals, and / or one or more objects can be identified based on the presence of people, animals, and / or objects, based on the movement of people, animals, and / or objects, and / or based on other scene attributes of the input video stream (such as color, contrast, lighting, differences in intervals between subjects, etc.).

[0057] Additionally or alternatively, at determination 404, one or more subjects may be identified by interacting with specific software (e.g., joining a call, joining a video call, joining a chat, etc.). For example, a person may be identified by logging into a specific application (e.g., via a password, biometric input, or registration number). Thus, when logging into a specific application, the person is thereby identified. For example, when a person joins a video call, it may be determined that the input video stream contains the subject (i.e., the person). Additionally or alternatively, at determination 404, the subject may be identified by identifying the subject using a radio frequency identification tag (RFID), an ID badge, a barcode, or some other identification means capable of identifying the subject via some technical interface, thereby determining that the input video stream contains the subject.

[0058] It should be appreciated that method 400 is provided as an example of identifying or not identifying a subject at determination 404. In other examples, a determination may be made to request user clarification or disambiguation (e.g., before proceeding to operation 406 or 408), such as when a confidence level associated with identifying a subject is below a predetermined threshold, or when multiple subjects are identified, etc. In examples where such clarifying user input is received, an indication of the subject may be stored (e.g., associated with a received input video stream) and used to improve accuracy when processing similar future video stream inputs.

[0059] If it is determined that the subject is not contained in the input video stream, the process will enter the branch "No" to operation 406, where a default action is performed. For example, the input video stream may have an associated preconfigured action (such as not presenting the input video stream). In other examples, method 400 may include determining whether the input video stream has an associated default action so that in some cases, no action is performed due to the received input video stream (e.g., conventional methods may be used to display the input video stream). Method 400 may terminate at operation 406. Alternatively, method 400 may return to operation 402 to provide a continuous video stream feedback loop.

[0060] However, if it is determined that the input video stream contains a subject, the process will enter the branch "yes" to operation 408, where a first subject is identified in one or more input video streams. The first subject can be a person, an animal, or an object. In addition, in the case where there are multiple input video streams, the first subject can be identified in each of the multiple input video streams, such as by visually identifying common features in each input video stream corresponding to the first subject.

[0061] The process proceeds to operation 410, where a first view of the first subject is identified based on one or more input video streams. The first view can be a front view, a side view, a rear view, a top view, a perspective view, or any type of view of the first subject captured by a video data source (e.g., video data source 106) (such as a camera). The first view can be identified relative to a reference point or shape within a room in which the first subject is located. For example, the room can have a specific geometry that is recognized or received by the mechanisms disclosed herein. The first view can be identified based on the geometry of the room. Additionally or alternatively, the first view can be identified based on a relative rotation of the first subject within one or more input video streams.

[0062] The process proceeds to operation 412, where a second view of the first subject is identified based on one or more input video streams. The second view can be a front view, a side view, a rear view, a top view, a perspective view, or any type of view of the first subject captured by a video data source (e.g., video data source 106) (such as a camera). The second view can be identified relative to a reference point or shape in a room in which the first subject is located. For example, the room can have a specific geometry recognized or received by the mechanism disclosed herein. The second view can be identified based on the geometry of the room. Additionally or alternatively, the second view can be identified based on a relative rotation of the first subject within one or more input video streams (such as based on a relative rotation of the first subject relative to the first view). In some examples, operations 410 and 412 can be combined, such as in the case of identifying multiple views of the first subject based on multiple input video streams.

[0063] At determination 414, it is determined whether the second view of operation 412 is substantially different from the first view of operation 410. For example, if the second view is substantially the same as the first view, it may be computationally advantageous to output one of the first view or the second view from the one or more input video streams without further processing based on the first view and the second view. In addition, determining whether the second view is substantially different from the first view may include calculating an error between a visual component of the first view and a visual component of the second view. A user may configure a predetermined threshold at which the second view is determined to be substantially different from the first view.

[0064] If it is determined that the second view is not substantially different from the first view, the process will enter the branch "No" to operation 406, where a default action is performed. For example, as described above, one of the first view or the second view from one or more input video streams can be output without further processing to reduce the computational workload of the mechanism described herein. Additionally or alternatively, in some examples, the first view or the second view can have an associated preconfigured action. In other examples, method 400 may include determining whether the first view or the second view has an associated default action, so that in some cases, no action is performed due to the first view and / or the second view of the identified first subject. Method 400 may terminate at operation 406. Alternatively, method 400 may return to operation 402 to provide a continuous video stream feedback loop.

[0065] However, if it is determined that the second view is substantially different from the first view, then the flow instead branches to "yes" to operation 416, where the first body is segmented into a plurality of planar objects based on the first view and the second view of the first body. The plurality of planar objects may be a plurality of kanban boards. Figure 3As described, the multiple boards can be planes (e.g., two-dimensional planes and / or curved surfaces) that are translated relative to each other. In some examples, a three-dimensional avatar of a first subject in a folded configuration is generated by combining multiple views (e.g., a first view and a second view) to provide a perspective of the first subject to one or more viewers. Therefore, each of the multiple boards can face the camera (e.g., a planar surface of each of the multiple boards faces the camera), so that a user can see the content on each of the multiple boards (e.g., a participant in a hybrid meeting views a virtual avatar formed by multiple boards).

[0066] The first object may include one or more pivot points about which two or more parts of the first body are configured to rotate. For example, if the first body is a person or an animal, the one or more pivot points may be joints about which the body parts are configured to rotate. The one or more planar objects may be split at the pivot points. In addition, the one or more planar objects may be connected via virtual joints. The virtual joints may be joint points of rotation corresponding to physical joints of the first body.

[0067] One or more planar objects (e.g., one or more billboards) may be bent toward one another, stretched to overlap or intersect one another, blurred toward one another, blended toward one another, discretely abut one another, or otherwise visually manipulated to form a subject (e.g., in the form of a generated avatar) that corresponds to a physical subject (e.g., a first subject within one or more input video streams).

[0068] Additionally or alternatively, a machine learning model may be used to segment the first subject into a plurality of planar objects. The machine learning model may be trained to generate segmented planar objects (e.g., billboards) based on a plurality of variables. For example, the plurality of variables may include a computational cost of generating the segmented planar objects, one or more input video streams, an error between a generated avatar formed by the segmented planar objects and a physical subject, and the like, the physical subject being within one or more input video streams corresponding to the generated avatar. Although generating a relatively large number of planar objects may reduce the error between the generated avatar and the physical subject, generating a relatively large number of planar objects may also increase computational overhead. Therefore, the training of the machine learning model may also depend on policy factors that weight variables from a plurality of variables based on user preferences and / or economic factors.

[0069] At operation 418, the plurality of planar objects may be output. The plurality of planar objects may be output in an output video stream. For example, the plurality of planar objects may form a generated three-dimensional avatar that is output in the output video stream. The output video stream may be output to a computing device (e.g., computing device 102), which may be, for example, a display screen and / or a head mounted display (e.g., HMD 216) of a computer with which one or more users are interacting in a video conference.

[0070] In general, method 400 provides an example of a subject that can be generated using segmented planar objects that are arranged or transformed relative to each other. The segmented planar objects can be envisioned as an origami configuration, where each segmented planar object is folded relative to each other to provide a three-dimensional avatar of a user that includes perspective from one or more views. Such a method can improve existing methods of generating three-dimensional avatars with less computational cost, while still providing perspective information to the viewer that improves the user experience.

[0071] The method 400 may terminate at operation 418. Alternatively, the method 400 may return to operation 402 to provide a continuous video stream feedback loop. The method 400 may run continuously. Alternatively, the method 400 may iterate at specified intervals for regular or irregular durations. Additionally or alternatively, the method 400 may be triggered to execute by a specific action, such as by a computing device receiving an input video stream, and / or by a user executing a specific command.

[0072] Figure 5 An example 500 of generating one or more views of a subject according to some aspects described herein is shown. Example 500 includes a first subject 502, a second subject 504, and a third subject 506 within a room 508. In some examples, room 508 can be a virtual room. Alternatively, in some examples, room 508 can be a physical room. First subject 502, second subject 504, and third subject 506 can be part of a hybrid conference (such as hybrid conference 200), as previously described herein with respect to Figure 2 described.

[0073] In some examples, the first subject 502, the second subject 504, and the third subject 506 are physically present in the room 508. Alternatively, in some examples, one or more of the first subject 502, the second subject 504, and the third subject 506 are virtually present in the room 508. For example, one or more of the first subject 502, the second subject 504, and the third subject 506 can join the hybrid conference occurring in the room 508 via one or more computing devices 510. The one or more computing devices 510 can be similar to the previous description of Figure 1The computing device 102 described above. The one or more computing devices 510 can be coupled to or otherwise communicate with one or more video data sources 512. The one or more video data sources 512 can be similar to those previously described with respect to Figure 1 Described video data source 106. Additionally, in some examples, one or more of first subject 502, second subject 504, and third subject 506 may be generated avatars that exist in a virtual and / or augmented reality environment.

[0074] The first subject 502 may be viewed from a plurality of views, such as a first view 520 and / or a second view 530. The first view 520 may correspond to the position of the second subject 504. That is, the first view 520 of the first subject 502 may correspond to the perspective of the second subject 504 viewing the first subject 502. In a virtual environment, as part of a hybrid conference, the first view 520 may be generated to provide a user participation experience for the hybrid conference, as if the second subject 504 is viewing the first subject 502 in person, even if viewed in a virtual reality or augmented reality environment.

[0075] The second view 530 may correspond to the position of the third subject 506. That is, the second view 530 of the first subject 502 may correspond to the perspective of the third subject 506 viewing the first subject 502. In a virtual environment, as part of a hybrid conference, the second view 530 may be generated to provide a user participation experience for the hybrid conference as if the third subject 506 is viewing the first subject 502 in person, even if it is viewed in a virtual reality or augmented reality environment.

[0076] The positions of the first body 502, the second body 504, and the third body 506 can be configured by the user. Additionally or alternatively, after the initial configuration, the user can update the positions. In addition, the positions can be determined based on the geometry of the room 508. For example, the room 508 can have a pre-configured shape, size, angle, or other geometric feature, by which the positions of the first body 502, the second body 504, and the third body 506 can be defined.

[0077] In some examples, the first view 520 and the second view 530 may be determined based on an input video stream (e.g., corresponding to the video data source 512). However, in some instances, the first view 520 and the second view 530 may be generated additionally and / or alternatively based on video signal processing. For example, a machine learning model may be trained to predict or estimate one or more views (e.g., the first view 520 and / or the second view 530) of a subject (e.g., subject 502) based on an input video stream. The training of the machine learning model may include pixel approximations (e.g., position, color, etc.) and / or semantic associations (e.g., what the pixel arrangement resembles) corresponding to the input video stream.

[0078] Figure 6 An example method 600 is shown according to some aspects described herein. The method 600 can be a method of representing a two-dimensional image (e.g., an image of a user) as a three-dimensional avatar or subject in a three-dimensional environment and / or displaying a user in a video conference (such as a video conference as part of a hybrid conference). In an example, aspects of the method 600 are as described above with respect to Figure 1 The discussed computing device 102 and / or server 104 may be executed by a device such as the computing device 102 and / or server 104.

[0079] Method 600 begins at operation 602, where a plurality of input video streams are received. For example, the input video streams may each be from a video data source (e.g., Figure 1 The operation 602 may include obtaining an input video stream. For example, the input video stream may be obtained by executing a command via a processor, which causes the input video stream to be received by, for example, a real subject identification component (such as real subject identification component 114).

[0080] In some examples, the input video stream may include a 3-dimensional (3D) scene. Therefore, the various aspects of the disclosure described herein (e.g., identification, segmentation, tracking, etc.) can be applied to a 3D scene in a manner similar to that applied to a 2D scene. For example, the system described herein can identify a 3D sub-scene (e.g., containing a person or object or animal of interest) of an input video stream and segment or segment subjects within the 3D sub-scene.

[0081] At operation 604, the geometry of the room is identified. For example, the room may correspond to a hybrid meeting, such as hybrid meeting 200. The geometry of the room may include the shape of the room, the dimensions of the room, and / or the relative positions of objects (e.g., computing devices, people, animals, etc.) within the room.

[0082] At determination 608, it is determined whether the input video stream contains a subject (e.g., one or more people, animals, objects, etc.). The subject can be identified by one or more computing devices (e.g., computing device 102 and / or server 104). Specifically, one or more computing devices can receive visual data from a visual data source (e.g., video data source 106) to identify a person. The visual data can be processed using the mechanisms described herein to identify the presence of one or more people of interest, one or more animals of interest, and / or one or more objects of interest. One or more people, one or more animals, and / or one or more objects can be identified based on the presence of people, animals, and / or objects, the movement of people, animals, and / or objects, and / or other scene attributes of the input video stream (such as color, contrast, lighting, spacing between subjects, etc.).

[0083] Additionally or alternatively, at determination 608, one or more subjects may be identified by interaction with specific software (e.g., joining a call, joining a video call, joining a chat, etc.). For example, a person may be identified by logging into a specific application (e.g., via a password, biometric input, or registration number). Thus, when logging into a specific application, the person is thereby identified. For example, when a person joins a video call, it may be determined that the input video stream contains the subject (i.e., the person). Additionally or alternatively, at determination 608, the subject may be identified by identifying the subject using a radio frequency identification tag (RFID), an ID badge, a barcode, or some other identification means capable of identifying the subject via some technical interface, thereby determining that the input video stream contains the subject.

[0084] It should be appreciated that method 600 is provided as an example of identifying or not identifying a subject at determination 608. In other examples, it may be determined that clarification or disambiguation is requested from the user (e.g., before proceeding to operations 610 or 612), such as when a confidence level associated with identifying a subject is below a predetermined threshold, or when multiple subjects are identified, etc. In examples where such clarifying user input is received, an indication of the subject may be stored (e.g., associated with the received input video stream) and used to improve accuracy when processing similar future video stream inputs.

[0085] If it is determined that the subject is not contained in the input video stream, the process will enter the branch "No" to operation 610, where a default action is performed. For example, the input video stream may have an associated preconfigured action. In other examples, the method 600 may include determining whether the input video stream has an associated default action, so that in some cases, no action is performed due to the received input video stream (e.g., conventional methods can be used to display one or more input video streams). The method 600 may terminate at operation 610. Alternatively, the method 600 may return to operation 602 to provide a continuous video stream feedback loop.

[0086] However, if it is determined that the input video stream contains a subject, the flow will enter the "yes" branch to operation 612, where a first subject is identified at a first position in the room. The first subject can be a person, an animal, an object, etc. In addition, the first subject can be a virtual person, an animal, an object, etc. or a real (e.g., physical) person, an animal, an object, etc. In addition, the first subject can be identified in each of the multiple input video streams, such as by visually identifying common features within each of the input video streams corresponding to the first subject.

[0087] At operation 614, a first view of a first subject is determined based on a first input video stream of a plurality of input video streams. The first view may be similar to Figure 5 5. A first view 520 is shown. A first input stream in a plurality of input streams may correspond to a first computing device in a plurality of computing devices. The first view may be determined relative to a second location in a room. The second location in the room may be a location where a second subject (e.g., a person or a computing device corresponding to the person) is located. Thus, the first view of the first subject may correspond to a perspective of the second subject (e.g., the person looking at the first subject, in person, through a computing device, and / or through an HMD).

[0088] At operation 616, a second view of the first subject is determined based on a second input video stream in the plurality of input video streams. The second view may be similar to Figure 5 5. A second view 530 is shown. A second input stream in the plurality of input streams may correspond to a second computing device in the plurality of computing devices. The second view may be determined relative to a third location in the room. The third location in the room may be where a third subject (e.g., a person or a computing device corresponding to the person) is located. Thus, the second view of the first subject may correspond to the perspective of the third subject (e.g., the person looking at the first subject, in person, through a computing device, and / or through an HMD).

[0089] At determination 618, it is determined whether the second view of operation 616 is substantially different from the first view of operation 614. Determining whether the second view is substantially different from the first view may include calculating an error between a visual component of the first view and a visual component of the second view. A user may configure a predetermined threshold at which the second view is determined to be substantially different from the first view.

[0090] If it is determined that the second view is not substantially different from the first view, the process will enter the branch "No" to operation 610, where a default action is performed. For example, one of the multiple input video streams can be output without further processing to reduce the computational workload of the mechanism described herein. Additionally or alternatively, in some examples, the first view or the second view can have an associated preconfigured action. In other examples, method 400 may include determining whether the first view or the second view has an associated default action, so that in some cases, no action is performed due to the first view and / or second view of the identified first subject. Method 600 may terminate at operation 610. Alternatively, method 600 may return to operation 602 to provide a continuous video stream feedback loop.

[0091] However, if it is determined that the second view is substantially different from the first view, flow instead branches “YES” to operation 620 where the first view of the first subject is output based on the second position in the room.

[0092] In general, using the mechanisms disclosed herein, a first person or first participant in a hybrid conference can be projected or otherwise generated in a room where other participants (such as a second and third person) may be viewing (e.g., via a head mounted display or other computing device). Based on the relative position of each participant, different views of the first person can be output to computing devices corresponding to the second and third persons within the hybrid conference.

[0093] The method 600 may terminate at operation 620. Alternatively, the method 600 may return to operation 602 to provide a continuous video stream feedback loop. The method 600 may run continuously. Additionally or alternatively, the method 600 may iterate at specified intervals for regular or irregular durations. Alternatively, the method 600 may be triggered to execute by a specific action, such as by a computing device receiving an input video stream, and / or by a user executing a specific command.

[0094] Figure 7 to Figure 1 0 and the associated description provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figure 7 to Figure 1 The devices and systems shown and discussed are for purposes of example and explanation and are not limiting of the wide variety of computing device configurations that may be used to practice the various aspects of the present disclosure described herein.

[0095] Figure 7 700 is a block diagram showing the physical components (eg, hardware) of a computing device 700 that can be used to practice various aspects of the present disclosure. The computing device components described below may be applicable to the computing devices described above, including Figure 11. In a basic configuration, computing device 700 may include at least one processing unit 702 and system memory 704. Depending on the configuration and type of computing device, system memory 704 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories.

[0096] The system memory 704 may include an operating system 705 and one or more program modules 706 suitable for running software applications 720, such as one or more components supported by the system described herein. As an example, the system memory 704 may store a real subject identification engine 724 and / or a virtual subject generation engine 726. The operating system 705, for example, may be suitable for controlling the operation of the computing device 700.

[0097] Furthermore, aspects of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. Figure 7 708. The computing device 700 may have additional features or functionality. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage Figure 7 , which is illustrated by a removable storage device 709 and a non-removable storage device 710.

[0098] As described above, a number of program modules and data files may be stored in system memory 704. When executed on processing unit 702, program modules 706 (e.g., application programs 720) may perform processes including, but not limited to, various aspects as described herein. Other program modules that may be used in accordance with various aspects of the present disclosure may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, and the like.

[0099] In addition, aspects of the present disclosure may be implemented in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. For example, aspects of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 7Each or more of the components shown in can be integrated onto a single integrated circuit. Such a SOC identification may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functionality described herein regarding the capabilities of the client-to-switch protocol can be operated via dedicated logic integrated with other components of the computing device 600 on a single integrated circuit (chip). Some aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, some aspects of the present disclosure may be practiced within a general-purpose computer or any other circuit or system.

[0100] The computing device 700 may also have one or more input devices 712, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. The aforementioned devices are examples, and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0101] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures or program modules). System memory 704, removable storage device 709 and non-removable storage device 710 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 700. Any such computer storage media may be part of computing device 700. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0102] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal having one or more characteristics that are set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0103] Fig. 8A and Figure 8B A mobile computing device 800 is shown, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., on which some aspects of the present disclosure may be practiced. In some aspects, the client may be a mobile computing device. Fig. 8A , illustrates one aspect of a mobile computing device 800 for implementing various aspects. In a basic configuration, the mobile computing device 800 is a handheld computer having both input elements and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow a user to enter information into the mobile computing device 800. The display 805 of the mobile computing device 800 may also be used as an input device (e.g., a touch screen display).

[0104] If included, the optional side input element 815 allows for additional user input. The side input element 815 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 800 can incorporate more or fewer input elements. For example, in some examples, the display 805 may not be a touch screen.

[0105] In yet another alternative example, the mobile computing device 800 is a portable telephone system, such as a cellular phone. The mobile computing device 800 may also include an optional keypad 835. The optional keypad 835 may be a physical keypad or a "soft" keypad generated on a touch screen display.

[0106] In various examples, output elements include a display 805 for showing a graphical user interface (GUI), a visual indicator 820 (e.g., a light emitting diode), and / or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 includes a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 800 includes input and / or output ports for sending or receiving signals to or from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0107] Figure 8B 802 is a block diagram illustrating an architecture of one aspect of mobile computing identification. That is, a mobile computing device 800 can be incorporated into a system (e.g., architecture) 802 to implement some aspects. In one example, the system 802 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 802 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0108] One or more application programs 866 may be loaded into the memory 862 and run on or in association with the operating system 864. Examples of application programs include a phone dialer program, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 may be used for persistent information that should not be lost if the system 502 loses power. The application programs 866 may use and store information in the non-volatile storage area 868, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 868 synchronized with the corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 862 and run on the mobile computing device 800 described herein (e.g., a real subject identification engine, a virtual subject generation engine, and the like).

[0109] System 802 has a power source 870, which can be implemented as one or more batteries. Power source 870 can also include an external power source, such as an AC adapter or a powered docking station that replenishes or recharges the batteries.

[0110] The system 802 may also include a radio interface layer 872 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 872 supports wireless connectivity between the system 802 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the radio interface layer 872 can be propagated to the application 866 via the operating system 864, and vice versa.

[0111] The visual indicator 820 can be used to provide visual notifications, and / or the audio interface 874 can be used to generate audible notifications via the audio transducer 825. In the example shown, the visual indicator 820 is a light emitting diode (LED), and the audio transducer 825 is a speaker. These devices can be directly coupled to the power supply 870 so that when activated, even if the processor 860 and / or the dedicated processor 861 and other components may be turned off to save battery power, they are also kept turned on for the duration indicated by the notification mechanism. The LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 874 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 874 can also be coupled to a microphone to receive audible input, such as supporting a telephone conversation. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to support the control of notifications, as described below. The system 802 can also include a video interface 876, which enables the vehicle-mounted camera 830 to operate to record still images, video streams, etc.

[0112] The mobile computing device 800 implementing the system 802 may have additional features or functionalities. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 8B , shown by non-volatile storage area 868.

[0113] Data / information generated or captured by the mobile computing device 800 and stored via the system 802 may be stored locally on the mobile computing device 800, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 872, or via a wired connection between the mobile computing device 800 and a separate computing device associated with the mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 800 via the radio interface layer 872 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage methods, including electronic mail and collaborative data / information sharing systems.

[0114] Fig. 9One aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer 904, a tablet computing device 906, or a mobile computing device 908, is shown, as described above. The content displayed at the server device 902 may be stored in different communication channels or other storage types. For example, a directory service 922, a web portal 924, a mailbox service 926, an instant messaging storage 928, or a social networking site 930 may be used to store various documents.

[0115] The real subject identification engine 920 can be employed by a client communicating with the server device 902, and / or the virtual subject generation engine 921 can be employed by the server device 902. The server device 902 can provide data to and receive data from client computing devices such as personal computers 904, tablet computing devices 906, and / or mobile computing devices 908 (e.g., smart phones) via the network 915. As an example, the above-described computer system can be embodied in a personal computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smart phone). In addition to receiving graphics data that can be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these examples of computing devices can obtain content from storage 916.

[0116] For example, the above reference is according to the block diagram and / or operation diagram of the method, system and computer program product of various aspects of the present disclosure to describe various aspects of the present disclosure. The function / action indicated in the frame may not occur in the order shown in any flow chart. For example, two blocks shown in succession can actually be performed substantially simultaneously, or these blocks can sometimes be performed in reverse order, depending on the functionality / action involved.

[0117] The present disclosure relates to systems and methods for representing a two-dimensional representation as a three-dimensional avatar according to at least the examples provided in the following sections.Any of the one or more examples provided herein may be used in combination with any other of the one or more examples provided herein.

[0118] In some examples, a method for creating a virtual avatar is provided. The method includes: receiving one or more input video streams; identifying a first subject within the one or more input video streams; identifying a first view of the first subject based on the one or more input video streams; identifying a second view of the first subject based on the one or more input video streams; and segmenting the first subject into a plurality of planar objects. The plurality of planar objects are transformed relative to each other, and the plurality of planar objects are based on the first view and the second view of the first subject. The method also includes outputting the plurality of planar objects in an output video stream. The plurality of planar objects provide a perspective of the first subject to one or more viewers.

[0119] In some examples, a method for creating a virtual avatar is provided. The method includes: receiving multiple input video streams; identifying a first subject within the multiple input video streams; based on the multiple input video streams, identifying multiple views of the first subject; and segmenting the first subject into multiple planar objects. The multiple planar objects are transformed relative to each other, and the multiple planar objects are based on the multiple views of the first subject. The method also includes outputting the multiple planar objects in an output video stream. The multiple planar objects provide a perspective of the first subject to one or more viewers.

[0120] In some examples, the plurality of planar objects are billboards.

[0121] In some examples, each sign faces the camera.

[0122] In some examples, the planar objects are connected via virtual joints, and the virtual joints are articulation points of rotation.

[0123] In some examples, one or more input video streams are received from one or more cameras.

[0124] In some examples, the first subject is identified within a room. The room is a hybrid meeting space.

[0125] In some examples, the hybrid meeting space includes multiple participants disposed therein, the multiple participants being participants from two or more of the following group: physically in the room, virtually displayed within the room, and represented within the room via an avatar.

[0126] In some examples, the first subject is one of a person, an animal, or an object.

[0127] In some examples, the first body includes one or more pivot points, the two or more portions of the first body are configured to rotate about the one or more pivot points at the one or more pivot points, and the plurality of planar objects are split at the pivot points.

[0128] In some examples, at least one of the plurality of kanbans is a curved surface.

[0129] In some examples, the output video stream is output to a head mounted display (HMD).

[0130] In some examples, a method is provided. The method includes receiving a plurality of input video streams as part of a hybrid conference, identifying a geometry of a room corresponding to the hybrid conference, identifying a first subject at a first location in the room, determining a first view of the first subject relative to a second location in the room based on a first input video stream of the plurality of input video streams, determining a second view of the first subject relative to a third location in the room based on a second input video stream of the plurality of input video streams, and outputting the first view of the first subject based on the second location in the room.

[0131] Some examples also include identifying a second subject at a second location in the room and identifying a third subject at a third location in the room. The first view of the first subject corresponds to the perspective of the second subject. In addition, the second view of the first subject corresponds to the perspective of the third subject.

[0132] In some examples, one or more of the first subject, the second subject, and the third subject are people.

[0133] In some examples, a second view of the first subject is output based on a third position in the room. In addition, outputting the second view is received by a computing device corresponding to a third subject in the room.

[0134] In some examples, outputting the first view is received by a computing device corresponding to a second subject in the room.

[0135] In some examples, multiple input video streams are received from multiple cameras.

[0136] In some examples, a hybrid conference includes multiple participants who are two or more of the following group: physically in the room, virtually displayed within the room, and identified within the room via an avatar.

[0137] In some examples, a system is provided. The system includes a head mounted display (HMD), a display screen, at least one processor, and a memory storing instructions, which when executed by the at least one processor causes the system to perform a set of operations. The set of operations includes: receiving one or more input video streams; identifying a first subject within the one or more input video streams; identifying a first view of the first subject based on the one or more input video streams; identifying a second view of the first subject based on the one or more input video streams; outputting the first view of the first subject to the HMD; and outputting the second view of the first subject to the display screen.

[0138] In some examples, the HMD is located at a first location within the hybrid meeting space and the display is located at a second location within the hybrid meeting space.

[0139] In some examples, the first view of the first subject corresponds to the first position and the second view of the first subject corresponds to the second position.

[0140] In some examples, the HMD is one of an augmented reality head-mounted device or a virtual reality head-mounted device.

[0141] In some examples, one or more input video streams are received as part of a hybrid conference.

[0142] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership, and enable others to make and use the claimed aspects of the present disclosure. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example or detail provided in this application. Regardless of being shown and described in combination or separately, various features (both structure and method) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. Description and explanation of the present application have been provided, and those skilled in the art can envision changes, modifications and alternative aspects within the spirit of the broader aspects of the general inventive concept embodied in this application, which do not depart from the broader scope of the disclosure claimed for protection.

Claims

1. A method for creating a virtual avatar, the method comprising: receiving one or more input video streams; identifying a first subject within the one or more input video streams; identifying, based on the one or more input video streams, a first view of the first subject; identifying a second view of the first subject based on the one or more input video streams; segmenting the first subject into a plurality of planar objects, the plurality of planar objects being transformed relative to each other and the plurality of planar objects being based on the first view and the second view of the first subject; as well as The plurality of planar objects are output in an output video stream, whereby the plurality of planar objects provide a perspective of the first subject to one or more viewers. The method according to claim 1 , wherein the plurality of planar objects are billboards. The method of claim 2 , wherein each of the billboards faces a camera. The method of claim 1 , wherein the planar objects are connected via virtual joints, and wherein the virtual joints are articulation points of rotation. The method of claim 1 , wherein the first subject is identified within a room, and wherein the room is a hybrid meeting space.

6. The method of claim 5, wherein the hybrid conference space includes a plurality of participants disposed therein, the plurality of participants being participants from two or more of the following groups: physically in the room, virtually displayed in the room, and represented in the room via a virtual avatar.

7. The method of claim 1, wherein the first body comprises one or more pivot points, two or more portions of the first body are configured to rotate around the one or more pivot points at the one or more pivot points, and wherein the plurality of planar objects are split at the pivot points.

8. A method for creating a virtual avatar, the method comprising: receiving multiple input video streams; identifying a first subject within the plurality of input video streams; identifying a plurality of views of the first subject based on the plurality of input video streams; segmenting the first subject into a plurality of planar objects, the plurality of planar objects being transformed relative to each other and the plurality of planar objects being based on the plurality of views of the first subject; as well as The plurality of planar objects are output in an output video stream, whereby the plurality of planar objects provide a perspective of the first subject to one or more viewers.

9. The method of claim 8, wherein the planar objects are connected via virtual joints, and wherein the virtual joints are articulation points of rotation.

10. The method of claim 8, wherein the first subject is identified within a room, and wherein the room is a hybrid meeting space.

11. The method of claim 10, wherein the hybrid conference space includes a plurality of participants disposed therein, the plurality of participants being participants from two or more of the following groups: physically in the room, virtually displayed in the room, and represented in the room via a virtual avatar.

12. A system, comprising: Head-mounted display (HMD); Display screen; at least one processor; a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: receiving one or more input video streams; identifying a first subject within the one or more input video streams; identifying, based on the one or more input video streams, a first view of the first subject; identifying a second view of the first subject based on the one or more input video streams; and outputting the first view of the first subject to the HMD; and The second view of the first body is output to the display screen.

13. The system of claim 12, wherein the HMD is located at a first location within a hybrid conference space, and wherein the display screen is located at a second location within the hybrid conference space.

14. The system of claim 13, wherein the first view of the first subject corresponds to the first position, and wherein the second view of the first subject corresponds to the second position.

15. The system of claim 12, wherein the HMD is one of an augmented reality head mounted device or a virtual reality head mounted device.

Images