System, method and storage medium based on the existence of spatial video

Abstract

A system for implementing spatial video-based virtual presence, comprising: one or more server computers implementing a virtual environment platform including at least one virtual environment; at least one sending client device generating a user real-time 3D virtual cutout by obtaining a 2D or 3D live video data feed and performing background removal thereof; and transmitting the user real-time 3D virtual cutout for presentation by at least one receiving client device; and at least one receiving client device receiving the user real-time 3D virtual cutout; and inserting and combining the user real-time 3D virtual cutout into a virtual environment, wherein the virtual environment is shared by at least one sending client device and at least one receiving client device and the corresponding user real-time 3D virtual cutout, so that a multi-user shared experience of spatial video-based virtual presence can be implemented within the virtual environment. A method for implementing spatial video-based presence is also disclosed herein.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 17 / 006,327 filed on August 28, 2020, and U.S. patent application No. 17 / 160,209 filed on January 27, 2021, which are incorporated herein by reference. Background Art

[0003] As situations such as the 2020 novel coronavirus pandemic force restrictions on the movement of people around the world, changing the way we meet, learn, shop and work, remote collaboration and interaction, especially social interaction, have become increasingly important. There are already a variety of solutions on the market to enable real-time communication and collaboration, from chat applications to video calls such as Skype TM and Zoom TM , or virtual offices for remote teams represented by 2D avatars, such as Pragli TM Virtual Offices Available.

[0004] Considering the current state of development of wearable immersive technologies such as extended reality (e.g., augmented and / or virtual reality) and the relatively low technology adoption rates, it is understandable that most solutions provide a flat, two-dimensional user interface on which most interactions occur. However, when comparing real-life experiences to these solutions, the low level of realism, lack of user presence, lack of shared space, and quality of interaction can cause many users to feel lonely or bored, which in turn sometimes leads to lower productivity than when performing the same activities in person.

[0005] What is needed is a technology solution that provides users with a sense of reality, their own and participants' presence, and a sense of interaction as if they were in real life when interacting remotely, without requiring the purchase of expensive equipment (e.g., in a head-mounted display) and the implementation of new or expensive infrastructure, all while using existing computing devices and cameras. Summary of the invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] The present disclosure relates generally to computer systems, and more particularly to a system and method for enabling interactions, particularly social interactions, in a virtual environment; a system and method for virtual presence based on image processing; a system and method for user authentication based on user graphical representations; a system and method for virtually broadcasting from within a virtual environment; a system and method for delivering applications within a virtual environment; a system and method for providing cloud-based virtual computing resources within a virtual environment cloud server computer; and a system and method for enabling self-organizing virtual communications between proximate user graphical representations.

[0008] The disclosed system for implementing interactions in a virtual environment, particularly including social interactions, includes one or more cloud server computers, the cloud server computers including at least one processor and a memory storing data and instructions for implementing a virtual environment platform including at least one virtual environment. The one or more cloud server computers are configured to insert a user graphical representation generated from a live data feed obtained by a camera at a three-dimensional coordinate location of at least one virtual environment, update the user graphical representation in at least one virtual environment, and implement real-time multi-user collaboration and interaction in the virtual environment.

[0009] In an embodiment, the system further includes at least one camera that obtains a live data feed from one or more users of the client device. In addition, the system includes a client device that is communicatively connected to one or more cloud server computers and at least one camera. The system generates a user graphical representation from the live data feed, which is inserted into the three-dimensional coordinates of the virtual environment and updated therein using the live data feed. In the described embodiment, inserting the user graphical representation into the virtual environment involves graphically combining the user graphical representation in the virtual environment so that the user graphical representation appears in the virtual environment (e.g., at a specified 3D coordinate position). The virtual environment platform serves the virtual environment to one or more client devices. The system implements real-time multi-user collaboration and (social) interaction in the virtual environment by accessing a graphical user interface via a client device. The client or peer device of the present disclosure may include, for example, a computer, a headset, a mobile phone, glasses, a transparent screen, a tablet computer, and an input device that is generally built with a camera, or an input device that can be connected to a camera and receives a data feed from the camera.

[0010] In some embodiments, the virtual environment is accessible by a client device via a downloadable client application or a web browser application.

[0011] In some embodiments, the user graphic representation includes a user 3D virtual cutout with the background removed, or a user real-time 3D virtual cutout with the background removed, or a video with the background removed, or a video without the background removed. In some embodiments, the user graphic representation is a user 3D virtual cutout with the background removed, constructed from a user uploaded or third-party source photo, or a user real-time 3D virtual cutout with the background removed, generated based on real-time 2D, stereo, depth data or 3D live video stream data feed obtained from a camera, and thus includes a user's real-time video stream, or a video without the background removed, or a video with the background removed and displayed using a polygonal structure. Such polygonal structures can be quadrilateral structures or more complex 3D structures, used as virtual frames to support videos. In other embodiments, one or more such user graphic representations are inserted into three-dimensional coordinates within a virtual environment and graphically combined therein.

[0012] The user 3D virtual cut may include a virtual copy of the user constructed from a 2D photo uploaded by the user or from a third-party source. In an embodiment, a 2D photo uploaded by the user or from a third-party source is used as input data, and a user 3D virtual cut is created through a 3D virtual reconstruction process by machine vision technology, thereby generating a 3D mesh or 3D point cloud of the user with the background removed. The user real-time 3D virtual cut may include a virtual copy of the user based on a real-time 2D or 3D live video stream data feed obtained from a camera and after removing the user background. In an embodiment, the user real-time 3D virtual cut is created by generating a 3D mesh or 3D point cloud of the user with the background removed, using the user live data feed as input data, and through a 3D virtual reconstruction process by machine vision technology. The video with the background removed may include a video streamed to a client device, in which a background removal process has been performed on the video so that only the user can be visible, and then displayed using a polygonal structure on the receiving client device. The video without the background removed may include a video streamed to a client device, in which the video faithfully represents the camera capture, so that the user and his or her background are visible, and then displayed using a polygonal structure on the receiving client device.

[0013] In some embodiments, data used as input data for inclusion in a live data feed and / or user uploaded or third party sourced 2D photos includes 2D or 3D image data, 3D geometry, video data, media data, audio data, text data, tactile data, temporal data, 3D entities, 3D dynamic objects, text data, temporal data, metadata, priority data, security data, position data, lighting data, depth data, infrared data, and the like.

[0014] In some embodiments, the user graphical representation is associated with a bird's-eye view, or a third-person view, or a first-person view, or a self-perspective. In an embodiment, the perspective of the user when accessing the virtual environment through the user graphical representation is a bird's-eye view, or a third-person view, or a first-person view, or a self-perspective, or a broadcast camera view. The self-perspective may include a user graphical representation seen by another user graphical representation, and optionally, a virtual background of the user graphical representation.

[0015] In other embodiments, the perspective is updated as the user manually navigates the virtual environment via the graphical user interface.

[0016] In other embodiments, the perspective is automatically established and updated by using a virtual camera, wherein the perspective of the live data feed is associated with the perspective of the user's graphical representation and the virtual camera, and wherein the virtual camera is automatically updated by tracking and analyzing the user's eye and head tilt data or head rotation data or a combination thereof. In an embodiment, the perspective is automatically established and updated by using one or more virtual cameras, which are virtually placed and aligned in front of the user's graphical representation, for example, in front of a video without background removal, or a video with background removal, or a user's 3D virtual cut, or a user's real-time 3D virtual cut. In one embodiment, one or more virtual cameras can point outward from eye level. In another embodiment, two virtual cameras, one for each eye, point outward from both eye levels. In another embodiment, one or more virtual cameras can point outward from the center of the user's graphical representation's head position. In another embodiment, one or more virtual cameras can point outward from the center of the user's graphical representation. In another embodiment, one or more virtual cameras can be placed in front of the user's graphical representation, for example, at the head level of the user's graphical representation, pointing to the user's graphical representation when in self-perspective. The user's perspective captured by the camera is associated with the perspective of the user graphical representation and the associated virtual camera using computer vision, thereby manipulating the virtual camera. In addition, the virtual camera is automatically established and updated by tracking and analyzing user eye and head tilt data, or head rotation data, or a combination thereof.

[0017] In other embodiments, the self-perspective includes a cut-out of the user graphical representation seen by another user graphical representation with the background removed (e.g., in a "selfie mode" of a phone camera). Alternatively, the self-perspective includes a virtual background of the virtual environment behind the user graphical representation for understanding the perception of himself or herself as visible to other participants. When including a virtual background of the user graphical representation, the self-perspective can be set to the area around the user graphical representation that can be captured by the virtual camera, which can produce a circle, square, rectangle, or any other suitable shape for constructing the self-perspective.

[0018] In some embodiments, the updating of the user graphical representation in the virtual environment includes updating the user status. In an embodiment, the available user status includes not being present, busy, available, offline, in a conference call, or in a meeting. The user status can be manually updated through a graphical user interface. In other embodiments, the user status is automatically updated via a connection to the user calendar information that contains and synchronizes the user status data. In other embodiments, the user status is automatically updated by detecting the use of a specific program, such as a programming development environment, a 3D editor, or other productivity software that specifies a busy state, which can be synchronized according to the user status. In other embodiments, the user status can be automatically updated by a machine vision algorithm based on data obtained from a camera feed.

[0019] In some embodiments, interactions between users represented by corresponding user graphics, particularly including social interactions, include: chatting; screen sharing; host options; remote sensing; recording; voting; document sharing; emoticon sending; agenda sharing and editing; virtual hugs; raising hands; shaking hands; walking; content addition, including interactive applications or static or interactive 3D assets, animations or 2D textures; meeting summary preparation; object movement; content projection; laser pointing; playing games; purchasing; participating in self-organized virtual communications; and participating in private or group conversations.

[0020] In some embodiments, the virtual environment is a persistent virtual environment stored in a persistent memory storage of one or more cloud server computers, or a temporary virtual environment stored in a temporary memory storage of one or more cloud server computers. In embodiments, the virtual environment is a persistent virtual environment that records changes performed thereon, including customizations stored in a persistent memory storage of at least one cloud server computer designated for the persistent virtual environment. In other embodiments, the virtual environment is a temporary virtual environment stored in a temporary memory storage of a cloud server.

[0021] In some embodiments, the arrangement of the virtual environment is associated with a contextual theme of the virtual environment, which contextual theme is related to one or more virtual environment verticals selected from the virtual environment platform. In embodiments, possible arrangements include arrangements for education, meetings, work, shopping, services, socializing, or entertainment, or a combination thereof. A complex of virtual environments within one or more verticals may represent a virtual environment cluster.

[0022] In other embodiments, the virtual environment cluster is one or more of: a virtual school comprising at least a plurality of classrooms; or a virtual company comprising at least a plurality of workspaces and conference rooms, some of which are shared as common working or networking spaces for members of different organizations; or an event facility comprising at least one indoor or outdoor event area that hosts live events including capture of live entertainment performers; or a virtual shopping mall comprising at least a plurality of stores; or a virtual casino comprising at least a plurality of gaming areas; or a virtual bank comprising at least a plurality of service areas; or a virtual nightclub comprising at least a plurality of VIP areas and / or party areas, including capture of live radio host (DJ) performers; or a virtual karaoke entertainment facility comprising a plurality of private or public karaoke rooms; or a virtual cruise ship comprising a plurality of virtual areas within the cruise ship and areas outside the cruise ship, the external areas comprising landscapes, islands, towns and cities, enabling users to get off the virtual cruise ship to visit those virtual areas; or an e-sports stadium or gym.

[0023] In other embodiments, the virtual environment further comprises a virtual computer including virtual resources. In an embodiment, the virtual resources are from one or more cloud computer resources, the virtual resources are accessed through a client device and are assigned to the virtual computer resources through a management tool.

[0024] In some embodiments, the virtual environment platform is configured to be able to multicast or broadcast remote events to multiple instances of the virtual environment. For example, this can be done to accommodate a large number of users from all over the world to experience the same multicast live event.

[0025] In some embodiments, a clickable link that redirects to the virtual environment is embedded in one or more third-party sources including a third-party website, application, or video game.

[0026] In another aspect of the present disclosure, a method for implementing interactions including social interactions in a virtual environment includes: providing a virtual environment platform including at least one virtual environment in the memory of one or more cloud server computers including at least one processor; receiving a live data feed (e.g., a live data feed of a user captured by at least one camera) from at least one corresponding client device; generating a user graphical representation from the live data feed; inserting the user graphical representation into a three-dimensional coordinate position of the virtual environment; updating the user graphical representation within the virtual environment according to the live data feed; and processing data generated by the interaction in the virtual environment. Such interactions may particularly include social interactions in the virtual environment. For such interactions, the method may include serving the updated virtual environment to the client device via direct P2P communication or indirectly by using one or more cloud servers to implement real-time multi-user collaboration and interaction in the virtual environment.

[0027] In some embodiments, the system (e.g., via the virtual environment platform) can also enable the creation of self-organizing virtual communications, which can include creating self-organizing voice communication channels between user graphical representations without changing the current perspective or position in the virtual environment. For example, a user graphical representation can approach another user graphical representation and participate in a self-organizing voice conversation at the location of the two user graphical representation areas within the virtual environment. Such communication will be achieved by, for example, considering the distance, position and orientation between the user graphical representations, and or their current availability status (e.g., available or unavailable) or the state configuration of such self-organizing communication, or a combination thereof. The approaching user graphical representation will see visual feedback on the other user graphical representation, signaling that self-organizing communication is possible, and thus setting the start of a conversation between the two user graphical representations, where the approaching user can speak, and the other user can hear and respond. In another embodiment, the virtual environment platform enables participation in self-organizing virtual communications in a virtual environment by processing data generated in response to steps performed by a client device, which may include the following steps: approaching a user graphical representation; selecting and clicking a user graphical representation; sending or receiving a self-organizing virtual communication participation invitation to another user graphical representation; and accepting the received invitation. In such a scenario, the platform may open a communication channel between the user client devices, with the user graphical representations maintaining a conversation in the virtual space of the virtual environment.

[0028] In some embodiments, the method further includes engaging one or more users in a conversation, transforming a user graphical representation from a user 3D virtual cutout to a user real-time 3D virtual cutout, or a video with a background removed, or a video without a background removed; and opening a peer-to-peer (P2P) communication channel between user client devices. In an embodiment, the step of causing two or more users to participate in a conversation includes: approaching a user graphical representation; selecting and clicking a user graphical representation; sending or receiving a conversation participation invitation to another user graphical representation; and accepting the received invitation. The step of opening a communication channel between user client devices can be performed when processing and rendering are performed by a client device, or the step of opening an indirect communication channel through one or more cloud server computers can be performed when processing and rendering are performed on at least one cloud server computer or between at least one cloud server and a client device. In other embodiments, the conversation includes sending real-time audio to the user 3D virtual cutout of the participant and receiving real-time audio from the user 3D virtual cutout of the participant. In other embodiments, the conversation includes sending and receiving real-time audio and video displayed from a user real-time 3D virtual cutout or video with a background removed, or a participant's video without a background removed.

[0029] In some embodiments, the method of enabling interaction in a virtual environment further comprises embedding a clickable link redirecting to the virtual environment into one or more third-party sources including a third-party website, application, or video game.

[0030] In another aspect of the present disclosure, a data processing system includes one or more computing devices including at least one cloud server computer, the one or more computing devices including at least one processor and a memory storing data and instructions for implementing image processing functions, wherein the one or more computing devices of the data processing system are configured to generate a user graphical representation from a live data feed through one or more image processing combinations of at least one cloud server computer and two or more client devices in a hybrid system architecture. In an embodiment, the system includes: two or more client devices, which are communicatively connected to each other and to one or more cloud server computers via a network, including at least one processor and a memory storing data and instructions for implementing image and media processing functions; and at least one camera, which obtains a live data feed from at least one user of at least one client device and is connected to at least one client device and one or more cloud server computers. The user graphical representation is generated from the live data feed through one or more image and media processing combinations of one or more cloud server computers and one or more client devices. The one or more cloud server computers and one or more client devices interact through the hybrid system architecture.

[0031] In some embodiments, data used as input data for the data processing system includes 2D or 3D image data, 3D geometry, video data, media data, audio data, text data, tactile data, time data, 3D entities, 3D dynamic objects, text data, time data, metadata, priority data, security data, position data, lighting data, depth data, infrared data, etc.

[0032] In some embodiments, the hybrid system architecture includes a client-server side and a peer-to-peer (P2P) side. In an embodiment, the client-server side includes a network or application server. The client-server side may also be configured to include: a secure communication protocol; a microservice; a database management system; a database; and / or a distributed message and resource distribution platform. The server-side component may be provided with a client device that communicates with the server through a network. The client-server side defines the interaction between one or more clients and a server through a network, including any processing performed by a client, a server, or a receiving client. In an embodiment, one or more of the corresponding clients and servers perform the necessary image and media processing according to various rule-based task allocation combinations. In an embodiment, the network or application server is configured to receive client requests using a secure communication protocol and process the client request by requesting microservices or data corresponding to the request from the database using a database management system. Microservices are distributed using a publish-subscribe model using a distributed message and resource distribution platform.

[0033] The P2P end includes: a P2P communication protocol that enables real-time communication between client devices in a virtual environment; and a rendering engine configured to enable the client device to perform real-time 3D rendering of live session elements (e.g., user graphical representations) included in the virtual environment. In an embodiment, the P2P end also includes a computer vision library configured to enable the client device to perform real-time computer vision tasks in the virtual environment. Using such a hybrid communication model can achieve fast P2P communication between users, reduce latency issues, while providing network services and resources to each session, and enable multiple interactions between users and with content in the virtual environment.

[0034] The P2P end defines the interaction between client devices and any processing that can be performed by one or another client device from the P2P end. In some embodiments, the P2P end is used for video and data processing tasks and synchronization, streaming, and rendering between client devices. In other embodiments, the P2P end is used for video streaming, rendering, and synchronization between client devices, while the client-server end is used for data processing tasks. In other embodiments, the client-server end is used for video streaming and data processing tasks, while the P2P end is used for video rendering and synchronization between client devices. In other embodiments, the client-server end is used for video streaming, rendering, and data processing tasks and synchronization.

[0035] In an embodiment, the data processing task includes generating a user graphical representation and inserting the user graphical representation into the virtual environment. Generating the user graphical representation may include performing background removal or other processing or improvement.

[0036] In some embodiments, data on the P2P side is sent directly from one client device to a peer client device, vice versa, or relayed through a server via a client-server connection.

[0037] In some embodiments, at least one cloud server can be an intermediate server, which means that the server is used to facilitate and / or optimize data exchange between client devices. In such embodiments, at least one cloud server can manage, analyze, process and optimize incoming image and multimedia streams, and manage, evaluate, optimize the forwarding of outbound streams as a router topology (for example, but not limited to SFU (Selective Forwarding Unit), SAMS (Spatial Analysis Media Server), multimedia router, etc.), or image and media processing (for example, but not limited to decoding, combining, improving, mixing, enhancing, expanding, calculating, manipulating, encoding) and forwarding server topology (for example, but not limited to, multi-point control unit-MCU, cloud media mixer, cloud 3D renderer, etc.) or other server topology.

[0038] In such embodiments, where the intermediate server is a SAMS, such media server manages, analyzes, and processes incoming data sent to each client device (e.g., including but not limited to metadata, priority data, data categories, spatial structure data, three-dimensional position, orientation or motion information, images, media, video based on a scalable video codec), and in such analysis modifies, amplifies, or reduces the media based on the spatial, three-dimensional orientation, distance, and priority relationship to such incoming data of the specific receiving client device user, thereby optimizing the forwarding of outbound data streams to each receiving client device, achieving optimal bandwidth and computing resource utilization for receiving one or more user client devices.

[0039] In some embodiments, multiple image processing tasks are classified based on whether they are performed by a client device, a cloud server and / or a receiving client device, and are therefore classified as client device image processing, server image processing and receiving client device image processing. Multiple image processing tasks can be performed on the client-server side, the P2P side or a combination thereof of a hybrid architecture. The image processing tasks include background removal, further processing or improvement, and insertion and combination into a virtual environment. The combination of three image processing tasks can be used to generate, improve and insert / combine user graphic representations into a virtual environment. The image processing combination of client device processing, server image processing and receiving client device processing and the corresponding usage level depend on the amount of data to be processed, the waiting time that allows maintaining a smooth user experience, the desired quality of service (QOS), the required services, etc. The following are eight such image processing combinations performed on the client-server side.

[0040] In some embodiments, at least one of the client devices is configured to, in a client-server image processing combination, generate a user graphical representation, perform background removal, and send the user graphical representation with the background removed to at least one cloud server for further processing. In a first illustrative image processing combination, the client device generates a user graphical representation including background removal, and sends the user graphical representation with the background removed to at least one cloud server for further processing or improvement, generating an enhanced user graphical representation with the background removed. The at least one cloud server sends the enhanced user graphical representation with the background removed to a receiving client device, which inserts and combines the enhanced user graphical representation with the background removed into a virtual environment.

[0041] In a second illustrative image processing combination, a client device generates a user graphical representation including background removal, performs further processing thereon, generates an enhanced user graphical representation with background removed, and then sends it to at least one cloud server. The at least one cloud server sends the enhanced user graphical representation with background removed to a receiving client device, which inserts and combines the enhanced user graphical representation with background removed into a virtual environment.

[0042] In a third illustrative image processing combination, a client device generates a user graphical representation including background removal, performs further processing thereon to generate an enhanced user graphical representation with background removed, and inserts and combines the enhanced user graphical representation with background removed into the virtual environment. The client device then sends the enhanced user graphical representation with background removed inserted and combined into the virtual environment to a cloud server for relaying to a receiving client device.

[0043] In a fourth illustrative image processing combination, a client device generates a user graphical representation including background removal and sends the user graphical representation with the background removed to at least one cloud server for further processing to generate an enhanced user graphical representation with the background removed. The at least one cloud server then inserts and combines the enhanced user graphical representation with the background removed into the virtual environment and then sends it to the receiving client device.

[0044] In a fifth illustrative image processing combination, a client device generates a user graphical representation including background removal and sends the user graphical representation with the background removed to at least one cloud server for relaying to a receiving client device. The receiving client device performs further processing on the user graphical representation with the background removed to generate an enhanced user graphical representation with the background removed, and the receiving client device inserts and combines the enhanced user graphical representation with the background removed into the virtual environment.

[0045] In a sixth illustrative image processing combination, a client device sends a camera live data feed received from at least one camera and sends unprocessed data to at least one cloud server, the at least one cloud server performs generation of a user graphical representation including background removal, performs further processing on the user graphical representation with the background removed, generates an augmented user graphical representation with the background removed, and sends it to a receiving client device. The receiving client device inserts and combines the augmented user graphical representation with the background removed into a virtual environment.

[0046] In a seventh illustrative image processing combination, a client device sends a camera live data feed received from at least one camera and sends unprocessed data to at least one cloud server. The at least one cloud server generates a user graphical representation including background removal, performs further processing on the user graphical representation with the background removed, generates an enhanced user graphical representation with the background removed, and then inserts and combines the enhanced user graphical representation with the background removed into a virtual environment that is sent to a receiving client device.

[0047] In an eighth illustrative image processing combination, a client device sends a camera live data feed received from at least one camera to at least one cloud server and sends unprocessed data for relaying to a receiving client device. The receiving client device uses the data to generate a user graphical representation including background removal, and performs further processing on the user graphical representation with the background removed to generate an enhanced user graphical representation with the background removed, and then inserts and combines the enhanced user graphical representation with the background removed into a virtual environment.

[0048] In some embodiments, when client-server data is relayed through at least one cloud server, at least one cloud server is configured to use a relay NAT traversal (TURN) server. TURN can be used in the case of symmetric NAT (network address translation) and can remain in the media path after the connection is established while processed and / or unprocessed data is relayed between client devices.

[0049] Following are descriptions of three illustrative image processing combinations performed on the P2P side by one or both of the first and second peer devices.

[0050] In a first image processing combination, the first peer device generates a user graphical representation including background removal, performs further processing on it, generates an enhanced user graphical representation with background removed, and inserts and combines the enhanced user graphical representation with background removed into the virtual environment. The first peer device then sends the enhanced user graphical representation with background removed, which is inserted and combined into the virtual environment, to the second peer device.

[0051] In a second image processing combination, the first peer device generates a user graphical representation including background removal and sends the user graphical representation with background removed to a second peer device. The second peer device performs further processing on the user graphical representation with background removed to generate an enhanced user graphical representation with background removed, and the second peer device inserts and combines the enhanced user graphical representation with background removed into the virtual environment.

[0052] In a third image processing combination, the first peer device sends a camera live data feed received from at least one camera and sends unprocessed data to a second peer device. The second peer device uses the data to generate a user graphic representation including background removal, and performs further processing on the user graphic representation with the background removed to generate an enhanced user graphic representation with the background removed, and then inserts and combines the enhanced user graphic representation with the background removed into the virtual environment.

[0053] In some embodiments, the three-image processing combination on the P2P side may further include relaying data through at least one cloud server. In these embodiments, at least one cloud server may be configured as a STUN server, which allows peer devices to discover their public IP addresses and the NAT type behind them, which information can be used to establish data connections and data exchanges between peer devices. In another embodiment, at least one cloud server computer may be configured for signaling, which can be used for peer devices to locate and connect to each other, and to exchange data through communication coordination performed by at least one cloud server.

[0054] In some embodiments, the media, video and / or data processing tasks include one or more of encoding, transcoding, decoding spatial or 3D analysis and processing, including one or more of image filtering, computer vision processing, image sharpening, background improvement, background removal, foreground blurring, eye masking, facial pixelation, speech distortion, image magnification, image cleansing, bone structure analysis, face or head counting, object recognition, marker or QR code tracking, eye tracking, feature analysis, 3D mesh or volume generation, feature tracking, facial recognition, SLAM tracking and facial expression recognition, or other modular plug-ins in the form of microservices running on such media routers or servers.

[0055] In some embodiments, background removal includes employing image segmentation through one or more of instance segmentation or semantic segmentation and using deep neural networks.

[0056] In some embodiments, one or more computing devices of the data processing system are further configured to insert the user graphical representation into the virtual environment by generating a virtual camera, wherein generating the virtual camera includes associating the captured perspective data with the perspective of the user graphical representation within the virtual environment. In an embodiment, inserting and combining the user graphical representation into the virtual environment includes generating one or more virtual cameras, which are virtually placed and aligned in front of the user graphical representation, for example, in front of a video with a background removed, or a video without a background removed, or a user 3D virtual cut, or a user real-time 3D virtual cut. In one embodiment, one or more virtual cameras can point outward from eye level. In another embodiment, two virtual cameras, one for each eye, can point outward from both eye levels. In another embodiment, one or more virtual cameras can point outward from the center of the head position of the user graphical representation. In another embodiment, one or more virtual cameras can point outward from the center of the user graphical representation. In another embodiment, one or more virtual cameras can be placed in front of the user graphical representation, for example, at the head level of the user graphical representation, pointing to the user graphical representation when in a self-perspective.

[0057] In an embodiment, one or more virtual cameras are created by at least correlating captured user perspective data with a perspective of a graphical representation of the user within the virtual environment using computer vision.

[0058] In another aspect of the present disclosure, an image processing method includes: providing data and instructions implementing image processing functions in the memory of at least one cloud server computer; and generating a user graphical representation in a virtual environment based on a live data feed from at least one client device through one or more image processing combinations of at least one cloud server computer and at least one client device, wherein at least one cloud server computer interacts with at least one client device through a hybrid system architecture. In an embodiment, the method includes: obtaining a live data feed of at least one user from at least one corresponding client device from at least one camera; and generating a user graphical representation through one or more image processing combinations of one or more cloud server computers and at least one client device. One or more cloud server computers and at least one client device can interact through the hybrid system architecture of the present disclosure including a P2P side and a client-server side.

[0059] In some embodiments, the method includes performing video and data processing and synchronization, streaming and rendering between client devices by the P2P side. In other embodiments, the method includes performing video streaming, rendering and synchronization between client devices by the P2P side, while the client-server side is used for data processing. In other embodiments, the method includes performing video streaming and data processing by the client-server side, while the P2P side is used for video rendering and synchronization between client devices. In other embodiments, the method includes performing video streaming, rendering and data processing and synchronization by the client-server side.

[0060] In some embodiments, the data processing task includes generating a graphical representation of the user and inserting the graphical representation of the user into the virtual environment. In an embodiment, the data processing task includes first generating a graphical representation of the user including performing background removal, then performing further processing, and then inserting and combining into the virtual environment. In other embodiments, the image processing task is performed by a plurality of image processing combinations of a client device and a cloud server computer in a client-server or P2P end.

[0061] In some embodiments, inserting the user graphical representation into the virtual environment includes generating a virtual camera, wherein generating the virtual camera includes associating the captured perspective data with the perspective of the user graphical representation within the virtual environment. In an embodiment, inserting and combining the user graphical representation into the virtual environment includes generating one or more virtual cameras, which are virtually placed and aligned in front of the user graphical representation, for example, in front of a video with a background removed, or a video without a background removed, or a user 3D virtual cut, or a user real-time 3D virtual cut. In one embodiment, one or more virtual cameras can point outward from eye level. In another embodiment, two virtual cameras, one for each eye, can point outward from both eye levels. In another embodiment, one or more virtual cameras can point outward from the center of the head position of the user graphical representation. In another embodiment, one or more virtual cameras can point outward from the center of the user graphical representation. In another embodiment, one or more virtual cameras can be placed in front of the user graphical representation, for example, at the level of the user's head, pointing to the user graphical representation when in a self-perspective. The virtual camera is created at least by associating the captured user perspective data with the perspective of the user graphical representation within the virtual environment using computer vision.

[0062] In some embodiments, the method further comprises embedding, on the graphical representation of the user, a clickable link embedded thereon, the link, in response to being clicked, pointing to a third party source containing profile information about the corresponding user.

[0063] In another aspect of the present disclosure, a user authentication system based on user graphical representation includes: one or more cloud server computers including at least one processor and a memory for storing data and instructions, including a user database storing user data associated with a user account and one or more corresponding user graphical representations, and a facial scanning and authentication module connected to the database; wherein the one or more cloud server computers are configured to perform steps including the following: authenticating a user by performing a facial scan of the user through the facial scanning and authentication module, wherein the facial scan includes extracting facial feature data from camera data received from a client device, and checking the extracted facial feature data for a match with a user graphical representation associated with a user account in the user database; if a matching user graphical representation is found in the checking step, providing the user with access to the corresponding user account; and if no matching user graphical representation is found in the checking step, generating a new user graphical representation from the camera data and a new user account stored in the user database, and access to the user account.

[0064] In an embodiment, the system includes at least one camera configured to obtain data from a user of at least one client device requesting access to a user account, wherein the at least one camera is connected to the at least one client device and one or more cloud server computers. The one or more cloud server computers authenticate the user by performing a facial scan of the user through the facial scanning and authentication module, checking a user database for a match with a user graphical representation, and providing the user with a corresponding user graphical representation and access to the user account if the user account is confirmed and available; and generating a new user graphical representation from the data and a new user account stored in the user database and access to the user account if the user account is not available.

[0065] The user account may be used, for example, to access the virtual environment platform or any other application (e.g., an application that may be linked to the environment platform), such as any interactive application, a game, an email account, a university profile account, a work account, etc. Given the additional authentication step of generating a user graphical representation or retrieving an existing user graphical representation from a user database, the graphical representation-based user authentication system of the present disclosure provides a higher level of security than standard camera-based facial detection authentication systems.

[0066] In some embodiments, the user graphic representation is a user 3D virtual cut, or a user real-time 3D virtual cut with a background removed, or a video with a background removed, or a video without a background removed. In an embodiment, the user graphic representation is a user 3D virtual cut constructed from a user uploaded or third-party source photo, or a user real-time 3D virtual cut with a background removed, or a video with a background removed, or a video without a background removed, generated based on a real-time 2D or 3D live video stream data feed obtained from a camera. In some embodiments, one or more cloud server computers are also configured to animate a matching user graphic representation or a new user graphic representation. Animating a matching user graphic representation includes applying a machine vision algorithm on a corresponding user graphic representation by a client device or at least one cloud server computer to recognize the user's facial expression and graphically simulate facial expressions on the user graphic representation. In other embodiments, the update of the user 3D virtual cut constructed from a user uploaded or third-party source photo includes applying a machine vision algorithm on the generated user 3D virtual cut by a client device or at least one cloud server computer to recognize the user's facial expression and graphically simulate facial expressions on the user 3D virtual cut.

[0067] In some embodiments, one or more cloud server computers are further configured to check the date of the matching user graphical representation and determine whether the matching user graphical representation needs to be updated. In an embodiment, if the user account is available, and in response to one or more cloud server computers checking the date of the available user graphical representation, one or more cloud server computers determine whether the existing user graphical representation needs to be updated by comparing with the corresponding threshold or security requirements. For example, if a system security update is to be performed, all user graphical representations may need to be updated, or at least the graphical representations created before the specified date need to be updated. If a user graphical representation is required, one or more cloud server computers generate a user graphical representation update request to the corresponding client device. If the user approves the request, one or more cloud server computers or client devices continue to generate user graphical representations based on the live camera feed. If no update is required, one or more cloud server computers continue to retrieve the existing user graphical representation from the user database.

[0068] In some embodiments, the user graphical representation is inserted into the two-dimensional or three-dimensional virtual environment, or on a third-party source linked to the virtual environment (e.g., by overlaying on the screen of a third-party application or website that is integrated or coupled with the disclosed system), and combined with the two-dimensional or three-dimensional virtual environment graphics.

[0069] In some embodiments, the generation of the user graphical representation occurs asynchronously with the user accessing the user account. For example, if the system determines that the user has been authenticated after performing facial scanning and detection based on the user graphical representation, the system can enable the user to access the user account while the new user graphical representation is being generated so that it can be provided to the user once it is ready and then inserted and combined into the virtual environment.

[0070] In some embodiments, one or more cloud server computers are further configured to authenticate the user via login authentication credentials, the authentication credentials comprising a personal identification number (PIN), or a username and password, or a combination thereof.

[0071] In some embodiments, authentication is triggered in response to activation of an invitation link or deep link sent from one client device to another client device. In an embodiment, clicking on the invitation link or deep link triggers at least one cloud server computer to request the user to authenticate. For example, the invitation link or deep link can be used for a phone call, a meeting, or a video game session invitation, where the invited user can be authenticated by the user graphical representation based authentication system of the present disclosure.

[0072] In another embodiment, facial scanning uses 3D authentication, including guiding the user to perform a head movement pattern and extracting 3D facial data based on the head movement pattern. This can be done using application instructions stored in at least one server computer, which implements 3D authentication by guiding the user to perform a head movement pattern, such as performing one or more head gestures, tilting or rotating the head horizontally or vertically in a circular motion, performing a user-generated gesture pattern, or a specific head movement pattern, or a combination thereof. 3D authentication recognizes other features based on data obtained from the camera live video data feed, rather than just comparing and analyzing one view or image. In this 3D authentication embodiment, the facial scanning process can recognize other features from the data, which can include facial data, including head movement patterns, facial volume, height, depth of facial features, facial scars, tattoos, eye color, facial skin parameters (e.g., skin color, wrinkles, pore structure, etc.), reflectance parameters, in addition to, for example, just the location of such features on the facial topology, which may be the case with other types of facial detection systems. Capturing such facial data can therefore increase the capture of a real face, which can be used to generate a realistic user graphical representation. Facial scanning using 3D authentication can be performed using a high-resolution 3D camera, a depth camera (e.g., LIDAR), a light field camera, etc. The facial scanning process and 3D authentication can use deep neural networks, convolutional neural networks, and other deep learning techniques to retrieve, process, and evaluate the user's authentication using facial data.

[0073] In another aspect of the present disclosure, a user authentication method based on a user graphical representation includes: providing a user database storing user data associated with a user account and one or more corresponding user graphical representations in the memory of one or more cloud server computers, and a facial scanning and authentication module connected to the user database; receiving a request to access a user account from a client device; performing a facial scan of a user of the client device through the facial scanning and authentication module by extracting facial feature data from camera data captured by at least one camera communicating with the client device; checking the extracted facial feature data for a match with a user graphical representation associated with the user account in the user database; if a matching user graphical representation is found in the checking step, providing the user with access to the user account; and if no matching user graphical representation is found in the checking step, generating a new user graphical representation from the camera data and a new user account stored in the user database, and providing access to the user account.

[0074] In an embodiment, the method includes: performing a facial scan of a user by a facial scanning and authentication module using images and / or media data received from at least one camera connected to at least one client device and one or more cloud server computers; checking a user database for a match with the user facial data associated with a user account; providing the user with a corresponding user graphical representation and access to the user account if the user account is available; and generating a new user graphical representation from the facial data and a new user account stored in the user database and access to the user account if the user account is not available.

[0075] In some embodiments, the user graphical representation is a 3D virtual cutout of the user constructed from a user-uploaded or third-party source photo, or a real-time 3D virtual cutout of the user comprising a real-time video stream of the user with the background removed, generated based on a real-time 2D or 3D live video stream data feed obtained from a camera, or a video with the background removed, or a video without the background removed. In other embodiments, the method includes animating a matching user graphical representation or a new user graphical representation, which may include applying a machine vision algorithm by a client device or at least one cloud server computer on the corresponding user graphical representation to recognize the user's facial expressions and graphically simulate the facial expressions on the user graphical representation. In an embodiment, the updating of the user 3D virtual cutout includes applying a machine vision algorithm by a client device or at least one cloud server computer on the generated user 3D virtual cutout to recognize the user's facial expressions and graphically simulate the facial expressions on the user 3D virtual cutout.

[0076] In some embodiments, the method further comprises: if a matching user graphical representation is found in the checking step, checking the date of the matching user graphical representation; determining whether the matching user graphical representation needs to be updated based at least in part on the date; and generating a user graphical representation update request in the affirmative case that the matching user graphical representation needs to be updated. In an embodiment, the method comprises: if the user account is available, checking the date of the available user graphical representation; determining whether the existing user graphical representation needs to be updated by comparing with a corresponding threshold or security requirement; and, in the affirmative case that the user graphical representation is needed, generating and transmitting a user graphical representation update request to a corresponding client device. If the user approves the request, one or more cloud server computers or client devices continue to generate the user graphical representation based on the live camera feed. If no update is required, one or more cloud server computers continue to retrieve the existing user graphical representation from the user database.

[0077] In some embodiments, the method further includes inserting the user graphical representation into the two-dimensional or three-dimensional virtual environment, or on a third-party source linked to the virtual environment (e.g., by overlaying on the screen of a third-party application or website that is integrated or coupled with the disclosed system), and combining the user graphical representation with the two-dimensional or three-dimensional virtual environment graphics.

[0078] In some embodiments, the generation of a new user graphical representation occurs asynchronously with the user accessing the user account.

[0079] In some embodiments, the method further comprises authenticating the user by logging in authentication credentials comprising at least a username and a password.

[0080] In some embodiments, the authentication is triggered in response to activation of an invitation link. In an embodiment, the method further comprises providing an invitation link or a deep link by one client device to another client device, wherein clicking on the invitation link triggers at least one cloud server computer to request the user to authenticate.

[0081] In another aspect of the present disclosure, a system for virtually broadcasting from within a virtual environment is provided. The system comprises: a server computer system comprising one or more server computers, each server computer comprising at least one processor and memory, the server computer system comprising data and instructions implementing a data exchange management module configured to manage data exchange between client devices; and at least one virtual environment comprising a virtual broadcast camera, the virtual broadcast camera being positioned within at least one virtual environment and configured to capture multimedia streams from within at least one virtual environment. The server computer system is configured to receive live feed data captured by at least one camera from at least one client device, and broadcast the multimedia stream to at least one client device based on the data exchange management, wherein the broadcasted multimedia stream is configured to be displayed to a corresponding user graphical representation generated from a user live data feed from at least one client device. The data exchange management between client devices performed by the data exchange management module comprises analyzing incoming multimedia streams, and evaluating and forwarding outbound multimedia streams based on the analysis of the incoming media streams.

[0082] In an embodiment, the multimedia stream is sent to at least one media server computer for broadcasting to at least one client device. In an embodiment, the system includes at least one camera that obtains live feed data from a user of the at least one client device and sends the live feed data from the user to the at least one media computer via the at least one client device; wherein based on data exchange management from the at least one media server computer, the multimedia stream is broadcast to the at least one client device and displayed to a corresponding user graphical representation generated from the user live data feed by the at least one client device, and wherein the data exchange management between the client devices performed by the data exchange management module includes analyzing and optimizing incoming multimedia streams, and evaluating and optimizing forwarding of outbound multimedia streams.

[0083] In some embodiments, when forwarding outbound multimedia streams, the server computer system utilizes a routing topology that includes a selective forwarding unit (SFU), traversal of relay NAT (TURN), SAMS, or other suitable multimedia server routing topologies, or a media processing and forwarding server topology, or other suitable server topologies.

[0084] In some embodiments, a server computer system uses a media processing topology to process outbound multimedia streams for viewing by a client device for a graphical representation of a user within at least one virtual environment. In embodiments, when utilizing the media processing topology, at least one media server computer is configured to decode, combine, improve, mix, enhance, augment, compute, manipulate, and encode multimedia streams to an associated client device for viewing by a graphical representation of a user within at least one virtual environment through the client device.

[0085] In some embodiments, when utilizing a forwarding server topology, the server computer system utilizes one or more of an MCU, a cloud media mixer, and a cloud 3D renderer.

[0086] In some embodiments, the incoming multimedia stream includes user priority data and distance relationship data, and the user priority data includes a higher priority score for a user graphical representation closer to the incoming multimedia stream source and a lower priority score for a user graphical representation farther from the incoming multimedia stream source. In an embodiment, the multimedia stream includes data related to the distance relationship between the user priority and the corresponding user graphical representation and the multimedia stream, including metadata, or priority data, or data category, or spatial structure data, or three-dimensional position, or orientation or motion information, or image data, or media data, and video data based on a scalable video codec, or a combination thereof. In other embodiments, the priority data includes a higher priority score for a user closer to the multimedia stream source, and a lower priority score for a user farther from the virtual multimedia stream source. In other embodiments, the forwarding of the outbound multimedia stream is based on the user priority data and the distance relationship data. In an embodiment, the forwarding of the outbound multimedia stream implemented by the media server based on the user priority and distance relationship data includes optimizing the bandwidth and computing resource utilization of one or more receiving client devices. In other embodiments, the forwarding of the outbound multimedia stream also includes modifying, enlarging or reducing the time, space, quality and / or color characteristics of the multimedia stream.

[0087] In some embodiments, the virtual broadcast camera is managed by a client device accessing the virtual environment. In an embodiment, the virtual broadcast camera is configured to manipulate a viewpoint of the camera updated in the virtual environment, broadcasting the updated viewpoint to at least one client device.

[0088] In some embodiments, at least one virtual environment includes a plurality of virtual broadcast cameras, each virtual broadcast camera providing a multimedia stream from a corresponding viewpoint within the at least one virtual environment. In embodiments, each virtual broadcast camera provides a multimedia stream from a corresponding viewpoint within the virtual environment, the multimedia streams being selectable by a user of at least one client device and alternating between each other to provide a corresponding viewpoint to a corresponding at least one user graphical representation.

[0089] In some embodiments, at least one virtual environment is hosted by at least one dedicated server computer connected to at least one media server computer via a network, or hosted in a peer-to-peer infrastructure and relayed through at least one media server computer.

[0090] In another aspect of the present disclosure, a method for virtually broadcasting from within a virtual environment includes: providing data and instructions in a memory of at least one media server implementing a client device data exchange management module that manages data exchange between client devices; capturing a multimedia stream by a virtual broadcast camera located within at least one virtual environment connected to at least one media server, sending the multimedia stream to at least one media server for broadcasting to at least one client device; obtaining live feed data from at least one client device (e.g., from at least one camera via at least one client device); performing data exchange management, including analyzing incoming multimedia streams from within at least one virtual environment and live feed data, and evaluating forwarding of the outbound multimedia stream; and broadcasting corresponding multimedia streams to client devices based on the data exchange management, wherein the multimedia streams are displayed to a user graphical representation of a user of at least one client device. In this context, this refers to what the user graphical representation can "see" based on their position in the virtual environment, which corresponds to what will be displayed to the user (via the client device) when viewing the virtual environment from the perspective of his or her user graphical representation.

[0091] In some embodiments, when forwarding outbound multimedia streams, the method utilizes a routing topology including SFU, TURN, SAMS, or other suitable multimedia server routing topology, or a media processing and forwarding server topology, or other suitable server topology.

[0092] In some embodiments, the method further comprises decoding, combining, improving, mixing, enhancing, augmenting, calculating, manipulating, and encoding multimedia streams while utilizing the media processing topology.

[0093] In some embodiments, when utilizing a forwarding server topology, the method further includes utilizing one or more of a multipoint control unit (MCU), a cloud media mixer, and a cloud 3D renderer.

[0094] In some embodiments, the incoming multimedia stream includes user priority data and distance relationship data, and the user priority data includes a higher priority score for user graphical representations that are closer to the source of the incoming multimedia stream and a lower priority score for user graphical representations that are farther away from the source of the incoming multimedia stream. In an embodiment, the method further includes optimizing the forwarding of the outbound multimedia stream implemented by the media server based on the user priority and distance relationship data, which can include optimizing the bandwidth and computing resource utilization of one or more receiving client devices. In other embodiments, optimizing the forwarding of the outbound multimedia stream implemented by the media server also includes modifying, amplifying or reducing the time, space, quality and / or color characteristics of the multimedia stream.

[0095] In some embodiments, at least one virtual environment includes a plurality of virtual broadcast cameras, each virtual broadcast camera providing a multimedia stream from a corresponding viewpoint within the at least one virtual environment. In embodiments, the method further includes providing a plurality of virtual broadcast cameras, each virtual broadcast camera providing a multimedia stream from a corresponding viewpoint within the virtual environment, the multimedia streams being selectable by a user of the at least one client device and alternating between each other, providing a corresponding viewpoint to a corresponding at least one user graphical representation.

[0096] In another aspect of the present disclosure, a system for delivering applications within a virtual environment is provided, comprising: at least one cloud server computer, the cloud server computer including at least one processor and memory, the memory including data and instructions implementing at least one virtual environment linked to an application module, the application module including one or more installed applications and corresponding application rules for multi-user interaction; wherein, in response to selection by a virtual environment host through a client device, the one or more installed applications are displayed and activated during a virtual environment session, so that a user graphical representation of the virtual environment host and any participant user graphical representation within the virtual environment can interact with the one or more installed applications through the corresponding client device, and wherein the at least one cloud server computer manages and processes received user interactions with the one or more installed applications according to the application rules for multi-user interaction in the application module, and forwards the processed interactions accordingly (e.g., to each client device) to establish a multi-user session for implementing a shared experience according to the multi-user interaction application rules.

[0097] In some embodiments, application rules for multi-user interaction are stored and managed in one or more independent application servers.

[0098] In some embodiments, one or more applications are installed from an application installation package available in an application library, and application services are provided through corresponding application programming interfaces.

[0099] In some embodiments, the application library is context filtered. In an embodiment, the context filter is designed to provide relevant applications for a specific context.

[0100] In some embodiments, one or more installed applications are shared with and viewed through a virtual display application installed in a corresponding client device. In an embodiment, after installation and activation, one or more installed applications are shared with and viewed through a virtual display application installed in a corresponding client device, wherein the virtual display application is configured to receive one or more installed applications from an application library and publish one or more selected applications to be displayed to a conference host user graphical representation and other participant user graphical representations in a virtual environment through their corresponding client devices. In other embodiments, an application module is represented as a 2D screen or 3D body application module graphical representation that displays content from an installed application to a user graphical representation in a virtual environment, and wherein the virtual display application is represented as a 2D screen or 3D body that displays content from an installed application to a user graphical representation in a virtual environment.

[0101] In some embodiments, one or more applications are installed directly within the virtual environment prior to or concurrently with a multi-user session.

[0102] In some embodiments, prior to starting a multi-user session, one or more applications are installed using a virtual environment setup tool.

[0103] In some embodiments, one or more of the application rules for multi-user interaction define synchronous interaction, or asynchronous interaction, or a combination thereof. In an embodiment, such rules are used to update the corresponding update views of user interactions and one or more applications accordingly.

[0104] In some embodiments, the asynchronous interaction is implemented through at least one server computer or through a separate server computer dedicated to handling a single user interaction with at least one installed application.

[0105] In some embodiments, the virtual environment is a classroom, or an office space, or a conference room, or a meeting room, or an auditorium, or a theater.

[0106] In another aspect of the present disclosure, a method for delivering applications within a virtual environment is provided, comprising: providing at least one virtual environment in the memory of at least one cloud server computer, and an application module including one or more installed applications and corresponding application rules for multi-user interaction, wherein the application module is linked to and visible within the virtual environment; receiving a selection instruction from a virtual environment host; displaying and activating the one or more installed applications during a session of the virtual environment, so that a user graphical representation of the virtual environment host and one or more participant user graphical representations within the virtual environment can interact with the one or more installed applications through corresponding client devices; managing and processing user interactions with the one or more installed applications according to the application rules for multi-user interaction in the application module; and forwarding the processed interactions to the client device to establish a multi-user session for a shared experience according to the application rules.

[0107] In some embodiments, the method further comprises storing and managing application rules for multi-user interaction in one or more independent application servers.

[0108] In some embodiments, the method further includes installing one or more applications from application installation packages available in an application library; and providing application services through corresponding application programming interfaces. In other embodiments, the application library is context-filtered to provide relevant applications. In other embodiments, one or more installed applications are shared with and viewed through a virtual display application installed in a corresponding client device. In an embodiment, the method includes, upon activation, sharing with and viewing through a virtual display application installed in a corresponding client device, the virtual display application being configured to receive one or more installed applications from the application library and publish one or more selected applications for display to a conference host user graphical representation and other participant user graphical representations in a virtual environment through their corresponding client devices.

[0109] In some embodiments, the method further includes installing one or more applications directly in the virtual environment before or simultaneously with the multi-user session. In other embodiments, the method further includes installing one or more applications by using a virtual environment setting tool before starting the multi-user session.

[0110] In some embodiments, the method further comprises defining one or more of the application rules for multi-user interaction to include synchronous interaction, or asynchronous interaction, or a combination thereof. In an embodiment, the method further comprises updating the corresponding updated views of the user interaction and one or more applications accordingly.

[0111] In another aspect of the present disclosure, a system for providing virtual computing resources within a virtual environment includes a server computer system, the server computer system includes one or more server computers including at least one cloud server computer, the at least one cloud server computer includes at least one processor and a memory including data and instructions for implementing at least one virtual environment, and at least one virtual computer associated with at least one virtual environment, wherein the at least one virtual computer receives virtual computing resources from the server computer system. The association may include connecting the virtual computer to the virtual environment. In an embodiment, the at least one virtual computer has a corresponding graphical representation in the virtual environment. The graphical representation can provide other benefits, such as facilitating user interaction with the virtual computer and increasing the realism of the user experience (e.g., for a home office experience). Therefore, in an embodiment, at least one virtual computer includes at least one corresponding associated graphical representation located within the virtual environment, wherein the virtual computer receives virtual computing resources from at least one cloud server computer; and at least one client device connected to at least one server computer via a network; wherein, in response to at least one client device accessing one or more virtual computers (e.g., by interacting with the corresponding graphical representation), at least one cloud server computer provides at least a portion of the available virtual computing resources to at least one client device.

[0112] In some embodiments, the server computer system is configured to provide at least a portion of virtual computing resources to at least one client device in response to a user graphical representation interacting with at least one corresponding graphical representation of at least one virtual computer within at least one virtual environment. In other embodiments, one or more virtual computer graphical representations are spatially positioned within the virtual environment for access by the user graphical representation. In an embodiment, the arrangement of the virtual environment is associated with a contextual theme of the virtual environment, such as an arrangement of virtual items, furniture, floor plans, etc., for education, meetings, work, shopping, services, socializing, or entertainment, respectively. In other embodiments, one or more virtual computer graphical representations are positioned within the arrangement of the virtual environment for access by one or more user graphical representations. For example, a virtual computer can be positioned in a virtual room that a user graphical representation will access when participating in an activity that may require or benefit from the ability to use resources associated with a virtual computer, such as working on a project in a virtual classroom, laboratory, or office.

[0113] In some embodiments, a server computer system is configured to provide at least a portion of virtual computing resources to at least one client device in response to a user accessing at least one cloud server computer by logging into at least one client device without accessing a virtual environment. In an illustrative scenario, a user accessing at least one cloud server computer accesses the virtual computing resources by physically logging into a client device connected to the at least one cloud server computer via a network, triggering provision of the virtual computing resources to the client device without accessing the virtual environment.

[0114] In some embodiments, at least a portion of the virtual computing resources are assigned to the client device via a management tool. In other embodiments, the provision of at least a portion of the virtual computing resources is performed based on a stored user profile. In an embodiment, the assignment of resources is performed based on a stored user profile, the user profile comprising one or more parameters associated with and assigned to the user profile, the parameters comprising priority data, security data, QOS, bandwidth, storage space, or computing power, or a combination thereof.

[0115] In some embodiments, at least one virtual computer includes a downloadable application available to an application library. In an illustrative scenario involving multiple virtual computers, each virtual computer is a downloadable application available to an application library.

[0116] In another aspect of the present disclosure, a method for providing virtual computing resources within a virtual environment includes: providing at least one virtual computer in a memory of at least one cloud server computer, and a virtual environment associated with the at least one virtual computer; associating virtual computing resources with the at least one virtual computer; receiving an access request to access one or more virtual computers from at least one client device; and providing a portion of available virtual computing resources associated with the at least one virtual computer to the at least one client device in response to the access request received from the at least one client device. In an embodiment, the association of the virtual computing resources with the at least one virtual computer may include receiving the virtual computing resources by the virtual computer from the at least one cloud server computer.

[0117] In some embodiments, the access request includes a request to allow a user graphical representation to interact with one or more graphical representations representing at least one virtual computer. In an embodiment, the method further includes: receiving an access request from the user graphical representation to access one or more graphical representations of virtual computers within the at least one virtual environment; and providing at least a portion of the available virtual computing resources to the corresponding client device. In other embodiments, the arrangement of the virtual environment is associated with a contextual theme of the virtual environment, including an arrangement for education, conferences, work, shopping, services, socializing, or entertainment, respectively, and wherein one or more virtual computers are positioned within the arrangement of the virtual environment for access by one or more user graphical representations.

[0118] In some embodiments, the access request is triggered by a user logged into at least one client device. In an embodiment, the method further comprises: receiving an access request from a user physically logged into a client device connected to at least one cloud server computer via a network; and providing the virtual computing resources to the client device without accessing the virtual environment.

[0119] In some embodiments, the method further comprises assigning at least a portion of the virtual computing resources to the client device via a management tool. In other embodiments, the assignment is performed based on a stored user profile, the user profile comprising one or more parameters associated with and assigned to the user profile, the parameters comprising priority data, security data, QOS, bandwidth, storage space, computing power, or a combination thereof.

[0120] In another aspect of the present disclosure, a system for implementing ad hoc virtual communication between user graphical representations includes one or more cloud server computers, which include at least one processor and a memory storing data and instructions for implementing a virtual environment. The virtual environment is configured to enable at least one approaching user graphical representation and at least one target user graphical representation in the virtual environment to open an ad hoc communication channel, and to enable ad hoc conversations via the ad hoc communication channel between user graphical representations within the virtual environment. In an embodiment, the system further includes two or more client devices that access at least one virtual environment through corresponding user graphical representations and are connected to one or more cloud server computers via a network; wherein the virtual environment enables at least one approaching user graphical representation and at least one target user graphical representation to open an ad hoc communication channel, thereby enabling ad hoc conversations between user graphical representations within the virtual environment.

[0121] In some embodiments, the opening of the ad hoc communication channel is performed based on the distance, position and orientation between the user graphical representations, or the current availability status, privacy settings, or state configuration of the ad hoc communication, or a combination thereof.

[0122] In some embodiments, the self-organizing dialogue is performed at the location of the two user graphical representation areas in the virtual environment. In other embodiments, the self-organizing dialogue is performed using the current perspective in the virtual environment.

[0123] In some embodiments, the ad hoc conversation enables optional changes in perspective, position, or a combination thereof within the same or another connected virtual environment in which the ad hoc conversation occurs.

[0124] In some embodiments, one or more cloud server computers are further configured to generate visual feedback in the virtual environment, signaling that ad hoc communication is possible. In an embodiment, the user graphical representation receives visual feedback, which signals that ad hoc communication is possible, thereby triggering the opening of an ad hoc communication channel, signaling the start of an ad hoc conversation between the user graphical representations.

[0125] In some embodiments, the ad hoc conversation includes sending and receiving real-time audio and video. In an illustrative scenario, such video can be displayed from a user graphical representation.

[0126] In some embodiments, the user corresponding to the approaching user graphical representation selects and clicks the target user graphical representation before opening the ad hoc communication channel. In other embodiments, one or more cloud server computers are also configured to open the ad hoc communication channel in response to the accepted invitation. For example, the user corresponding to the approaching user graphical representation sends an ad hoc communication participation invitation to the target user graphical representation, and opens the ad hoc communication channel after receiving the invitation approval from the target user graphical representation.

[0127] In some embodiments, the ad hoc communication channel is implemented through at least one cloud server computer or as a P2P communication channel.

[0128] In another aspect of the present disclosure, a method for implementing self-organizing virtual communication between user graphical representations includes: providing a virtual environment in the memory of one or more cloud server computers including at least one processor; detecting two or more client devices accessing at least one virtual environment through corresponding graphical representations, wherein the client devices are connected to the one or more cloud server computers via a network; and opening a self-organizing communication channel in response to at least one user graphical representation approaching another user graphical representation, thereby implementing a self-organizing conversation between the user graphical representations in the virtual environment.

[0129] In some embodiments, the method further includes detecting and evaluating one or more of a distance, a position and an orientation between user graphical representations, or a current availability status, a privacy setting, or a state configuration of the ad hoc communication, or a combination thereof, before opening an ad hoc communication channel.

[0130] In some embodiments, the method enables the self-organizing dialogue to be performed in the virtual environment where the two user graphical representation areas are located. In other embodiments, the self-organizing dialogue is performed using the current perspective in the virtual environment.

[0131] In some embodiments, the method includes optionally changing perspective, position, or a combination thereof within the same or another connected virtual environment where the ad hoc conversation occurs.

[0132] In some embodiments, the method further comprises generating visual feedback in the virtual environment, signaling that ad hoc communication is possible. The method may also include sending visual feedback to the target user graphical representation, signaling that ad hoc communication is possible, thereby triggering the opening of the ad hoc communication channel, signaling the start of a conversation between the user graphical representations.

[0133] In some embodiments, the conversation involves sending and receiving real-time audio and video displayed from the user's graphical representation.

[0134] In some embodiments, the method further comprises selecting and clicking a target user graphical representation by a user graphical representation approaching from the user graphical representation. In other embodiments, the ad hoc communication channel is opened in response to an accepted invitation. In an embodiment, the method further comprises sending to another user graphical representation or receiving an ad hoc virtual communication participation invitation from another user graphical representation before opening the ad hoc communication channel.

[0135] Also described are computer readable media having stored thereon instructions configured to cause one or more computers to perform any of the methods described herein.

[0136] The above summary of the invention does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those systems and methods disclosed in the following detailed description and specifically pointed out in the claims filed with the present application. Such combinations have advantages not specifically described in the above summary of the invention. Other features and advantages will be apparent from the accompanying drawings and from the detailed description below. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] The specific features, aspects and advantages of the present disclosure will be better understood with reference to the following description and accompanying drawings, in which:

[0138] Figure 1 Depicted is a schematic diagram of a system enabling interactions including social interactions in a virtual environment according to an embodiment.

[0139] Figure 2A-2BA schematic diagram depicting deployment of a system for enabling interaction, including social interaction, in a virtual environment across multiple verticals including a virtual environment platform.

[0140] Figure 3 Depicted is a schematic diagram of a hybrid system architecture employed in a system for enabling interaction in a virtual environment, according to an embodiment.

[0141] Figure 4 Depicted is a schematic diagram of a graphical user interface by which a user may interact in a virtual environment, according to an embodiment.

[0142] Figure 5 Depicted is a block diagram of a method of transitioning from user 3D virtual cutout to user real-time 3D virtual cutout, or video with background removed, or video without background removed, according to an embodiment.

[0143] Figures 6A-6C Depicted is a schematic diagram of multiple image processing combinations performed on the client-server side by corresponding client devices and a cloud server.

[0144] Figures 7A-7C A schematic diagram depicting multiple image processing combinations performed by corresponding peer clients on the P2P side.

[0145] Figure 8 Depicted is a schematic diagram of a user authentication system based on a graphical representation of a user, according to an embodiment.

[0146] Fig. 9 A schematic diagram depicting a third-person perspective of an office virtual environment according to an embodiment is depicted.

[0147] Figures 10A-10B Depicted is a schematic diagram of a classroom virtual environment in accordance with an embodiment.

[0148] Fig.11 Depicted is a schematic diagram of multiple virtual camera positions in accordance with an embodiment.

[0149] Fig.12 A schematic diagram of a system for virtually broadcasting from within a virtual environment is depicted.

[0150] Fig.13 A schematic diagram of a system for delivering applications within a virtual environment is depicted.

[0151] Fig.14 Describes a method based on a method for Fig.13 A schematic diagram of a virtual environment of a system for delivering applications within a virtual environment is depicted in FIG.

[0152] Fig.15 Depicted is a schematic diagram of a system for providing virtual computing resources within a virtual environment according to an embodiment.

[0153] Fig.16 Depicted is a schematic diagram of a system enabling ad hoc virtual communications between graphical representations of users, according to an embodiment.

[0154] Fig.17 Embodiments of methods of enabling interaction in a virtual environment according to embodiments are depicted.

[0155] Fig.18 Embodiments of image processing methods according to embodiments are described.

[0156] Fig.19 Depicted is a method 1900 of user authentication based on a graphical representation of a user, according to an embodiment.

[0157] Fig. 20 A block diagram of a method for virtually broadcasting from within a virtual environment is presented in accordance with an embodiment.

[0158] Fig.21 A block diagram of a method for delivering an application from within a virtual environment according to an embodiment is presented.

[0159] Fig. 22 A block diagram of a method for providing virtual computing resources within a virtual environment according to an embodiment is presented.

[0160] Fig.23 A block diagram of a method for implementing self-organizing virtual communication between graphical representations of users is presented. DETAILED DESCRIPTION

[0161] In the following description, reference is made to the accompanying drawings which illustrate various embodiments by way of illustration. Moreover, various embodiments will be described below by reference to several examples. It should be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0162] The system and method of the present disclosure solves at least some of the aforementioned shortcomings by providing a virtual environment platform including one or more virtual environments, which can achieve real-time multi-user collaboration and interaction similar to that available in real life, and can be used for meetings, work, education, shopping, and services, etc. The virtual environment can be selected from multiple virtual environments of different vertical fields available on the virtual environment platform. The combination of virtual environments from the same and / or different vertical fields can form a virtual environment cluster, which can include hundreds or even thousands of virtual environments. The virtual environment can be a 2D or 3D virtual environment, including an arrangement and visual appearance associated with the vertical field of the virtual environment, which can be customized by users according to their preferences or needs. Users can access the virtual environment through a graphical representation, which can be inserted into the virtual environment and graphically combined with a two-dimensional or three-dimensional virtual environment.

[0163] The user graphic representation can be a 3D virtual cutout of the user constructed from a photo with the background removed, uploaded by the user or from a third-party source, or a real-time 3D virtual cutout of the user, or a video with the background removed, or a video without the background removed, any of which can be switched to each other at any time according to the needs of the user. The user graphic representation can include a user status that provides further details about the current availability or other data related to other users. Interactions such as conversations and collaborations between users in the virtual environment and interactions with objects within the virtual environment are implemented. The present disclosure also provides a data processing system and method, including multiple image processing combinations that can be used to generate user graphic representations. The present disclosure also provides: a user authentication system and method based on user graphic representations, which can be used to access a virtual environment platform or link to other applications in the virtual environment from the virtual environment platform; a system and method for virtual broadcasting from within a virtual environment; a system and method for delivering applications within a virtual environment; a system and method for providing virtual computing resources based on cloud computing within a virtual environment cloud server computer; and a system and method that can achieve self-organizing virtual communication between close user graphic representations.

[0164] Enabling virtual presence in the virtual environment and real interaction and collaboration between users can increase the realism of remote activities, such as required in epidemic or other situations that restrict mobility. The systems and methods of the present disclosure also enable access to various virtual environments on client devices such as mobile devices or computers without the need for more expensive immersive devices such as extended reality head-mounted displays or expensive new system infrastructure. The client or peer device of the present disclosure can include, for example, a computer, a headset, a mobile phone, glasses, a transparent screen, a tablet computer, and an input device that typically has a camera built in, or an input device that can be connected to a camera and receive a data feed from the camera.

[0165] Figure 1 Depicted is a schematic diagram of a system 100 for enabling social interaction in a virtual environment, according to an embodiment.

[0166] The disclosed system 100 for implementing interactions in a virtual environment includes one or more cloud server computers 102, the cloud server computers including at least one processor 104 and a memory 106 storing data and instructions for implementing a virtual environment platform 108, the virtual environment platform including at least one virtual environment 110, such as virtual environment AC. The one or more cloud server computers are configured to insert a user graphic representation generated from a live data feed obtained by a camera at a three-dimensional coordinate location of at least one virtual environment, update the user graphic representation in at least one virtual environment, and implement real-time multi-user collaboration and interaction in the virtual environment. In the described embodiment, inserting the user graphic representation into the virtual environment involves graphically combining the user graphic representation in the virtual environment so that the user graphic representation appears in the virtual environment (e.g., at a specified 3D coordinate location). Figure 1 In the example shown, the system 100 also includes at least one camera 112 that obtains a live data feed 114 from a user 116 of a client device 118. One or more client devices 118 are communicatively connected to one or more cloud server computers 102 and at least one camera 112 via a network. A user graphical representation 120 generated from the live data feed 114 is inserted into a three-dimensional coordinate location of a virtual environment 110 (e.g., virtual environment A) and combined with the virtual environment graphics, and updated using the live data feed 114. The updated virtual environment is provided to the client device through direct P2P communication or indirectly through the use of one or more cloud servers 102. The system 100 enables real-time multi-user collaboration and interaction in the virtual environment 110 by accessing a graphical user interface via the client device 118.

[0167] exist Figure 1 In , two users 116 (e.g., users A and B, respectively) are accessing a virtual environment A and are interacting with elements therein and with each other via their respective user graphical representations 120 (e.g., user graphical representations A and B, respectively) accessed via respective client devices 118 (e.g., client devices A and B, respectively). Figure 1 Only two users 116 , client devices 118 , and user graphical representations 120 are depicted in FIG. 1 , but those skilled in the art will appreciate that the system may enable multiple users 116 to interact with each other through their corresponding graphical representations 120 via corresponding client devices 118 .

[0168] In some embodiments, the virtual environment platform 108 and corresponding virtual environment 110 may enable real-time sharing of multiple experiences, such as live performances, concerts, webinars, keynote speeches, etc., to multiple (e.g., thousands or even millions) of user graphical representations 120. These virtual performances may be presented by and / or multicasted to multiple instances of the virtual environment 110 to accommodate a large number of users 116 from around the world.

[0169] In some embodiments, the client device 118 may be one or more of a mobile device, a personal computer, a game console, a media center, a head mounted display, etc. The camera 110 may be one or more of a 2D or 3D camera, a 360 degree camera, a web camera, an RGBD camera, a CCTV camera, a professional camera, a mobile phone camera, a depth camera (e.g., LIDAR), or a light field camera, etc.

[0170] In some embodiments, the virtual environment 110 refers to a virtual building (e.g., a virtual model) designed by a computer-aided drafting (CAD) method through any suitable 3D modeling technology. In other embodiments, the virtual environment 110 refers to a virtual building scanned from a real building (e.g., a physical room) by any suitable scanning tool, including an image scanning pipeline input by various photos, videos, depth measurements and / or simultaneous localization and mapping (SLAM) scans to generate the virtual environment 110. For example, radar imaging such as synthetic aperture radar, real aperture radar, light detection and ranging (LIDAR), inverse aperture radar, monopulse radar, and other types of imaging technologies can be used to map and model real-world structures and transform them into a virtual environment 110. In other embodiments, the virtual environment 110 is a virtual building after modeling a real building (e.g., a room, building, or facility in the real world).

[0171] In some embodiments, the client device 118 is connected to at least one cloud server computer 102 via a wired or wireless network. In some embodiments, the network may include a millimeter wave (mmW) or a combination of mmW and sub-6 GHz communication systems, such as fifth generation wireless system communication (5G). In other embodiments, the system may be connected via wireless LAN (Wi-Fi). In other embodiments, the system may be connected via fourth generation wireless system communication (4G), may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0172] In some embodiments, when the live data feed 114 of the user 116 is received, the processing and rendering involved in the generation, updating, and insertion of the user graphical representation 120 into the selected virtual environment 110, and combinations thereof, are performed by at least one processor of the client device 118. The one or more cloud server computers 102 may receive the client-rendered user graphical representation 120, insert the client-rendered user graphical representation 120 into the three-dimensional coordinates of the virtual environment 110, combine the inserted user graphical representation 120 with the virtual environment 110, and then continue to transmit the client-rendered user graphical representation 120 to the receiving client device. For example, Figure 1 As shown, client device A may receive a live data feed 114 from a corresponding camera 112, may process and render data from the live data feed 114, generate a user graphical representation A, and may then transmit the client-rendered user graphical representation A to at least one cloud server computer 102, which may position user graphical representation A in three-dimensional coordinates of a virtual environment 118, and then transmit user graphical representation A to client device B. A similar process applies to client device B and user graphical representation B from user B. User graphical representations A and B may thus view and interact with each other in virtual environment A. However, various other image processing combinations may be implemented by the systems and methods of the present disclosure, such as reference 102. Figures 6A-7C As shown and described.

[0173] In some embodiments, the processing and rendering involved in the generation, updating, and insertion of the user graphical representation 120 and the combination with the virtual environment is performed by at least one processor 104 of one or more cloud server computers 102 when the client device 118 sends the unprocessed live data feed 114 of the user 116. The one or more cloud server computers 102 thus receive the unprocessed live data feed 114 of the user 116 from the client device 118, and then generate, process, and render the user graphical representation 120 located within the three-dimensional coordinates of the virtual environment 110 from the unprocessed live data feed, and then transmit the cloud-rendered user graphical representation within the virtual environment to other client devices 118. For example, as Figure 1 As shown, client device A can receive live data feed 114 from corresponding camera 112, and can then transmit unprocessed user live data feed 114 to at least one cloud server computer 102, which can generate, process and render user graphical representation A and position user graphical representation A in three-dimensional coordinates of virtual environment 118, and then transmit user graphical representation A to client device B. A similar process applies to client device B and user graphical representation B from user B. User graphical representations A and B can thus view and interact with each other in virtual environment A.

[0174] In some embodiments, the virtual environment platform 108 is configured to embed a clickable link redirecting to a virtual environment into one or more third-party sources including third-party websites, applications, or video games. The link can be, for example, an HTML link. The linked virtual environment 110 can be associated with the content of the website in which the link is embedded. For example, the link can be embedded on a car dealer or manufacturer website, where the clickable link redirects to a virtual environment 110 representing a car dealer showroom that the user can visit through the user graphical representation 120.

[0175] In some embodiments, the user graphical representation 120 includes a clickable link embedded therein, such as a link to a third-party source containing profile information about the corresponding user. For example, the clickable link can be an HTML link embedded in the source code of the user graphical representation 120, which can authorize a link to a social media (e.g., a professional social media website such as LinkedIn). TM ) to provide other information about the corresponding user. In some embodiments, if the user allows, when other users click or hover the cursor on the corresponding user graphical representation, at least some basic information of the user can be displayed, which can be completed by accessing and retrieving user data from a database or third-party source.

[0176] In some embodiments, the user graphical representation is a 3D virtual cutout of the user constructed from a photo uploaded by the user or from a third-party source (e.g., from a social media website), or a real-time 3D virtual cutout of the user containing a real-time video stream of the user 116 with the background removed, or a video with the background removed, or a video without the background removed. In other embodiments, the client device 118 generates the user graphical representation 120 by processing and analyzing the live camera feed 114 of the user 116, generating animation data that is sent to other peer client devices 118 via a peer-to-peer (P2P) system architecture or a hybrid system architecture, as will be referred to in the accompanying drawings. Figure 3 As further described, the receiving peer client device 118 uses the animation data to locally construct and update the user graphical representation.

[0177] The user 3D virtual cut may include a virtual copy of the user constructed from a 2D photo uploaded by the user or from a third-party source. In an embodiment, the user 3D virtual cut is created via a 3D virtual reconstruction process using a 2D photo uploaded by the user or from a third-party source as input data, thereby generating a 3D mesh or 3D point cloud of the user with the background removed. In one embodiment, the user 3D virtual cut may have a static facial expression. In another embodiment, the user 3D virtual cut may include a facial expression updated by a camera feed. In another embodiment, the user 3D virtual cut may include an expression that can be changed by a button on a user graphical interface, such as a button that allows the user 3D virtual cut to smile, frown, be serious, etc. In yet another embodiment, the user 3D virtual cut uses a combination of the above techniques to display facial expressions. After the user 3D virtual cut is generated, the state and / or facial expression of the user 3D virtual cut can be continuously updated by, for example, processing a camera feed from the user. However, if the camera is not turned on, the user 3D virtual cut may still be visible to other users with an unavailable state and a static facial expression. For example, the user may currently be focused on a task and may not want to be disturbed (e.g., has a "do not disturb" or "busy" status), and therefore turns off his or her camera. At this point, the user 3D virtual cutout may simply be sitting at his or her desk, and may be stationary or may be performing a preconfigured movement, such as typing. However, when the user camera is turned on again, the user 3D virtual cutout may again be updated in real time with respect to the user's facial expressions and / or movements. Standard 3D facial model reconstruction (e.g., 3D facial fitting and texture fusion) techniques for creating the user 3D virtual cutout may be used so that the resulting user graphical representation is clearly recognizable as the user.

[0178] A real-time 3D virtual cutout of a user may include a virtual copy of a user based on a real-time 2D or 3D live video stream data feed obtained from a camera and after removing the user's background. In an embodiment, the real-time 3D virtual cutout of a user is created by generating a 3D mesh or 3D point cloud of the user with the background removed, using the user's live data feed as input data, and through a 3D virtual reconstruction process using machine vision technology. For example, the real-time 3D virtual cutout of a user may be generated from a 2D video from a camera (e.g., a webcam), which may be processed to create a holographic 3D mesh or 3D point cloud. In another example, the real-time 3D virtual cutout of a user may be generated from a 3D video from a depth camera (e.g., LIDAR or any depth camera), which may be processed to create a holographic 3D mesh or 3D point cloud. Therefore, the real-time 3D virtual cutout of a user graphically represents the user in three dimensions and in real time.

[0179] The video with the background removed may include a video streamed to a client device where a background removal process has been performed so that only the user is visible and then displayed using a polygonal structure on the receiving client device. The video without the background removed may include a video streamed to a client device where the video faithfully represents the camera capture so that the user and his or her background are visible and then displayed using a polygonal structure on the receiving client device. The polygonal structure may be a quadrilateral structure or a more complex 3D structure that serves as a virtual frame to support the video.

[0180] The video without the background removed may include a video streamed to a client device where the video faithfully represents the camera capture, making the user and his or her background visible, and then displayed using a polygonal structure on the receiving client device. The polygonal structure may be a quadrilateral structure or a more complex 3D structure that serves as a virtual frame supporting the video.

[0181] In some embodiments, data used as input data for inclusion in a live data feed and / or user uploaded or third party sourced 2D photos includes 2D or 3D image data, 3D geometry, video data, media data, audio data, text data, tactile data, temporal data, 3D entities, 3D dynamic objects, text data, temporal data, metadata, priority data, security data, position data, lighting data, depth data, infrared data, and the like.

[0182] In some embodiments, the background removal process required to achieve the user's real-time 3D virtual cutout is performed through the use of image segmentation and deep neural networks, which can be implemented by one or more processors of the client device 118 or at least one cloud server computer 102 implementing instructions. Image segmentation is the process of dividing a digital image into multiple objects, which can help locate objects and boundaries that can separate the foreground (e.g., the user's real-time 3D virtual cutout) from the background obtained from the live data feed 114 of the user 116. Sample image segmentation that can be used in embodiments of the present disclosure can include, for example, a watershed transform algorithm available from OpenCV.

[0183] Suitable image segmentation processes that can be used for background removal in the present disclosure use artificial intelligence (AI) techniques, such as computer vision, to achieve such background removal, and may include instance segmentation and / or semantic segmentation. Instance segmentation gives different labels to each individual instance of one or more multi-object classes. In some examples, instance segmentation is performed by a mask (Mask R-CNN), which detects objects in the image, such as from a user live data feed 114, while generating high-quality segmentation masks for each instance, and also adds a branch for predicting object masks, which is parallel to the existing branch for bounding box recognition. The segmentation masks created for the user and background are then extracted, and the background can be removed. Semantic segmentation uses deep learning or deep neural network (DNN) technology to achieve an automatic background removal process. Semantic segmentation divides an image into semantically meaningful parts according to predefined rules by giving each pixel a category label from one or more categories, such as color, texture, and smoothness. In some examples, semantic segmentation can utilize end-to-end, pixel-by-pixel semantic segmentation trained with a fully convolutional network (FCN), as disclosed in the document “Fully Convolutional Networks for Semantic Segmentation” by Evan Shelhamer, Jonathan Long, and Trevor Darrell, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 4 (April 2017), which is incorporated herein by reference. After the above-described background removal process, a point cloud within the boundaries of the user's face and body can be retained, and one or more processors of the client device 118 or at least one cloud server computer 102 can process the point cloud to generate a 3D mesh or 3D point cloud of the user, which can be used in the construction of a real-time 3D virtual cutout of the user. The real-time 3D virtual cutout of the user is then updated from the live data feed 114 from the camera 112.

[0184] In some embodiments, the updating of the user graphical representation 120 includes applying a machine vision algorithm on the generated user graphical representation 120 by the client device 118 or at least one cloud server computer 102 to recognize the facial expression of the user 116 and graphically simulate the facial expression on the user graphical representation 120 within the virtual environment 110. In general, such facial expression recognition can be performed through the principles of affective computing, which deals with the recognition, interpretation, processing, and simulation of human emotions. A review of conventional facial expression recognition (FER) techniques is provided by Daniel Canedo and António J. R. Neves in "Facial Expression Recognition Using Computer Vision: A Systematic Review," Applied Sciences, Vol. 9, No. 21 (2019), which is incorporated herein by reference.

[0185] Conventional FER techniques include steps of image acquisition, preprocessing, feature extraction, and classification or regression. In some embodiments of the present disclosure, image acquisition is performed by feeding image data from a camera feed 114 to one or more processors. The preprocessing step may be necessary to provide the most relevant data to the feature classifier, and typically includes face detection techniques that are capable of creating a bounding box that defines the target user's face, which is the desired region of interest (ROI). The ROI is preprocessed by intensity normalization for illumination changes, noise filters for image smoothing, data augmentation for increased training data, rotation correction for rotated faces, image resizing for different ROI sizes, and image cropping for better background filtering. After preprocessing, the algorithm retrieves relevant features from the preprocessed ROI, including action units (AUs), motion of certain facial landmarks, distances between facial landmarks, facial texture, gradient features, etc. These features can then be fed into a classifier, which can be, for example, a support vector machine (SVM) or a convolutional neural network (CNN). After training the classifier, emotions can be detected in real time in the user and constructed in the user graphical representation 120 by, for example, connecting all facial feature relationships.

[0186] In some embodiments, the user graphical representation is associated with a bird's-eye view, or a third-person view, or a first-person view, or a self-perspective. In an embodiment, the perspective of the user 116 when accessing the virtual environment through the user graphical representation is a bird's-eye view, or a third-person view, or a first-person view, or a self-perspective, or a broadcast camera view. The self-perspective may include a user graphical representation seen by another user graphical representation, and optionally, a virtual background of the user graphical representation.

[0187] In some embodiments, the perspective is updated as the user 116 manually navigates the virtual environment 110 via the graphical user interface.

[0188] In other embodiments, the perspective is automatically established and updated by using a virtual camera, wherein the perspective of the live data feed is associated with the perspective of the user graphical representation and the virtual camera, and wherein the virtual camera is automatically updated by tracking and analyzing the user's eye and head tilt data or head rotation data or a combination thereof. In an embodiment, the perspective is automatically established and updated by using one or more virtual cameras, which are virtually placed and aligned in front of the user graphical representation 120, for example, in front of a video with the background removed, or a video without the background removed, or a user 3D virtual cut, or a user real-time 3D virtual cut. In one embodiment, one or more virtual cameras can point outward from eye level. In another embodiment, two virtual cameras, one for each eye, can point outward from both eyes horizontally. In another embodiment, one or more virtual cameras can point outward from the center of the head position of the user graphical representation. The perspective of the user 116 captured by the camera 112 is associated with the perspective of the user graphical representation 120 and the associated virtual camera using computer vision, thereby manipulating the virtual camera.

[0189] The virtual camera provides a virtual representation of the perspective of the user graphical representation 120 associated with the perspective of the user 116, so that the user 116 can view the area of ​​the virtual environment 110 that the user graphical representation 120 may be looking at in one of many perspectives. The virtual camera is automatically updated by tracking and analyzing user eye and head tilt data, or head rotation data, or a combination thereof. The virtual camera position can also be manually changed by the user 116 according to the perspective selected by the user 116.

[0190] The self-perspective is the perspective of the user graphical representation 120 as seen by another user graphical representation 120 with the background removed (e.g., in a "selfie mode" of a phone camera). Alternatively, the self-perspective may include a virtual background of the user graphical representation 120 for understanding the perception of the user 116 as seen by other participants. When a virtual background of the user graphical representation is included, the self-perspective may be set to an area around the user graphical representation that may be captured by a virtual camera, which may result in a circle, square, rectangle, or any other suitable shape for constructing the self-perspective. For example, in a situation where the user graphical representation 120 is virtually located in a house, where, behind the user, there may be a window from which a tree may be seen, the self-perspective displays the user graphical representation and, alternatively, displays the background including the window and the tree.

[0191] In other embodiments, tracking and analysis of user eye and head tilt data, or head rotation data, or a combination thereof, includes using computer vision to capture and analyze viewing position and orientation captured by at least one camera 112, thereby manipulating the virtual camera in the virtual environment 110. For example, such manipulation may include receiving and processing eye and head tilt data captured by at least one camera through computer vision methods; extracting viewing position and orientation from the eye and head tilt data; identifying one or more coordinates of the virtual environment included in the position and orientation from the eye tilt data; and manipulating the virtual camera based on the identified coordinates.

[0192] In some embodiments, the instructions in the memory 106 of the at least one cloud server computer 102 are further capable of performing data analysis of user activities within the at least one virtual environment 110. The data analysis can be used for interactions, including participation in conversations with other users, interactions with objects within the virtual environment 110, purchases, downloads, engagement with content, etc. The data analysis can utilize a variety of known machine learning techniques to collect and analyze data from interactions in order to perform recommendations, optimization, predictions, and automation. For example, the data analysis can be used for marketing purposes.

[0193] In some embodiments, at least one processor 104 of one or more cloud server computers 102 is further configured to enable trading and monetization of content added in at least one virtual environment 110. At least one cloud server computer 102 can be communicatively connected to an application and object library where a user can find, select, and insert content in at least one virtual environment through a suitable application programming interface (API). One or more cloud server computers 102 can be further connected to one or more payment gateways capable of performing corresponding transactions. The content can include, for example, interactive applications or static or interactive 3D assets, animations, or 2D textures, etc.

[0194] Figure 2A-2B Schematic diagrams of deployments 200a and 200b of systems that enable interaction in a virtual environment that includes multiple verticals of a virtual environment platform are depicted.

[0195] Figure 2A A schematic diagram of a deployment 200a of a system for enabling interaction in a virtual environment comprising a plurality of vertical domains 202 of a virtual environment platform 108 is depicted in accordance with an embodiment. Figure 2A Some elements of Figure 1 The same or similar elements are shown in the drawings and therefore the same reference numerals may be used.

[0196] Verticals 202 are associated with contextual themes of virtual environments, including, for example, contextual themes related to use in meetings 204, such as virtual conference rooms, work 206 as, for example, virtual office spaces, learning 208 as, for example, virtual classrooms, and shopping 210 as, for example, virtual stores. Figure 2A Other verticals not shown may include, for example, services such as banking, reservations (e.g., hotels, travel agencies, or restaurants) and government agency services (e.g., consulting for opening a new business for a fee); as well as entertainment (e.g., karaoke, event halls or arenas, theaters, nightclubs, stadiums, museums, cruise ships, etc.), etc.

[0197] Each virtual environment vertical domain 202 may include multiple available virtual environments 110 (e.g., virtual environment AL), each virtual environment having one or more available arrangements and visual appearances associated with the context of the corresponding vertical domain 202. For example, the virtual environment A of the conference 204 vertical domain 202 may include a conference table with seats, a whiteboard, and a projector. At least one cloud server computer may provide corresponding resources (e.g., storage, network, and computing power) to each virtual environment 110. The vertical domain 202 may be obtained from the virtual environment platform 108, which may be accessed by one or more users 116 via a client device 118 through a graphical user interface 212. The graphical user interface 212 may be included in a downloadable client application or a web browser application, providing application data and instructions required to execute the selected virtual environment 110, and implementing multiple interactions therein. In addition, each virtual environment 110 may include one or more human or artificial intelligence (AI) hosts or assistants, which may provide the required data and / or services to assist users within the virtual environment through their corresponding user graphical representations. For example, human or AI banking staff can assist users of a virtual bank by providing the required information in the form of presentations, tables, lists, etc. as requested by the user.

[0198] In some embodiments, each virtual environment 110 is a persistent virtual environment that records the customized changes performed thereon, wherein the changes are stored in the persistent memory of at least one cloud server computer 102. For example, returning to the example of virtual environment A, the seating arrangement around the table, the color of the walls, or even the size and capacity of the room can be modified to suit the needs or preferences of the user. The performed changes can be saved in the persistent memory and thereafter available during subsequent sessions in the same virtual environment A. In some examples, implementing the persistent storage of modifications in the virtual environment 110 may require payment of a subscription fee to the room host or owner (e.g., via the virtual environment platform 108 connected to a payment gateway).

[0199] In other embodiments, the virtual environment 110 is a temporary virtual environment stored in a temporary memory of at least one cloud server computer 102. In these embodiments, changes performed on the virtual environment 110 may not be stored and, therefore, may not be available in future sessions. For example, the temporary virtual environment may be selected from the virtual environment platform 108 from predefined available virtual environments from different verticals 202. Changes in arrangement (such as decorations or modifications) may or may not be implemented, but in the event that changes are implemented, these changes may be lost after ending the session.

[0200] In some embodiments, a complex of virtual environments 110 within one or more vertical domains 202 can represent a virtual environment cluster 214. For example, some virtual environment clusters 214 can contain hundreds or even thousands of virtual environments 110. For users, virtual environment clusters 214 can appear to be part of the same system, where users can interact with each other or can seamlessly access other virtual environments within the same virtual environment cluster 214. For example, virtual environments D and E from the work virtual environment vertical domain 206 plus virtual environment B from the conference virtual environment vertical domain 204 can form a virtual environment cluster 214 representing a company. The user in this example can have two different workspaces, for example, a game development room and a business development room, and a conference room for video conferencing. Users from either the game development room and the business development room can meet in the conference room and conduct private virtual meetings, while the rest of the staff can continue to perform their current activities in their original workspaces.

[0201] In other examples, the virtual environment cluster 214 can represent a theater or event facility, where each virtual environment 110 represents an indoor or outdoor event area (e.g., an auditorium or event venue) where one or more performers are performing live performances through their corresponding user graphic representations. For example, an orchestra and / or singers can perform a concert by recording their performances live via a camera and through their user graphic representations (e.g., through their user live 3D virtual cutouts). The user graphic representation of each performer can be inserted into the corresponding three-dimensional coordinates of the stage where they can perform. The audience can watch the performance from the auditorium through their corresponding user graphic representations, and can achieve multiple interactions, such as virtual clapping, singing along to the song, virtual dancing, virtual jumping or cheering, etc.

[0202] In other examples, virtual environment cluster 214 can represent a casino comprising multiple game areas (e.g., 21-point area, poker area, roulette area and slot machine area), token purchase area and activity room. The machine in each game area can be configured as a casino application, which is configured to provide the user experience relevant to each game. Casino operators can include corresponding user graphics 120 (such as s) or user real-time 3D virtual shearing. The casino operator represented by s can be a real human operator or an artificial intelligence program that assists users in virtual casinos. Each casino game can be coupled to a payment gateway from the casino company operating a virtual casino, so that it is possible to pay to users and pay from users.

[0203] In other examples, the virtual environment cluster 214 can represent a shopping mall containing multiple floors, each floor containing multiple virtual environments, such as stores, showrooms, public areas, food courts, etc. Each virtual room can be managed by a corresponding virtual room administrator. For example, each store can be managed by a corresponding store administrator. Sales staff can be available in each area as a 3D live virtual avatar or a user's real-time 3D virtual cutout, and can be a real person or an AI assistant. In the current example, each virtual store and restaurant in the sample food court can be configured to enable online purchase of goods and delivery of goods to the user's address through a corresponding payment gateway and delivery system.

[0204] In another example, virtual environment cluster 214 includes multiple virtual gathering areas of virtual nightclubs, where users can meet and socialize there through their corresponding user graphical representations. For example, each virtual gathering area can include different themes and associated music and / or decorations. In addition to talking and texting, some other interactions in the virtual nightclub can include (for example) virtual dancing or drinking, sitting down in different lounge areas (for example, lounges or bars), etc. In addition, in this example, indoor concerts can be held in virtual nightclubs. For example, electronic concerts can be played by radio show hosts (DJs) performing behind virtual tables on the stage, where DJs can be represented by 3D live virtual avatars or user real-time 3D virtual cuts. If DJs are represented by user real-time 3D virtual cuts, the real-time movement of the DJ playing the audio mixing console can be projected onto the real-time 3D virtual cuts from the live data feed obtained by the camera capturing images from the DJ at the DJ position (for example, from the DJ's house or recording studio). Additionally, each member of the audience may also be represented by their own user graphical representation, with some users being represented by 3D live virtual avatars and other users being represented by user live 3D virtual cutouts depending on user preferences.

[0205] In other examples, virtual environment cluster 214 can represent a virtual karaoke entertainment facility comprising multiple private or public karaoke rooms. Each private karaoke room can include a virtual karaoke machine, a virtual screen, a stage, a microphone, a speaker, decorations, a sofa, a table, and beverages and / or food. The user can select a song through the virtual karaoke machine, which can be connected to a song database, triggering the system to play the song for the user, and projecting the lyrics onto the virtual screen for the user to sing together through their user graphical representations. The public karaoke room can also include a human or AIDJ, which selects songs for the user, calls the user into the stage, and mutes or unmutes the user to listen to the performance when necessary. The user can sing remotely from their client device through a microphone.

[0206] In other examples, the virtual environment cluster 214 can represent a virtual cruise ship containing multiple areas, such as bedrooms, cabins, activity rooms, bows, sterns, port, starboard, bridges, and multiple decks. Some areas may have human or AI assistants to take care of users through corresponding user graphical representations, such as providing other information or services. If available, a virtual environment or simple graphical representation of the exterior of the cruise ship may be available, such as a landscape depicting an island, town, or city that may be visited when arriving at a particular destination. Users can therefore experience traveling on the high seas and discovering new places through their user graphical representations while being able to interact with each other virtually.

[0207] In other examples, the virtual environment cluster 214 may represent an e-sports stadium or gymnasium containing multiple virtual environments representing a stadium, court, or room where users can play games via their user graphical representations through suitable input / output devices (e.g., computer keyboards, game controllers, etc.). The mechanics of each e-sports may depend on the sport to be played. The e-sports stadium or gymnasium may include a public area where users can select sports areas to visit. Available sports schedules may also be available, informing users which sports activities are available at which times.

[0208] Figure 2B Deployment 200b shows a virtual school 216 that combines multiple virtual environments from various verticals 202. Virtual school 216 includes four classrooms (e.g., classrooms A-D 218-224), an auditorium 226, a sports area 228, a cafeteria 230, a teacher's lounge 232, a library 234, and a bookstore 236. Each virtual environment may include virtual objects associated with the corresponding environment represented by corresponding graphical representations.

[0209] For example, a virtual classroom (e.g., any one of virtual classrooms AD 218-224) can enable students to attend classes, and can be configured to enable students to participate in classes through various interactions (e.g., raising hands, content projection, demonstration, expressing questions or contributions verbally or through text, etc.), and can provide special administrative permissions to teachers (e.g., letting someone speak, muting one or more students during a lecture, sharing content through a digital whiteboard, etc.). The auditorium can enable speakers to give speeches, or multiple activities can be held. The sports area 228 can be configured to enable students to play multiple e-sports through their corresponding user graphical representations. The cafeteria 230 can enable students to order food online and socialize through user graphical representations. The teacher's lounge 232 can be configured for teachers to meet and discuss agendas, student progress, etc. through corresponding teacher user graphical representations. The library 234 can enable students to borrow e-books for their courses or leisure reading. Finally, the bookstore 236 can be configured to enable students to purchase books (e.g., e-books or physical books) and / or other school materials.

[0210] Figure 3 A schematic diagram of a sample hybrid system architecture 300 that may be employed in a system for implementing interactions in a virtual environment according to an embodiment is depicted. In some embodiments, the hybrid system architecture 300 is a hybrid communication model for interacting with other peer clients (e.g., other attendees of a virtual conference, classroom, etc.), including a client-server side 304 and a P2P side 306, each of which is configured to communicate with a peer client 304 and a P2P side 306. Figure 3 The use of such a hybrid communication model can achieve fast P2P communication between users, reduce latency issues, and provide network services, data, and resources to each session, enabling multiple interactions between users and with content in the virtual environment. Figure 3 Some elements of Figure 1-2A The same or similar elements are shown in the drawings and therefore the same reference numerals may be used.

[0211] In various embodiments, the usage level and ratio of the client-server side 304 relative to the P2P side 306 depends on the amount of data to be processed, the latency that allows a smooth user experience to be maintained, the desired quality of service (QOS), the services required, etc. In one embodiment, the P2P side 306 is used for video and data processing, streaming, and rendering. For example, when low latency and low data volume need to be processed, and when there is a "heavy" client, this mode of using the hybrid system architecture 300 may be appropriate, which means that the client device contains enough computing power to perform such operations. In another embodiment, a combination of the client-server side 304 and the P2P side 306 is used, such as the P2P side 306 is used for video streaming and rendering, while the client-server side 304 is used for data processing. For example, when there is a large amount of data to be processed, or when other microservices may be needed, this mode of using the hybrid system architecture 300 may be appropriate. In other embodiments, the client-server side 304 can be used for video streaming and data processing, and the P2P side 306 is used for video rendering. For example, when there is a greater amount of data to be processed and / or when only thin clients are available, this mode of using the hybrid system architecture 300 may be appropriate. In other embodiments, the client-server side 304 may be used for video streaming, rendering, and data processing. When very thin clients are available, this mode of using the hybrid system architecture 300 may be appropriate. The hybrid system architecture 300 can be configured to alternate between different forms of use of the client-server side 304 and the P2P side 306 within the same session as needed.

[0212] In some embodiments, at least one cloud server from the client-server end 304 can be an intermediate server, which means that the server is used to facilitate and / or optimize data exchange between client devices. In such embodiments, at least one cloud server can manage, analyze, process and optimize incoming image and multimedia streams, and manage, evaluate and / or optimize the forwarding of outbound streams as a router topology (for example, but not limited to SFU (selective forwarding unit), SAMS (spatial analysis media server), multimedia router, etc.), or an image and media processing server topology (for example, for tasks including but not limited to decoding, combining, improving, mixing, enhancing, expanding, calculating, manipulating, encoding) or a forwarding server topology (including but not limited to MCU, cloud media mixer, cloud 3D renderer, media server) or other server topology can be used.

[0213] In such embodiments, where the intermediate server is a SAMS, such media server manages, analyzes and processes incoming data (e.g., metadata, priority data, data categories, spatial structure data, three-dimensional position, orientation or motion information, images, media, video based on a scalable video codec, or a combination thereof) sent to each client device, and in such analysis manages and / or optimizes the forwarding of outbound data streams to each receiving client device. This may include modifying, scaling up or down the media for time (e.g., varying frame rates), space (e.g., different image sizes), quality (e.g., quality based on different compression or encoding), and color (e.g., color resolution and range), and may achieve optimal bandwidth and computing resource utilization for receiving one or more user client devices based on factors such as the spatial, three-dimensional orientation, distance, and priority relationship to such incoming data of a particular receiving client device user.

[0214] In some embodiments, the media, video and / or data processing tasks include one or more of encoding, transcoding, decoding spatial or 3D analysis and processing, including one or more of image filtering, computer vision processing, image sharpening, background improvement, background removal, foreground blurring, eye masking, facial pixelation, speech distortion, image magnification, image cleansing, bone structure analysis, face or head counting, object recognition, marker or QR code tracking, eye tracking, feature analysis, 3D mesh or volume generation, feature tracking, facial recognition, SLAM tracking and facial expression recognition, or other modular plug-ins in the form of microservices running on such media routers or servers.

[0215] The client-server side 304 uses a secure communication protocol 308 to implement secure end-to-end communication between the client device 118 and the network / application server 310 over the network. Sample suitable secure communication protocols 306 can include, for example, Datagram Transport Layer Security (DTLS), which is itself a secure User Datagram Protocol (UDP), Secure Real-time Transport Protocol (SRTP), Hypertext Transfer Protocol Secure (https: / / ), and Web Sockets Secure (wss: / / ), which are compatible with each other and can provide full-duplex authenticated application access, privacy protection, and integrity of data exchanged in transit. Suitable network / application servers 310 can include, for example, Jetty network application server, which is a Java HTTP network server and Java Servlet container, to implement machine-to-machine communication and proper deployment of network application services.

[0216] Although the network / application server 310 is Figure 3306, and the web server and application server may be separate elements. For example, the web server may be configured to receive client requests via a secure communication protocol 306 and route the requests to the application server. The web / application server 310 may thus receive client requests using the secure communication protocol 306 and process the requests, which may include requesting one or more microservices 312 (e.g., Java-based microservices) and / or looking up data from a database 314 using a corresponding database management system 316. The application / web server 310 may provide session management and many other services, such as state persistence of 3D content and application logic and sessions (e.g., for persistent storage of shared documents, synchronization of interactions and changes in a virtual environment, or persistence of visual states and modifications of a virtual environment). A suitable database management system 316 may be, for example, an object-relational mapping (ORM) database management system, which may be suitable for database management using open source and commercial (e.g., proprietary) services, given the ability of ORM to convert data between incompatible type systems using object-oriented programming languages. In other embodiments, the distributed spatial data bus 318 may further serve as a distributed messaging and resource distribution platform between microservices and client devices by using a publish-subscribe model.

[0217] The P2P end 306 can use a suitable P2P communication protocol 320 that enables real-time communication between peer client devices 118 in a virtual environment through a suitable application programming interface (API), enabling real-time interaction and synchronization, thereby allowing a multi-user collaborative environment. For example, through the P2P end 306, the contributions of one or more users can be directly transmitted to other users, who can observe the changes performed in real time. An example of a suitable P2P communication protocol 320 can be a Web Real-Time Communication (WebRTC) communication protocol, which is a collection of standards, protocols, and JavaScript APIs that combine to enable sharing of P2P audio, video, and data between peer client devices 118. The client device 118 in the P2P end 306 can use one or more rendering engines 322 to perform real-time 3D rendering of a live session. An example of a suitable rendering engine 322 may be a 3d engine based on WebGL, which is a JavaScript API for rendering 2D and 3D graphics within any compatible web browser without the use of plug-ins, allowing one or more processors (e.g., one or more graphics processing units (GPUs)) of the client device 118 to accelerate the use of physics and image processing and effects. In addition, the client device 118 in the P2P end 306 may perform image and video processing and machine learning computer vision techniques through one or more suitable computer vision libraries 324. In one embodiment, the image and video processing performed by the client device in the P2P end 306 includes a background removal process used in the creation of the user's graphical representation before inserting the user's graphical representation into the virtual environment, which may be performed in real time or near real time on the received media stream, or in non-real time on, for example, a photograph. An example of a suitable computer vision library 324 may be OpenCV, which is a library of programming functions configured primarily for real-time computer vision tasks.

[0218] Figure 4 A schematic diagram of a graphical user interface 400 of a virtual environment live session module 402 is depicted by which a user may interact in a virtual environment according to an embodiment.

[0219] Before a user can access the graphical user interface 400 of the virtual environment live session module 402, the user can first receive an invitation from a peer client device to participate in a conversation with a peer user, which can open a P2P communication channel between the user client devices when processing and rendering are performed by the client devices, or alternatively, when processing and rendering are performed by at least one cloud server computer, an indirect communication channel can be opened through the cloud server computer. In addition, as later referenced Figure 5As shown in the description of , a transition may occur from user 3D virtual cutout to user real-time 3D virtual cutout, or a video with a background removed, or a video without a background removed.

[0220] The virtual environment live session module 402 may include a virtual environment screen 404, which includes a graphical user interface that displays the selected virtual environment, which may include an arrangement of the virtual environment associated with the context of the selected vertical field of the virtual environment, and corresponding virtual objects, applications, other user graphical representations, etc. The graphical user interface 400 of the virtual environment live session module 402 may implement and display multiple interactions 406, which are configured for users to participate in each other, such as through their users' real-time 3D virtual cutouts. The virtual environment live session module 402 may include one or more data models associated with the corresponding tasks for implementing each interaction 406, plus the computer instructions required to implement the tasks. Each interaction 406 can be represented in different ways; Figure 4 In the example shown, each interaction 406 is represented as a button on the graphical user interface 400 from the virtual environment live session module 402, where clicking each interaction button can request a corresponding service to perform a task associated with the interaction 406. The virtual environment live session module 402 can, for example, Figure 3 The disclosed hybrid system architecture 300 is implemented.

[0221] Interaction 406 may include, for example, chat 408, screen sharing 410, host options 412, telemetry 414, recording 416, voting 418, document sharing 420, emoticon sending 422, agenda sharing and editing 424, or other interactions 426. Other interactions 426 may include, for example, virtual hugs, hand-raising, handshakes, walking, content addition, meeting summary preparation, object movement, projection, laser pointing, playing games, purchasing, and other social interactions that promote communication, competition, cooperation, conflict resolution between users. Various interactions 406 are described in more detail below.

[0222] Chat 408 may open a chat window, enabling the sending and receiving of text comments and instant resources.

[0223] Screen sharing 410 may enable sharing of a user's screen in real-time to any other participants.

[0224] Host options 412 are configured to provide other options to the conversation host, such as muting one or more users, inviting or removing one or more users, ending the conversation, and the like.

[0225] Remote sensing 414 can view the user's current status, such as absent, busy, available, offline, on a conference call, or in a meeting. The user status can be manually updated through a graphical user interface or automatically updated through a machine vision algorithm based on data obtained from a camera feed.

[0226] Recording 416 can record audio and / or video from the conversation.

[0227] Poll 418 can provide a vote for one or more suggestions posted by any other participant. Through Poll 418, the host or other participant with such permission can initiate a voting session at any time. Topics and options can be displayed for each participant. Depending on the configuration of the Poll 418 interaction, the results can be displayed to all attendees at the end of a timeout period or at the end of everyone's response.

[0228] Document sharing 420 enables sharing of documents in any suitable format with other participants. These documents may also be permanently stored by storing them in the persistent storage of one or more cloud server computers and may be associated with the virtual environment in which the virtual communication occurs.

[0229] Emoticon sending 422 enables sending emoticons to other participants.

[0230] Agenda sharing and editing 424 enables sharing and editing of an agenda that may have been prepared by any participant. In some embodiments, the host can configure a checklist of agenda items before the meeting. The host or other participants with such permission can bring the agenda to the foreground at any time. Through the agenda editing option, items can be written off or can be deferred when consensus is reached.

[0231] Other interactions 426 provide a non-exhaustive list of possible interactions that can be provided in the virtual environment according to the vertical direction of the virtual environment. Raising a hand enables raising a hand during a virtual communication or meeting so that a host or other participant with such rights can enable the user to speak. Walking enables moving in the virtual environment through a user's real-time 3D virtual cutout. Content addition enables a user to add an interactive application or static or interactive 3D asset, animation or 2D texture to the virtual environment. Meeting summary preparation enables automatic preparation of the results of the virtual meeting and distribution of such results to participants at the end of the session. Object movement enables moving objects within the virtual environment. Projection enables content to be projected from the attendee's screen onto a screen or wall available in the virtual environment. Laser pointing enables pointing with a laser to highlight desired content in a presentation. Playing games enables playing one or more games or other types of applications that can be shared during a live session. Purchasing enables purchasing content during a session. Other interactions not mentioned herein can also be configured according to the specific use of the virtual environment platform.

[0232] In some embodiments, the system can also realize the creation of self-organizing virtual communication, which can include the creation of self-organizing voice communication channels between user graphical representations without changing the current perspective or position in the virtual environment. For example, a user graphical representation can approach another user graphical representation and participate in a self-organizing voice dialogue at the place where the two user graphical representation areas are located in the virtual environment. Such communication will be realized by, for example, considering the distance, position and orientation between the user graphical representations, and / or their current availability status (e.g., available or unavailable) or the state configuration of such self-organizing communication, or a combination thereof. In this example, the user graphical representation approaching will see visual feedback on the other user graphical representation, signaling that self-organizing communication is possible, and thus setting the start of the dialogue between the two user graphical representations, wherein the user approaching can speak, and the other user can hear and respond. In another example, the user graphical representation can approach another user graphical representation, click on the user graphical representation, send a dialogue invitation, and after the invitee approves, participate in a self-organizing voice dialogue at the place in the virtual environment where the two user graphical representation areas are located. According to the privacy settings between the two user graphical representations, other users can view the interaction, expression, hand movements, etc. between the user graphical representations, whether or not they can listen to the dialogue. Any of the foregoing interactions of 406 or other interactions 426 may also be performed directly within the virtual environment screen 404 .

[0233] Figure 5 A method 500 is depicted according to an embodiment that enables transitioning from one type of user graphical representation to another type of user graphical representation, for example, from a user 3D virtual cutout to a user real-time 3D virtual cutout, or to a video with or without a background removed.

[0234] This transition may be achieved when the user is engaged in a conversation with other user graphical representations. For example, the user may currently be sitting in an office chair and working on a computer in a virtual office. The user's current graphical representation may be a graphical representation of the user's 3D virtual cutout. At this point, the camera may not be turned on because a live data feed from the user may not be needed. However, if the user decides to turn on the camera, the user's 3D virtual cutout may contain facial expressions provided by analysis of the user's face captured from the user's live data feed, as explained in more detail herein.

[0235] When the user participates in a conversation with another user graphical representation and initiates a live session, if the user camera is not activated, the camera can be activated and a live data feed capture can be initiated, which can provide a live stream of the user, thereby transforming the user 3D virtual cutout into a real-time 3D virtual cutout of the user or a video with or without the background removed. Figure 1 As described in , the live stream of the user's real-time 3D virtual cutout 504 can be processed and rendered by the client or server, or can be sent to other peer client devices in a P2P system architecture or a hybrid system architecture for their own real-time processing and rendering (e.g., by referring to Figure 3 Hybrid system architecture 300 described above).

[0236] Figure 5 The method 500 may start from step 502, approaching a user graphical representation. Then, in step 504, the method 500 may continue by selecting and clicking on the user graphical representation. In step 506, the method 500 may continue by sending a conversation participation invitation to another user graphical representation through a client device or receiving a conversation participation invitation from another user graphical representation. In step 508, the method 500 continues by accepting the received invitation by the corresponding client device. Then, the method 500 continues in step 510, transitioning from the user 3D virtual cut to the user real-time 3D virtual cut or a video with the background removed, or a video without the background removed. Finally, in step 512, the method 500 ends by opening a P2P communication channel between the user client devices when processing and rendering are performed by the client device, or opening an indirect communication channel through the cloud server computer when processing and rendering are performed by at least one cloud server computer. In some embodiments, the conversation includes sending and receiving real-time audio and video displayed from the user real-time 3D virtual cut of the participant.

[0237] Figures 6A-6C 1 and 10. A schematic diagram depicting a plurality of image processing combinations performed by corresponding client devices 118 and cloud servers 102 at a client-server end 304 is shown. The client-server end may be, for example, part of a hybrid system architecture such as Figure 3 The hybrid system architecture 300 is depicted in FIG.

[0238] exist Figures 6A-6C In one embodiment, at least one cloud server 102 may be configured to use a relay network address translation (NAT) traversal (sometimes referred to as TURN) server, which may be useful in situations where a server cannot establish a connection between client devices 118. TURN is an extension of Session Traversal Tools for NAT (STUN).

[0239] NAT is a method of remapping the Internet Protocol (IP) address space to another address space by modifying the network address information in the IP header of the data packet as it is transported through a service routing device. Thus, NAT can provide access to a network (such as the Internet) for a private IP address and allow a single device (such as a routing device) to act as a proxy between the Internet and the private network. NAT can be symmetric or asymmetric. A framework called Interactive Connectivity Establishment (ICE) is configured to find the best path to connect client devices, which can determine whether a symmetric or asymmetric NAT is needed. Symmetric NAT is responsible not only for converting IP addresses from private addresses to public addresses and vice versa, but also for converting ports. On the other hand, asymmetric NAT uses a STUN server to allow clients to discover their public IP addresses and the type of NAT behind them, which can be used to establish a connection. In many cases, STUN may only be used during connection setup, and once the session is established, data can begin to flow between client devices.

[0240] TURN can be used in the context of symmetric NAT and remains in the media path after the connection is established while processed and / or unprocessed data is relayed between client devices.

[0241] Fig. 6A The client-server side 304 is depicted including client device A, cloud server 102, and client device B. Fig. 6A , client device A is the sender of data to be processed, and client device B is the receiver of the data. A plurality of image processing tasks are depicted and categorized based on whether they are performed by client device A, cloud server 102, or client device B, and are thus categorized as client device A processing 602, server image processing 604, and client device B processing 606.

[0242] Image processing tasks include background removal 608, further processing or improvement 610, and insertion and integration into a virtual environment 612. Figure 6B and 6C and from Figure 7B It will become apparent that the combination of the three image processing tasks shown herein can be used to generate, improve, and insert / integrate user graphical representations into a virtual environment. Figure 6B-6C and Figure 7B-7C , background removal 608 is depicted as “BG” 608 , further processing or improvement 610 is depicted as “++” 610 , and insertion and combination into a virtual environment 612 is depicted as “3D” 612 .

[0243] In some embodiments, inserting and combining the user graphical representation into the virtual environment includes generating one or more virtual cameras that are virtually placed and aligned in front of the user graphical representation, for example, in front of a video with background removed, or a video without background removed, or a user 3D virtual cutout, or a user real-time 3D virtual cutout. In one embodiment, one or more virtual cameras can point outward from eye level. In another embodiment, two virtual cameras, one for each eye, can point outward from both eye levels. In another embodiment, one or more virtual cameras can point outward from the center of the head position of the user graphical representation. In another embodiment, one or more virtual cameras can point outward from the center of the user graphical representation. In another embodiment, one or more virtual cameras can be placed in front of the user graphical representation, for example, at the head level of the user graphical representation, pointing to the user graphical representation when in a self-perspective. One or more virtual cameras are created by at least associating the captured user perspective data with the perspective of the user graphical representation within the virtual environment using computer vision. One or more virtual cameras are automatically updated by tracking and analyzing user eye and head tilt data, or head rotation data, or a combination thereof, and can also be manually changed by the user according to the perspective selected by the user.

[0244] The image processing combination and corresponding usage level of client device A processing 602, server image processing 604 and client device B processing 606 depends on the amount of data to be processed, the latency to allow a smooth user experience to be maintained, the desired quality of service (QOS), the required services, etc.

[0245] Figure 6B Describe image processing combinations 1-4.

[0246] In image processing combination 1, client device A generates a user graphic representation with background removed 608, and sends the user graphic representation with background removed to at least one cloud server 102 for further processing or improvement 610, generating an enhanced user graphic representation with background removed. At least one cloud server sends the enhanced user graphic representation with background removed to client device B, and client device B inserts and combines the enhanced user graphic representation with background removed into the virtual environment.

[0247] In image processing combination 2, client device A generates a user graphic representation with background removed 608, performs further processing or improvement 610 on it, generates an enhanced user graphic representation with the background removed, and then sends it to at least one cloud server 102. At least one cloud server 102 sends the enhanced user graphic representation with the background removed to client device B, and client device B inserts and combines the enhanced user graphic representation with the background removed into the virtual environment.

[0248] In image processing combination 3, client device A generates a user graphic representation with background removal 608, performs further processing or improvement 610 on it, generates an enhanced user graphic representation with background removed, and inserts and combines the enhanced user graphic representation with background removed into the virtual environment. Client device A then sends the enhanced user graphic representation with background removed, inserted and combined into the virtual environment, to the cloud server for relaying to client device B.

[0249] In the image processing combination 4, client device A generates a user graphic representation including background removal 608, and sends the user graphic representation with the background removed to at least one cloud server 102 for further processing or improvement 610, generating an enhanced user graphic representation with the background removed. The at least one cloud server then inserts and combines the enhanced user graphic representation with the background removed into the virtual environment and then sends it to client device B.

[0250] Figure 6C Describe image processing combinations 5-8.

[0251] In image processing combination 5, client device A generates a user graphical representation including background removal 608, and sends the user graphical representation with the background removed to at least one cloud server 102 for relaying to client device B. Client device B performs further processing or improvement 610 on the user graphical representation with the background removed to generate an enhanced user graphical representation with the background removed, and client device B inserts and combines the enhanced user graphical representation with the background removed into the virtual environment.

[0252] In the image processing combination 6, client device A sends a camera live data feed received from at least one camera and sends unprocessed data to at least one cloud server 102, which performs generation of a user graphical representation including background removal 608, and performs further processing or improvement 610 on the user graphical representation with the background removed, generates an enhanced user graphical representation with the background removed, and sends it to client device B. Client device B inserts and combines the enhanced user graphical representation with the background removed into the virtual environment.

[0253] In the image processing combination 7, the client device sends a camera live data feed received from at least one camera and sends unprocessed data to the at least one cloud server 102. The at least one cloud server 102 generates a user graphical representation including background removal 608, performs further processing or improvement 610 on the user graphical representation with the background removed, generates an enhanced user graphical representation with the background removed, and then inserts and combines the enhanced user graphical representation with the background removed into the virtual environment sent to the client device B.

[0254] In the image processing combination 8, client device A sends a camera live data feed received from at least one camera to at least one cloud server 102 and sends unprocessed data for relaying to client device B. Client device B uses the data to generate a user graphic representation including background removal 608, and performs further processing or improvement 610 on the user graphic representation with the background removed to generate an enhanced user graphic representation with the background removed, and then inserts and combines the enhanced user graphic representation with the background removed into the virtual environment. In some embodiments, it can be appreciated that at least one cloud server 102 can be an intermediary server, meaning that the server uses an intermediary server topology to facilitate and / or optimize data exchange between client devices.

[0255] In such embodiments, at least one cloud server may be an intermediate server, meaning that the server is used to facilitate and / or optimize data exchange between client devices. In such embodiments, at least one cloud server may manage, analyze, and optimize incoming multimedia streams, and manage, evaluate, and optimize forwarding of outbound streams as a router topology (e.g., SFU, SAMS, multimedia router, etc.), or a media processing (e.g., performing tasks including decoding, combining, improving, mixing, enhancing, augmenting, calculating, manipulating, or encoding) or forwarding server topology (e.g., but not limited to, a multipoint control unit, a cloud media mixer, a cloud 3D renderer), or other server topology.

[0256] In such embodiments, where the intermediate server is a SAMS, such media server manages, analyzes, and processes incoming data (e.g., metadata, priority data, data categories, spatial structure data, three-dimensional position, orientation or motion information, images, media, or video based on a scalable video codec) of a sending client device, and in such analysis manages or optimizes the forwarding of outbound data streams to receiving client devices. This may include modifying, scaling up, or scaling down the media for time (e.g., varying frame rates), space (e.g., different image sizes), quality (e.g., quality based on different compression or encoding), and color (e.g., color resolution and range) based on, for example, the spatial, three-dimensional orientation, distance, and priority relationship of a particular receiving client device user to such incoming data, thereby achieving optimal bandwidth and computing resource utilization for receiving one or more user client devices.

[0257] The intermediate server topology may be suitable for, for example, image processing combinations 1-8, where at least one cloud server 102 performs processing between client devices A and B, such as Figures 6A-6C shown.

[0258] Figures 7A-7C A schematic diagram depicting a plurality of image processing combinations performed by corresponding client devices in the P2P end 306 is shown. Figure 7A-7B The P2P end 306 may be part of a hybrid system architecture, such as Figure 3 The hybrid system architecture 300 is depicted in FIG.

[0259] Fig. 7A A P2P peer 306 is depicted that includes a peer A and a peer B, where peer A is the sender of data to be processed and peer B is the receiver of the data. A number of image and media processing tasks are depicted and categorized based on whether they are performed by peer A or peer B, and are therefore categorized as peer A processing 702 and peer B processing 704. Image and media processing tasks may include, but are not limited to, background removal 608, further processing or improvement 610, and insertion and composition into a virtual environment 612.

[0260] Figure 7B Describe image processing combinations 1-3.

[0261] In image processing combination 1, peer device A generates a user graphic representation with background removal 608, performs further processing or improvement 610 on it, generates an enhanced user graphic representation with background removed, and inserts and combines the enhanced user graphic representation with background removed into a virtual environment with three-dimensional coordinates. Peer device A then sends the enhanced user graphic representation with background removed, which is inserted and combined into the virtual environment, to peer device B.

[0262] In image processing combination 2, peer device A generates a user graphical representation including background removal 608 and sends the user graphical representation with background removed to peer device B. Peer device B performs further processing or improvement 610 on the user graphical representation with background removed to generate an enhanced user graphical representation with background removed, which peer device B inserts and combines into the virtual environment.

[0263] In image processing combination 3, peer device A sends a camera live data feed received from at least one camera and sends encoded data to peer device B. Peer device B decodes and uses the data to generate a user graphics representation with background removal 608, and performs further processing or improvement 610 on the user graphics representation with background removed, generating an enhanced user graphics representation with background removed, and then inserting and combining the enhanced user graphics representation with background removed into the virtual environment.

[0264] Figure 7C Describes image processing combinations 4-6.

[0265] exist Figure 7CIn one embodiment, at least one cloud server 102 may be configured as a STUN server, which allows peer devices to discover their public IP addresses and the NAT types behind them, which information may be used to establish data connections and data exchanges between peer devices. Figure 7C In another embodiment, at least one cloud server 102 may be configured for signaling, which may be used for peer devices to locate and connect to each other, as well as to exchange data through communication coordination performed by the at least one cloud server.

[0266] In all image and processing combinations 4-6, at least one cloud server 102 can use SAMS, SFU, MCU or other functional server topologies because at least one cloud server 102 serves between peer devices A and B.

[0267] In the image processing combination 4, peer device A generates a user graphic representation including background removal 608, performs further processing or improvement 610 on it, generates an enhanced user graphic representation with background removed, and inserts and combines the enhanced user graphic representation with background removed into the virtual environment. Peer device A then sends the enhanced user graphic representation with background removed inserted and combined into the virtual environment to peer device B via at least one cloud server acting as a STUN or signaling server.

[0268] In image processing combination 5, peer device A generates a user graphic representation including background removal 608 and sends the user graphic representation with background removed to peer device B via at least one cloud server acting as a media router server. Peer device B performs further processing or improvement 610 on the user graphic representation with background removed to generate an enhanced user graphic representation with background removed, which client device B inserts and combines into the virtual environment.

[0269] In the image processing combination 6, peer device A sends a camera live data feed received from at least one camera and sends unprocessed data to peer device B via at least one cloud server acting as a STUN or signaling server. Peer device B uses the data to generate a user graphics representation with background removal 608, and performs further processing or improvement 610 on the user graphics representation with the background removed, generating an enhanced user graphics representation with the background removed, and then inserting and combining the enhanced user graphics representation with the background removed into the virtual environment.

[0270] Figure 8 A user authentication system 800 based on a user graphical representation that can be used in embodiments of the present disclosure is shown. For example, the user authentication system 800 based on a user graphical representation can be used to access a user account that can be authorized to access a virtual environment platform, such as Figure 1 and Figure 2A A virtual environment platform 108 .

[0271] The user authentication system 800 based on user graphical representation includes one or more cloud server computers 802, the cloud server computers including at least one processor 804 and a memory 806 storing data and instructions, the memory including a user database 808 storing user data associated with user accounts 810 and one or more corresponding user graphical representations 812. The user authentication system 800 based on user graphical representation also includes a facial scanning and authentication module 814 connected to the database 808 storing data associated with the user accounts 810. The one or more cloud server computers 802 are configured to authenticate the user by performing a facial scan of the user through the facial scanning and authentication module 814. The facial scan includes extracting facial feature data from the camera data received from the client device 822 and checking the extracted facial feature data for a match with the user graphical representation in the user database 808.

[0272] exist Figure 8 In the example shown, the system 800 also includes at least one camera 816 configured to obtain image data 818 from a user 820 of at least one client device 822 requesting access to a user account 810. The at least one camera 816 is connected to the at least one client device 822, which is configured to transmit the data captured by the camera 816 to one or more cloud server computers 802 for further processing. Alternatively, the camera 816 can be directly connected to the one or more cloud server computers 802. The one or more cloud server computers 802 are configured to perform a facial scan of the user through the facial scanning and authentication module 814, check the user database 808 for a match with an existing user graphical representation, and if the user account 810 is confirmed and available, authenticate the user by providing the user with a corresponding user graphical representation 812 and access to the user account 810. Alternatively, if the user account 810 is not available, the one or more cloud server computers 802 are configured to authenticate the user by generating a new user graphical representation 812 and a new user account 810 stored in the user database 808 through data 818 obtained from the live data feed.

[0273] The user account 810 may be used, for example, to access the virtual environment platform or any other application (e.g., an application that may be linked to the environment platform), such as any interactive application, a game, an email account, a university profile account, a work account, etc. Given the steps of, for example, generating a user graphical representation 812 or retrieving an existing user graphical representation 812 from a user database 808, the graphical representation-based user authentication system 800 of the present disclosure provides a higher level of convenience and security than a standard camera-based facial detection authentication system.

[0274] In some embodiments, one or more cloud server computers are further configured to check the date of the matching user graphical representation and determine whether the matching user graphical representation needs to be updated. In an embodiment, if the user account 810 is available, and in response to one or more cloud server computers 802 checking the date of the available user graphical representation 812, one or more cloud server computers 802 determine whether the existing user graphical representation 814 needs to be updated by comparing with the corresponding threshold or security requirements. For example, if a system security update is to be performed, all user graphical representations may need to be updated, or at least the graphical representations created before the specified date need to be updated. If the user graphical representation 814 is required, one or more cloud server computers 802 generate a user graphical representation update request to the corresponding client device 822. If the user 820 approves the request, one or more cloud server computers 802 or client devices 822 continue to generate the user graphical representation 814 based on the data 818 from the live camera feed. If no update is required, after authentication, one or more cloud server computers 802 continue to retrieve the existing user graphical representation 812 from the user database 808.

[0275] In some embodiments, the user graphical representation 812 is inserted into the two-dimensional or three-dimensional virtual environment, or on a third-party source linked to the virtual environment, and combined with the two-dimensional or three-dimensional virtual environment. For example, the user graphical representation 812 can be inserted on a third-party source linked to the virtual environment by overlaying it on the screen of a third-party application or website that is integrated or coupled with the system of the present disclosure).

[0276] In one example, overlaying the user graphic representation 812 on the screen of the third-party source is done on top of a 2D website or application linked to a virtual environment. For example, two or more friends who go to a shopping website together can overlay their user graphic representations on the shopping website to explore the content of the website and / or interact with the content of the website. In another example, overlaying the user graphic representation 812 on the screen of the third-party source is done on top of a 3D game session linked to a virtual environment. For example, a user can access an e-sports game session linked to a virtual environment through his or her user graphic representation 812, and the user graphic representation 812 can be overlaid on top of the e-sports game session with the user graphic representations 812 of other team members. In these examples, such overlays of the user graphic representation 812 can achieve a coherent and multicast view of the expressions and communications of all users during accessing a 2D website or experiencing a 3D game session.

[0277] In some embodiments, the generation process of the user graphical representation 812 occurs asynchronously with the user 820 accessing the user account 810. For example, if the user graphical representation based authentication system 800 determines that the user 820 has already been authenticated after performing a facial scan, the user graphical representation based authentication system 800 can enable the user 820 to access the user account 810 while the new user graphical representation 812 is being generated, so that it can be provided to the user 812 once it is ready, and then inserted and combined into the virtual environment.

[0278] In some embodiments, the one or more cloud server computers 802 further authenticate the user 802 via a login authentication credential comprising a personal identification number (PIN), or a username and password, or a combination of camera authentication and a PIN, or a username and password.

[0279] In some embodiments, the user graphical representation based authentication system 800 triggers authentication in response to activation of an invitation link or deep link sent from one client device 822 to another client device. Clicking on the invitation link or deep link triggers at least one cloud server computer 802 to request the user to authenticate. For example, the invitation link or deep link can be used for a phone call, a meeting, or a video game session invitation, where the invited user can be authenticated by the user graphical representation based authentication system 800.

[0280] In another embodiment, facial scanning uses 3D authentication, including guiding the user to perform a head movement pattern and extracting 3D facial data based on the head movement pattern. This can be done using application instructions stored in at least one server computer, which implements 3D authentication by guiding the user to perform a head movement pattern, such as performing one or more head gestures, tilting or rotating the head horizontally or vertically in a circular motion, performing a user-generated gesture pattern, or a specific head movement pattern, or a combination thereof. 3D authentication recognizes other features based on data obtained from the camera live video data feed, rather than just comparing and analyzing one view or image. In this 3D authentication embodiment, the facial scanning process can recognize other features from the data, which can include facial data, including head movement patterns, facial volume, height, depth of facial features, facial scars, tattoos, eye color, facial skin parameters (e.g., skin color, wrinkles, pore structure, etc.), reflectance parameters, in addition to, for example, just the location of such features on the facial topology, which may be the case with other types of facial detection systems. Capturing such facial data can therefore increase the capture of a real face, which can be used to generate a realistic user graphical representation. Facial scanning using 3D authentication can be performed using a high-resolution 3D camera, a depth camera (e.g., LIDAR), a light field camera, etc. The facial scanning process and 3D authentication can use deep neural networks, convolutional neural networks, and other deep learning techniques to retrieve, process, and evaluate the user's authentication using facial data.

[0281] Fig. 9 A schematic diagram of a third person perspective 900 of a virtual environment 110 is depicted by a user graphical representation 120 , where the virtual environment 110 is a virtual office.

[0282] The virtual office includes one or more desks 902 , office chairs 904 , office computers 906 , a projection surface 908 for projecting content 910 , and a plurality of user graphical representations 120 representing corresponding users accessing the virtual environment 110 through their client devices.

[0283] The user graphical representation 120 may initially be a user 3D virtual cutout, and may be transformed into a user real-time 3D virtual cutout after an invitation approval process, containing a user real-time video stream with background removed, or a video with background removed, or a video without background removed, generated based on a real-time 2D or 3D live video stream data feed obtained from a camera. Figure 5 As described, the process may include opening a communication channel to enable multiple interactions within the live session, as described with reference to Figure 4For example, a user may initially sit in an office chair 904 and work on a corresponding office computer 906, which may represent actual actions that the user is performing in real life. Other users may be able to view (e.g., via Figure 4 Remote sensing 414 of the current user status, such as whether the user is absent, busy, available, offline, on a conference call, or in a meeting. If the user is available, another user graphical representation can approach the user in question and send an invitation to participate in the conversation. For example, both users can decide to move to a private meeting room in the virtual office and start a live session that enables multiple interactions. The users may also be able to project desired content on the projection surface 908 (e.g., via screen sharing).

[0284] In some embodiments, the virtual office also includes a virtual computer, which includes virtual resources from one or more cloud computer resources, which are accessed by a client device and assigned to the virtual computer resources by a management tool. The virtual computer can be associated with an office computer 906. However, the virtual computer can also be associated with a personal home computer or a computer from any other location that can access the virtual computing resources based on the cloud computer. Resources can include storage, network and processing power required for performing various tasks. In addition, in the example of an office space, the virtual computer associated with the virtual office computer 906 can then be coupled to the user's real office computer, so that, for example, when the user logs on to such a virtual computer, the data stored in the virtual office computer 906 can be obtained from the real office computer of the physical office or any other space with a physical computer. The virtual infrastructure including all virtual computers associated with the virtual office computer 906 can be managed by the virtual environment platform using an administrator option based on exclusive administrator privileges (for example, using the virtual environment 110 to provide to the IT team of the organization). Thus, the virtual environment platform of the present disclosure enables virtual office management and provides multiple options that expand the possibilities of typical virtual meeting and conference applications, increase the realism of collaboration and interaction, and simplify the way collaboration occurs.

[0285] Figures 10A-10B Depicted is a schematic diagram of a virtual environment viewed through a corresponding user graphical representation, wherein the virtual environment is a virtual classroom 1000 , according to an embodiment. Figures 10A-10B The user graphical representations of students and teachers in can be either 3D virtual cutouts of users constructed from user uploaded or third party source photos, or real-time 3D virtual cutouts of users with background removed generated based on real-time 2D or 3D live video stream data feed obtained from a camera, or videos with background removed, or videos without background removed.

[0286] exist Fig. 10AIn the example, multiple user graphical representations of students 1002 are remotely participating in a classroom lecture provided by a user graphical representation of a teacher 1004. The teacher 1004 can project classroom content 1006 onto one or more projection surfaces 1008, such as on a virtual classroom whiteboard. The virtual classroom 1000 can also include multiple virtual classroom desks 1010 that can support user learning. Figure 4 As disclosed, multiple interaction options may be provided to the student 1002, such as raising a hand, screen sharing (e.g., on the projection surface 1008), laser pointing to specific content, etc., as appropriate. Fig. 10A , a user graphical representation of the teacher 1004 is graphically projected onto the projection surface.

[0287] Fig. 10B Depicted with Fig. 10A A similar embodiment, except that the user graphical representation of the teacher 1004 sits behind a virtual desk 1012, and only the content 1006 is shared or projected on the virtual classroom whiteboard projection surface 1008. Because the teacher 1004 shares the same virtual space with the students 1002, and can move around in the classroom 1000, a more realistic and interactive experience is created for the students 1002 and the teacher 1004.

[0288] Fig.11 A schematic diagram of multiple virtual camera positions 1100 is depicted in accordance with an embodiment.

[0289] exist Fig.11 In FIG. 1 , two user graphical representations 1102, a user 3D virtual cutout 1104, and a user real-time 3D virtual cutout 1106 have one or more virtual camera positions 1100 for one or more virtual cameras, each position including a viewing direction, angle, and field of view that creates a perspective for the user graphical representation.

[0290] In one embodiment, one or more virtual cameras may be located at eye level 1108, pointing outward from the eye level of the user graphical representation 1102. In another embodiment, two virtual cameras, one for each eye, may point outward from both eye levels 1110 of the user graphical representation 1102. In another embodiment, one or more virtual cameras may point outward from the center of the user graphical representation's head position 1112. In another embodiment, one or more virtual cameras may point outward from the center 1114 of the user graphical representation 1102. In another embodiment, one or more virtual cameras may be placed in front of the user graphical representation 1102, such as at the head level of the user graphical representation 1102, pointing toward the user graphical representation 1102 when in the ego perspective 1116. As described with reference to FIG. Figures 6A-7C As explained, one or more virtual cameras may be created during the insertion and incorporation of the user graphical representation into the virtual environment.

[0291] In an embodiment, the user's perspective captured by the camera is associated with the perspective of the user graphical representation and an associated virtual camera using computer vision, thereby manipulating the virtual camera. In addition, the virtual camera can be automatically updated by, for example, tracking and analyzing user eye and head tilt data, or head rotation data, or a combination thereof.

[0292] Fig.12 A schematic diagram of a system 1200 for virtually broadcasting from within a virtual environment is depicted.

[0293] System 1200 may include one or more server computers. Fig.12 The illustrative system 1200 shown includes at least one media server computer 1202 including at least one processor 1204 and memory 1206 including data and instructions implementing a data exchange management module 1208 that manages data exchange between client devices 1210. The system 1200 also includes at least one virtual environment 1212 connected to the at least one media server computer 1202, the at least one virtual environment 1212 including a virtual broadcast camera 1214 located within the at least one virtual environment 1212 and configured to capture multimedia streams from within the at least one virtual environment 1212. The at least one virtual environment 1212 may be hosted by at least one dedicated server computer connected to the at least one media server computer 1202 via a network, or may be hosted in a peer-to-peer infrastructure and relayed through the at least one media server computer 1202. The multimedia streams are sent to the at least one media server computer 1202 for broadcast to at least one client device 1210. The system 1200 also includes at least one camera 1216 that obtains live feed data from a user 1218 of at least one client device 1210 and transmits the live feed data from the user to at least one media computer 1202 via the at least one client device 1210. Figures 6A-7C As disclosed, the live feed data received by at least one media computer 1202 may be generated by a combination of multiple image processing operations.

[0294] At least one virtual broadcast camera 1214 sends multimedia streams to at least one media server computer 1202 for broadcasting corresponding multimedia streams to receiving client devices 1210 based on data exchange management from the at least one media server computer 1202. The multimedia streams are displayed via corresponding displays to corresponding user graphical representations 1220 of users 1218 of at least one client device 1210. Data exchange management between client devices 1210 by the data exchange management module 1208 includes analyzing incoming multimedia streams and evaluating and forwarding outbound multimedia streams.

[0295] In some embodiments, when forwarding outbound multimedia streams, at least one media server computer 1202 utilizes a routing topology that includes a selective forwarding unit (SFU), traversal of relay NAT (TURN), a spatial analysis media server (SAMS), or other suitable multimedia server routing topologies, or a media processing and forwarding server topology, or other suitable server topologies. In other embodiments, when utilizing a media processing topology, at least one media server computer 1202 is configured to decode, combine, improve, mix, enhance, augment, compute, manipulate, and encode multimedia streams. In other embodiments, when utilizing a forwarding server topology, at least one media server computer 1202 utilizes one or more of a multipoint control unit (MCU), a cloud media mixer, and a cloud 3D renderer.

[0296] In some embodiments, the incoming multimedia stream includes user priority data and distance relationship data, and the user priority data includes a higher priority score for user graphical representations closer to the incoming multimedia stream source and a lower priority score for user graphical representations farther from the incoming multimedia stream source. In an embodiment, the multimedia stream sent to at least one media server by at least one client device 1210 and / or broadcast camera 1214 includes data related to user priority and the distance relationship between the corresponding user graphical representation 1202 and the multimedia stream, including metadata, or priority data, or data category, or spatial structure data, or three-dimensional position, or orientation or motion information, or image data, or media data, and video data based on a scalable video codec, or a combination thereof. In other embodiments, the priority data includes a higher priority score for users closer to the virtual multimedia stream source 1224, and a lower priority score for users farther from the virtual multimedia stream source 1224. In other embodiments, the forwarding of the outbound multimedia stream is based on the user priority data and the distance relationship data. In an embodiment, the forwarding of the outbound multimedia stream implemented by the media server based on the user priority and distance relationship data includes optimizing the bandwidth and computing resource utilization of one or more receiving client devices.

[0297] In some embodiments, at least one virtual broadcast camera 1214 is virtually viewed in at least one virtual environment 1212 as a virtual broadcast camera 1214 configured to broadcast a multimedia stream within the at least one virtual environment 1212. The virtual broadcast camera 1214 may be located near a virtual multimedia stream source 1224 and may also be moved around within the virtual environment 1212. In other embodiments, the virtual broadcast camera 1214 may be managed by a client device 1210 accessing the virtual environment, may be configured to manipulate a viewpoint of the camera updated in the virtual environment, and broadcast the updated viewpoint to at least one client device associated with the virtual broadcast camera 1214.

[0298] In some embodiments, the virtual multimedia stream source 1224 includes a live virtual event that includes one or more of a panel discussion, lecture, conference, demonstration, webinar, entertainment show, sporting event, and performance, in which multiple user graphical representations of real speakers speaking remotely (e.g., while being recorded to their corresponding cameras 1216 while speaking from their homes) are placed within the virtual environment 1212.

[0299] In some embodiments, the multimedia stream is either viewed as a real-time 3D view in a web browser rendered by a client or cloud computer, or can be streamed to be viewed live in a suitable video platform (e.g., YouTube TM Live broadcast, Twitter TM , Facebook TM Live broadcast, Zoom TM wait).

[0300] exist Fig.12 In the example shown, users AC access the virtual environment 1212 through their corresponding client devices, wherein each user AC has a camera 1216 that sends a multimedia stream corresponding to each user AC, which can be used to generate a user graphical representation AC and insert and combine it into the virtual environment 1212, as described in relation to the embodiments of the present disclosure. Therefore, in the virtual environment 1212, each user AC has a corresponding user graphical representation AC. The multimedia stream sent by at least one camera 1216 through at least one client device 1210 and the multimedia stream sent by at least one broadcast camera 1214 to at least one media server 1202 contain data related to user priority and the distance relationship between the corresponding user graphical representation and the multimedia stream. The data includes, for example, metadata, priority data, data category, spatial structure data, three-dimensional position, orientation or motion information, image data, media data, video data based on a scalable video codec, etc. The data can be used by the data exchange management module 1208 to manage data exchange between client devices 1210, including analyzing and optimizing incoming multimedia streams and evaluating and optimizing the forwarding of outbound multimedia streams.

[0301] Thus, for example, when user graphical representation A is closer to virtual multimedia stream source 1224 in virtual environment 1212, forwarding of outbound media streams may be optimized to include, for example, images for user graphical representation A having a higher resolution than images provided to user graphical representations B and C. The multimedia streams may be viewed by users via their client devices 1210 through their user graphical representations 1222, for example, in first person within virtual environment 1212. In some examples, the multimedia streams are viewed as real-time 3D views in a web browser rendered by a client or cloud computer. Users may view multimedia streams of an event (e.g., a webinar, conference, panel discussion, presentation, etc.) as real-time 3D views in a web browser rendered by a client or cloud computer, or may be streamed for live viewing in a suitable video platform and / or social media.

[0302] Fig.13 A schematic diagram of a system 1300 for delivering applications within a virtual environment is depicted.

[0303] The system 1300 includes at least one cloud server computer 1302, which includes at least one processor 1304 and memory 1306, the memory 1306 containing data and instructions to implement at least one virtual environment 1308 linked to an application module 1310. The application module 1310 contains one or more installed applications 1312 and corresponding application rules 1314 for multi-user interaction. In response to selection of a virtual environment host 1316 via a client device 1318, the one or more installed applications 1312 are displayed and activated during a session of the virtual environment 1302, so that a virtual environment host user graphical representation 1320 and any participant user graphical representations 1322 within the virtual environment 1308 can interact with the one or more installed applications 1312 via the corresponding client device 1318. The at least one cloud server computer 1302 manages and processes received user interactions with the one or more installed applications 1312 according to the application rules for multi-user interaction 1314 in the application module 1310. The at least one cloud server computer 1302 further forwards the processed interactions accordingly to each client device 1318 to establish a multi-user session in the virtual environment 1308 to achieve a shared experience based on the multi-user interaction application rules 1314 .

[0304] In some embodiments, the multi-user interactive application rules 1314 are stored and managed in one or more separate application servers, which can be connected to at least one cloud server computer 1302 via a network.

[0305] In some embodiments, one or more applications are installed from application installation packages available in the application library, and application services are provided through corresponding application programming interfaces. In other embodiments, the application library is context-filtered. In an embodiment, context filtering is designed to provide only relevant applications for a specific context. For example, the host 1316 can context-filter an application library (e.g., an application store) to find applications related to a specific context (e.g., learning, entertainment, sports, reading, purchasing, weather, work, etc.), and can select an application of interest to be installed in the application module 1310. In other embodiments, the application library is hosted on one or more third-party server computers, or hosted on at least one cloud server computer 1302.

[0306] In some embodiments, one or more installed applications are shared with and viewed through a virtual display application installed in a corresponding client device. In an embodiment, after installation and activation, one or more installed applications 1312 are shared with and viewed through a virtual display application 1324 installed in a corresponding client device 1318. The virtual display application 1324 can be configured to receive one or more installed applications 1312 from an application library and publish one or more selected installed applications 1312 to be displayed to the conference host user graphical representation 1320 and other participant user graphical representations 1322 in the virtual environment 1308 through their corresponding client devices 1318. The virtual display application 1324 can be an online or installed file viewer application that can be configured to receive and display the installed applications 1312.

[0307] In some embodiments, application module 1310 is represented as a 2D screen or 3D volume application module graphical representation 1326 within a virtual environment, where content from installed applications 1312 is displayed to user graphical representation 1322. In other embodiments, virtual display application 1324 is represented as a 2D screen or 3D volume, where content from installed applications is displayed to user graphical representation in virtual environment 1308.

[0308] In some embodiments, one or more applications 1312 are installed directly within virtual environment 1308 before or while a multi-user session occurs. In other embodiments, one or more applications 1312 are installed using a virtual environment setup tool before a multi-user session is started.

[0309] In some embodiments, some application rules for multi-user interaction can define synchronous interaction or asynchronous interaction or a combination thereof, thereby updating user interaction and corresponding updated views of one or more applications. Both synchronous and asynchronous interactions can be configured by multi-user interaction application rules 1314, and can be implemented via parallel processing by at least one server computer 1302, or by a separate server computer dedicated to processing a single user interaction with at least one installed application 1312.

[0310] For example, if the host 1316 is a teacher, the teacher can select a workbook application that displays the contents of a book to the user. The teacher can edit the workbook, and the students can view the same workbook with edits from the teacher through their virtual display application 1324, when choosing to use synchronous interaction and corresponding update views, or without edits from the teacher when synchronous interaction is selected. In another example, in a presentation application containing a presentation file with multiple slides, asynchronous interaction can enable each user to view a single slide asynchronously. In another example, in the case of an educational application, a heart anatomy is presented when testing students, where the student's interaction is synchronous so that other students witness and observe the interaction performed by the student. In another example, the teacher can write on a whiteboard, allowing students to synchronously view the text written on the whiteboard through their virtual display application. In another example, a video player application can synchronously display a video to all students.

[0311] In some exemplary embodiments, virtual environment 1308 is a classroom, or an office space, or a conference room, or a meeting room, or an auditorium, or a theater.

[0312] Fig.14 Describes a method based on a method for Fig.13 Schematic diagram of virtual environment 1308 of system 1300 for delivering applications within a virtual environment is depicted in FIG.

[0313] The virtual environment 1308 includes an application module graphical representation 1326, which includes at least one installed application 1312 selected by the host 1316 of the virtual environment 1308, and two users AB viewing and interacting with the installed application 1312 through their corresponding virtual display application 1324. It can be understood that user A can view a certain page (e.g., page 1) of the book application through the virtual display application 1324, and the page can be the same as the page selected by the host 1316 through the application module graphical representation 1326, indicating synchronous interaction and management of the installed application 1312. On the other hand, through the asynchronous interaction and management of the installed application 1312 by the virtual display application 1324, user B can view a different page from the host 1316 and user A.

[0314] Fig.15 Depicted is a schematic diagram of a system 1500 for providing virtual computing resources within a virtual environment, according to an embodiment.

[0315] The system 1500 includes a server computer system including one or more server computers, including at least one cloud server computer 1502, the at least one cloud server computer 1502 including at least one processor 1504 and memory 1506, the memory 1506 including data and instructions to implement at least one virtual environment 1508, and at least one virtual computer 1510 associated with the at least one virtual environment 1508. The at least one virtual computer 1510 receives virtual computing resources from the server computer system. In an embodiment, the at least one virtual computer has a corresponding graphical representation 1512 in the virtual environment 1508. The graphical representation 1512 can provide other benefits, such as facilitating user interaction with the virtual computer and increasing the realism of the user experience (e.g., for a home office experience). Therefore, in an embodiment, the at least one virtual computer includes at least one corresponding associated graphical representation 1512 located within the virtual environment 1508, wherein the at least one virtual computer 1510 receives virtual computing resources from the at least one cloud server computer 1502. The system 1500 also includes at least one client device 1514 connected to the at least one server computer 1510 via a network. In response to at least one client device 1514 accessing one or more virtual computers 1510 (eg, by interacting with corresponding graphical representations), at least one cloud server computer 1502 provides at least a portion of available virtual computing resources to the at least one client device 1514 .

[0316] In some embodiments, the virtual computing resources are accessed by a user graphical representation 1516 of a user 1518, who accesses (e.g., interacts with) one or more graphical representations of a virtual computer 1512 within at least one virtual environment 1508 through a corresponding client device 1514 and is provided thereby.

[0317] In some embodiments, the virtual computer graphical representation 1512 is spatially located within the virtual environment for access by the user graphical representation. In an embodiment, the arrangement of the virtual environment 1508 is associated with the contextual theme of the virtual environment 1508, and may include an arrangement of virtual items, furniture, floor plans, etc., for education, meetings, work, shopping, services, socializing, and entertainment, respectively. In other embodiments, one or more virtual computer graphical representations are located within the arrangement of the virtual environment 1508 for access by one or more user graphical representations 1516. For example, a virtual computer may be located in a virtual room that the user graphical representation 1516 will access when participating in an activity that may require or benefit from the ability to use resources associated with a virtual computer, such as working on a project in a virtual classroom, lab, or office.

[0318] In some embodiments, the server computer system is configured to provide at least a portion of virtual computing resources to at least one client device in response to a user accessing at least one cloud server computer by logging into at least one client device without accessing a virtual environment. In an illustrative scenario, a user 1518 accessing at least one cloud server computer 1502 accesses virtual computing resources by physically logging into a client device 1514 connected to at least one cloud server computer 1502 via a network, triggering provision of virtual computing resources to client device 1514 without accessing a virtual environment. For example, user 1518 can log into cloud server computer 1502 from his or her home computer and access virtual computer 1510 to receive virtual computing resources accordingly. In another example, user 1518 can log into cloud server computer 1502 from his or her work computer to access virtual computer 1510 and receive virtual computing resources accordingly.

[0319] In some embodiments, at least a portion of the virtual computing resources are assigned to the client device through the management tool. Thus, the virtual infrastructure including all associated virtual computers can be managed using administrator options based on exclusive administrator privileges (e.g., using the virtual environment provided to the organization's IT team).

[0320] In some embodiments, the provision of virtual computing resources is performed based on a stored user profile. In an embodiment, the assignment of virtual computing resources is performed based on a stored user profile, the user profile comprising one or more parameters associated with and assigned to the user profile, the parameters comprising priority data, security data, QOS, bandwidth, storage space, or computing power, or a combination thereof. For example, a user accessing a work virtual computer from home may have a personal profile configured to provide the user with specific virtual computing resources associated with the profile.

[0321] In some embodiments, each virtual computer is a downloadable application available from an application library.

[0322] Fig.16 Depicted is a schematic diagram of a system 1600 that enables ad hoc virtual communications between graphical representations of users, according to an embodiment.

[0323] The system 1600 includes one or more cloud server computers 1602 including at least one processor 1604 and memory 1606 storing data and instructions implementing a virtual environment 1608. The virtual environment 1608 is configured to enable at least one approaching user graphical representation and at least one target user graphical representation in the virtual environment 1608 to open an ad hoc communication channel and enable an ad hoc conversation via the ad hoc communication channel between the user graphical representations within the virtual environment 1608. Fig.16 In the example shown, the system also includes two or more client devices 1610 that access at least one virtual environment through corresponding user graphical representations and are connected to one or more cloud server computers 1602 via a network 1612. The virtual environment 1608 enables at least one approaching user graphical representation 1614 and at least one target user graphical representation 1616 to open an ad hoc communication channel 1618 from a corresponding user 1620, enabling an ad hoc conversation between the user graphical representations within the virtual environment 1608.

[0324] In some embodiments, the opening of the ad hoc communication channel 1618 is performed based on the distance, position and orientation between the user graphical representations, or the current availability status, privacy settings, or state configuration of the ad hoc communication, or a combination thereof.

[0325] In some embodiments, the ad hoc conversation is performed at the place where the two user graphical representation areas are located in the virtual environment 1608. For example, if the user graphical representation 1614 approaching meets the target user graphical representation 1614 in a specific area of ​​a lounge or office space, the ad hoc communication can be opened so that the two users can keep the conversation in the specific area of ​​the lounge or office space without changing the position. In other embodiments, the current viewing angle in the virtual environment is used to perform the ad hoc conversation. In the above example, the ad hoc communication can be opened so that the two users can keep the conversation without changing the viewing angle. In other embodiments, the ad hoc conversation is realized in the virtual environment of the same or another connection where the ad hoc conversation occurs, and the viewing angle, position or a combination thereof are optionally changed.

[0326] In some embodiments, one or more cloud server computers are further configured to generate visual feedback in the virtual environment, signaling that ad hoc communication is possible. In an embodiment, the user graphical representation receives visual feedback, which signals that ad hoc communication is possible, thereby triggering the opening of an ad hoc communication channel, signaling the start of an ad hoc conversation between the user graphical representations.

[0327] In some embodiments, the ad hoc conversation involves sending and receiving real-time audio and video displayed from the user's graphical representation.

[0328] In some embodiments, the user corresponding to the approaching user graphical representation 1614 selects and clicks on the target user graphical representation 1616 before opening the ad hoc communication channel 1618 .

[0329] In some embodiments, one or more cloud server computers are further configured to open a self-organizing communication channel in response to an accepted invitation. For example, the user corresponding to the approaching user graphical representation 1614 further sends a self-organizing communication participation invitation to the target user graphical representation 1616, and opens a self-organizing communication channel 1618 after receiving the invitation approval from the target user graphical representation 1614.

[0330] In some embodiments, the ad hoc communication channel 1618 is implemented through at least one cloud server computer or as a P2P communication channel.

[0331] Fig.17 An embodiment of a method 1700 of implementing interactions in a virtual environment is depicted according to an embodiment.

[0332] The method 1700 for implementing interaction in a virtual environment according to the present disclosure begins in steps 1702 and 1704 , where a virtual environment platform including at least one virtual environment is provided in a memory of one or more cloud server computers including at least one processor.

[0333] As shown in steps 1706 and 1708, the method receives a live data feed from a user of the client device from at least one camera, and then generates a user graphical representation based on the live data feed. The method 1700 then inserts the user graphical representation into the three-dimensional coordinates of the virtual environment, as shown in step 1710.

[0334] Subsequently, at step 1712, the method updates the user graphical representation within the virtual environment based on the live data feed. Finally, at step 1714, the method processes the data generated from the interaction in at least one virtual environment through the corresponding graphical representation located within the virtual environment, and ends at step 1716.

[0335] Fig.18An embodiment of an image processing method 1800 according to an embodiment is depicted.

[0336] Method 1800 begins at steps 1802 and 1804 by providing data and instructions implementing image processing functions in a memory of at least one cloud server computer. Method 1800 continues at step 1806 by obtaining a live data feed from at least one camera of at least one user of at least one corresponding client device. Method 1800 then continues at step 1808 by providing one or more image processing combinations (e.g., Figures 6A-7C The image processing combination of the image processing of the image processing combination ... Figure 3 The hybrid system architecture 300) interacts.

[0337] Fig.19 Depicted is a method 1900 of user authentication based on a graphical representation of a user, according to an embodiment.

[0338] Method 1900 begins at steps 1902 and 1904, providing a user database storing user data associated with a user account and a corresponding user graphical representation in a memory of one or more cloud server computers, and a facial scanning and authentication module connected to the user database. Method 1900 continues in step 1906, receiving a request to access a user account from a client device, and then in step 1908, performing a facial scan on a user of at least one client device by a facial scanning and authentication module using an image received from at least one camera, which camera can be connected to at least one client device and / or one or more cloud server computers. In check 1910, method 1900 continues by checking for a match of user data associated with a user account in a user database. If the user account is available, method 1900 continues in step 1912, providing the user with a corresponding user graphical representation and access to the user account. In the negative case, if the user account is not available, method 1900 can continue in step 1914, generating a new user graphical representation and a new user account stored in the user database, and access to the user account based on the data. The process can end in step 1916.

[0339] Fig. 20 A block diagram of a method 2000 for virtually broadcasting from within a virtual environment is shown in accordance with an embodiment.

[0340] Method 2000 begins in step 2002 by providing data and instructions implementing a client device data exchange management module for managing data exchange between client devices in a memory of at least one media server. Method 2000 continues in step 2004 by capturing a multimedia stream by a virtual broadcast camera positioned within at least one virtual environment connected to the at least one media server.

[0341] In step 2006, method 2000 continues by sending a multimedia stream to at least one media server for broadcasting to at least one client device. In step 2008, method 2000 continues by obtaining, via at least one client device, live feed data from a user of at least one client device from at least one camera.

[0342] In step 2010, the method continues to perform data exchange management, including analyzing and optimizing incoming multimedia streams and live feed data from users within at least one virtual environment, and evaluating and optimizing forwarding of outbound multimedia streams. Finally, in step 2012, the method 2000 ends by broadcasting the corresponding multimedia streams to the client devices based on the data exchange management, wherein the multimedia streams are displayed to the user graphical representation of the user of the at least one client device.

[0343] Fig.21 A block diagram of a method 2100 for delivering an application within a virtual environment is shown according to an embodiment.

[0344] The method 2100 begins at step 2102, providing at least one virtual environment in the memory of at least one cloud server computer, and an application module including one or more installed applications and corresponding application rules for multi-user interaction, wherein the application module is linked to the virtual environment and visible within the virtual environment. In step 2104, the method 2100 continues by receiving a selection instruction from the virtual environment host. Then, in step 2106, the method 2100 continues by displaying and activating the one or more installed applications during a session of the virtual environment, so that a user graphical representation of the virtual environment host and any participant user graphical representation within the virtual environment can interact through a corresponding client device.

[0345] In step 2108, the method 2100 continues by receiving user interactions with one or more installed applications. Subsequently, the method 2100 continues, as shown in step 2110, managing and processing the user interactions with the one or more installed applications according to the application rules for multi-user interactions in the application module. Finally, the method 2100 ends in step 2112, and the processed interactions are forwarded to each client device accordingly to establish a multi-user session for implementing a shared experience according to the application rules.

[0346] Fig. 22 A block diagram of a method 2200 of providing virtual computing resources within a virtual environment according to an embodiment is shown.

[0347] The method 2200 begins at step 2202 by providing at least one virtual computer and a virtual environment including one or more graphical representations representing the virtual computer in a memory of at least one cloud server computer. The method continues at step 2204 by receiving virtual computing resources from at least one cloud server computer by the virtual computer. Then, at step 2206, the method continues by receiving a request for access to the one or more virtual computers from at least one client device. Finally, at step 2208, the method ends by providing a portion of the available virtual computing resources to at least one client device based on the needs of the client device.

[0348] Fig.23 A block diagram of a method 2300 for implementing ad hoc virtual communications between user graphical representations is shown.

[0349] The method 2300 begins in step 2302 by providing a virtual environment in a memory of one or more cloud server computers including at least one processor. Then, in step 2304, the method continues by detecting two or more client devices accessing at least one virtual environment through corresponding graphical representations, wherein the client devices are connected to the one or more cloud server computers via a network. Finally, in step 2306, the method 2300 ends by opening an ad hoc communication channel in response to at least one user graphical representation approaching another user graphical representation, enabling an ad hoc conversation between the user graphical representations in the virtual environment.

[0350] Also described are computer-readable media having instructions stored thereon, which are configured to cause one or more computers to perform any of the methods described herein. As used herein, the term "computer-readable media" includes volatile and nonvolatile, and removable and non-removable media implemented in any method or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data. In general, the functionality of the computing devices described herein can be implemented in computing logic embodied in hardware or software instructions, which can be written in a programming language such as C, C++, COBOL, JAVA, or the like. TM , PHP, Perl, Python, Ruby, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft.NET TMLanguages ​​such as C#, etc. The computing logic can be compiled into an executable program or written in an interpreted programming language. Typically, the functionality described herein can be implemented as logic modules, which can be replicated to provide greater processing power, merged with other modules, or divided into sub-modules. The computing logic can be stored in any type of computer-readable medium (e.g., non-transitory media, such as memory or storage media) or computer storage device, and can be stored on and executed by one or more general or special-purpose processors, thereby creating a special-purpose computing device configured to provide the functionality described herein.

[0351] Although specific embodiments have been described and shown in the drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to one of ordinary skill in the art. Accordingly, the description is to be regarded as illustrative and not restrictive.

Claims

1. A system for realizing virtual presence based on spatial video, characterized in that: Include: one or more server computers, at least one sending client device, and at least one receiving client device; wherein the one or more server computers include at least one processor and memory storing data and instructions, the data and instructions implementing a virtual environment platform including at least one virtual environment, the virtual environment configured to open a peer-to-peer communication channel between the at least one sending client device and the at least one receiving client device; wherein the at least one sending client device comprises at least one processor and a memory storing data and instructions, wherein the data and instructions are configured to, when executed by the at least one processor, cause the at least one sending client device to: A real-time 3D virtual cutout of the user is generated for transmission via the peer-to-peer communication channel by: Obtaining a 2D or 3D live video data feed from at least one camera; removing, by the at least one sending client device, a background on the live video data feed; as well as generating a polygonal structure for use as a virtual frame supporting the live video data feed with the background removed; as well as transmitting, by the at least one sending client device via the peer-to-peer communication channel, the user real-time 3D virtual cutout to the at least one receiving client device for presentation; as well as wherein the at least one receiving client device comprises at least one processor and a memory storing data and instructions, wherein the data and instructions are configured to, when executed by the at least one processor, cause the at least one receiving client device to: receiving, by the at least one receiving client device via the peer-to-peer communication channel, the user real-time 3D virtual cutout from the at least one sending client device; as well as inserting and combining a real-time 3D virtual cutout of the user into the virtual environment by presenting the live video data feed with the background removed having the polygonal structure within the virtual environment; as well as The virtual environment is shared by the at least one sending client device, the at least one receiving client device and corresponding users in real-time 3D virtual cutouts, so that a multi-user shared experience of virtual presence based on spatial video can be achieved within the virtual environment.

2. The system according to claim 1, characterized in that The data and instructions further cause the at least one sending client device to establish and automatically update a perspective of the virtual environment provided to the at least one sending client device using one or more virtual cameras virtually positioned and aligned relative to the user's real-time 3D virtual cutout.

3. The system according to claim 2, characterized in that The alignment of the one or more virtual cameras is in front of the user's real-time 3D virtual cutout and tracks the movement of the user's real-time 3D virtual cutout.

4. The system according to claim 3, characterized in that Wherein the orientation of the real-time 3D virtual cutout of the user is automatically updated by tracking and analyzing one or more of the user's eye and head tilt data and head rotation data using computer vision methods.

5. The system according to claim 3, characterized in that wherein the one or more virtual cameras point outward from eye level, or wherein the one or more virtual cameras include two virtual cameras, one for each eye, pointing outward from both eye levels, or wherein the one or more virtual cameras point outward from a center of a head position of a real-time 3D virtual cutout of the user, or wherein the one or more virtual cameras point outward from a center of a real-time 3D virtual cutout of the user.

6. The system according to claim 1, characterized in that Wherein the one or more server computers include a spatial analysis media server configured to manage, analyze and process incoming data for a plurality of transmitting client devices and, in such analysis, manage or optimize transmission of data streams to a plurality of receiving client devices.

7. The system according to claim 6, characterized in that Wherein the spatial analysis media server is further configured to: In response to detecting that a distance between two or more user real-time 3D virtual cutouts has changed: Calculating a distance difference between the two or more user real-time 3D virtual cutouts; updating at least one of a position and an orientation of each user's real-time 3D virtual cutout within the virtual environment; Real-time 3D virtual shearing updates the corresponding perspectives of each user; as well as The updated position, orientation, and corresponding viewing angle of each user's real-time 3D virtual cutout are sent to corresponding client devices.

8. The system according to claim 1, characterized in that The at least one sending client device is further configured to perform further processing or improvement on the user real-time 3D virtual cutout.

9. The system according to claim 1, characterized in that Wherein image segmentation is applied by one or more of instance segmentation or semantic segmentation, and background removal is performed by the sending client device or one or more server computers using a deep neural network.

10. The system according to claim 1, characterized in that wherein the instructions further cause the at least one sending client device to: The user 3D virtual cutout is generated by: Obtaining a user's photo; and Based on the photo, generating a 3D mesh or a 3D point cloud for the user 3D virtual cutout; as well as The user 3D virtual cutout or the user real-time 3D virtual cutout is selectively transmitted for presentation by the at least one receiving client device.

11. A method for realizing presence based on space video, characterized in that: Include: opening a peer-to-peer communication channel between at least one sending client device and at least one receiving client device; obtaining, by the at least one sending client device, a 2D or 3D live video data feed; The user's real-time 3D virtual cutout is generated by performing the following actions: removing, by the at least one sending client device, a background on the live video data feed; as well as generating a polygonal structure for use as a virtual frame supporting the live video data feed with the background removed; as well as transmitting the user real-time 3D virtual cutout to the at least one receiving client device via the peer-to-peer communication channel by the at least one sending client device, wherein the at least one receiving client device is configured to insert and combine the user real-time 3D virtual cutout into a virtual environment implemented by at least one server computer, for rendering the user real-time 3D virtual cutout within the virtual environment by the at least one receiving client device, The virtual environment is shared by the at least one sending client device and the at least one receiving client device and corresponding users in real-time 3D virtual cutouts, so that a multi-user shared experience of virtual presence can be achieved within the virtual environment.

12. The method according to claim 11, characterized in that Also included is establishing and automatically updating a perspective of the virtual environment provided to the at least one sending client device using one or more virtual cameras virtually positioned and aligned relative to the user's real-time 3D virtual cutout.

13. The method according to claim 12, characterized in that The alignment of the one or more virtual cameras is in front of the user's real-time 3D virtual cutout and tracks the movement of the user's real-time 3D virtual cutout.

14. The method according to claim 13, characterized in that Also included is updating the orientation of the user's real-time 3D virtual cutout by tracking and analyzing one or more of the user's eye and head tilt data and head rotation data using computer vision methods.

15. The method according to claim 14, characterized in that wherein the one or more virtual cameras point outward from eye level, or wherein the one or more virtual cameras include two virtual cameras, one for each eye, pointing outward from both eye levels, or wherein the one or more virtual cameras point outward from a center of a head position of a real-time 3D virtual cutout of the user, or wherein the one or more virtual cameras point outward from a center of a real-time 3D virtual cutout of the user.

16. The method according to claim 11, characterized in that Wherein the at least one server computer is a spatial analysis media server configured to manage, analyze and process incoming data for a plurality of transmitting client devices and in such analysis manage or optimize transmission of data streams to a plurality of receiving client devices.

17. The method according to claim 16, characterized in that Also includes: In response to detecting that a distance between two or more user real-time 3D virtual cutouts has changed, calculating, by the spatial analysis media server, a distance difference between the two or more user real-time 3D virtual cutouts; updating at least one of a position and an orientation of each user's real-time 3D virtual cutout within the virtual environment; Real-time 3D virtual cutouts for each user update their respective perspectives, and The updated position, orientation, and corresponding viewing angle of each user's real-time 3D virtual cutout are sent to corresponding client devices.

18. The method according to claim 11, characterized in that The method also includes performing further processing or improvement on the user real-time 3D virtual cutout by the at least one sending client device.

19. The method according to claim 11, characterized in that Wherein image segmentation is applied by one or more of instance segmentation or semantic segmentation, and background removal is performed by the sending client device or one or more server computers using a deep neural network.

20. A non-transitory computer readable medium having instructions stored thereon, characterized in that: The instructions are configured to cause a computer system including at least one sending client device including a processor and a memory to execute steps including: obtaining, by the at least one sending client device, a 2D or 3D live video data feed; The at least one sending client device generates a user real-time 3D virtual cutout by performing the following actions: removing, by the at least one sending client device, a background on the live video data feed; generating a polygonal structure for use as a virtual frame supporting the live video data feed with the background removed; as well as transmitting the user real-time 3D virtual cutout to at least one receiving client device via a peer-to-peer communication channel by the at least one sending client device, wherein the at least one receiving client device is configured to insert and combine the user real-time 3D virtual cutout into a virtual environment implemented by at least one server computer, for rendering the user real-time 3D virtual cutout within the virtual environment by the at least one receiving client device, The virtual environment is shared by the at least one sending client device and the at least one receiving client device and corresponding users in real-time 3D virtual cutouts, so that a multi-user shared experience of virtual presence can be achieved within the virtual environment.

Images