Interactive Video Streaming for 3D Applications

Abstract

A video processing system and method for generating 3D streaming data via an API is provided. After validating the client device, a server supporting the Encapsulated 3D Streaming Engine is selected to initiate a 3D streaming session. A library file is introduced into the Encapsulated 3D Streaming Engine executable to generate a digital representation of the interactive 3D environment. The interactive 3D environment is rendered, encoded, and streamed to the client device via a GPU. The 3D streaming session is coordinated through the client device and maintained until terminated on the client device. The number of virtual compute instances of the Encapsulated 3D Streaming Engine provided by the server is dynamically adjusted based on metrics and machine learning predictions of the connected client device.

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 377,986, filed September 30, 2022, the entire teachings of which are incorporated herein by reference. [Background technology]

[0002] Streaming video applications have recently taken on new dimensions, aiming to provide users with an even more immersive experience than ever before. As virtual alternatives to face-to-face interaction continue to develop, users are offered the opportunity to connect safely and reliably across vast geographical distances. Summary of the Invention [Problem to be solved by the invention]

[0003] Embodiments of the present disclosure provide techniques for streaming interactive video and audio content that is engaging to users. The streaming techniques provide users with a high-quality, low-latency video and audio streaming experience. Graphics processing unit (GPU) instances are configured to directly host services that support streaming content, facilitating implementation of the approach without middleware or plug-ins. Embodiments of the present invention relate to systems and methods for video processing and streaming processed media content to the user's client device. [Means for solving the problem]

[0004] In some embodiments, the video processing system includes a plurality of servers operating in a peer-to-peer communication network via a WebRTC application programming interface (API) and generating 3D streaming data. The plurality of servers may be responsive to a session handler receiving a request for a 3D streaming session at a first client device, the plurality of servers including, in part, receiving, from the session handler communicating with the first client device, context verification information for the first client device, the context verification information including an Internet Protocol (IP) location of the first client device. By allocating the first server from the plurality of servers, the plurality of servers may be responsive to the context verification information for the first client device and direct the 3D streaming session for or at the first client device based on the IP location of the first client device.

[0005] The servers can then launch or command a first instance of an encapsulated 3D streaming engine located on the first server to initiate the 3D streaming session with the first client device via the WebRTC API, and the servers can incorporate library files of the WebRTC API into an executable file of the encapsulated 3D streaming engine to generate a digital representation of an interactive 3D environment for the 3D streaming session to the first client device.

[0006] The plurality of servers can render the digital representation of the interactive 3D environment into a buffer of the GPU, encode the rendered representation and transmit the 3D streaming session to the first client device, and stream the encoded representation of the interactive 3D environment to the first client device. The plurality of servers can dynamically adjust the 3D streaming session of the interactive 3D environment in response to input received from the first client device.

[0007] The multiple servers can maintain the digital representation of the interactive 3D environment of the 3D streaming session by repeatedly performing the rendering, encoding, streaming, and adjustment processes described above until the 3D streaming session is completed at the first client device.

[0008] The first server may be configured to establish a predetermined number of virtual computing instances (VCIs) of the 3D streaming session on one or more client devices, and the first server may be configured to automatically adjust the number of VCIs provided based on a change in the number of the client devices, at least one of which may be the first client device.

[0009] At least one of the plurality of servers may be configured to respond to requests for the 3D streaming session from the client device by assigning a server from the plurality of servers to respond to each request for the 3D streaming session, and the first server may be one of the plurality of servers.

[0010] At least one of the plurality of servers may calculate an analysis to determine an appropriate number of 3D streaming sessions for the plurality of client devices scaled based on the predetermined number of the VCI. The calculated analysis may include concurrent users (CCU), session duration, daily active users (DAU), monthly active users (MAU), and / or sessions. The encapsulated 3D streaming engine may configure the 3D streaming sessions as a media streaming cluster, such as a video streaming cluster, and cause at least one of the plurality of servers to perform multicast transmission of packets including at least a portion of the digital representation of the interactive 3D environment to the one or more client devices.

[0011] Dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device may further include configuring the encapsulated 3D streaming engine to capture events related to the interactive 3D environment. The events may be detected by one or more listeners on the one or more client devices. The dynamic adjustment may further include configuring the encapsulated 3D streaming engine to capture metadata, consumption information, and interaction data during the 3D streaming session at the one or more client devices. The dynamic adjustment may further include configuring the encapsulated 3D streaming engine to re-render the interactive 3D environment at the one or more client devices in response to the captured events, metadata, consumption information, and interaction data to maintain the digital representation of the interactive 3D environment.

[0012] The Encapsulated 3D Streaming Engine may compute composite 3D media data, such as composite 3D video data, based on the captured events, metadata, consumption information, and interaction data, define a composite image layout based on properties derived from the 3D streaming session at the one or more client devices, configure the 3D streaming session to provide a composite video signal according to the defined composite image layout, and transmit the composite video signal in packets to the one or more client devices via a packetizer, thereby maintaining the digital representation of the interactive 3D environment. The Encapsulated 3D Streaming Engine may interface with the one or more client devices and facilitate control of a communication interface enabling audio, text, and video transmission between the one or more client devices in a packet-switched communication system of the peer-to-peer network.

[0013] The encapsulated 3D streaming engine may be configured as a software container including bindings of software components. The software components may include at least one of a library file of the WebRTC API, an event manager, a scene manager, a resource manager, a session manager, a physics manager, and an artificial intelligence system. The WebRTC API may include one or more WebRTC API functions configured to call one or more of the software components. The encapsulated 3D streaming engine may be other than a software plug-in.

[0014] The input may be received over a data channel established using an RTCDataChannel API. The encapsulated 3D streaming engine may be configured to train an interactive frame prediction model for the digital representation of the interactive 3D environment based on the encoded streaming data of the 3D streaming session and the events, metadata, interaction data, and consumption data from the one or more client devices.

[0015] The encapsulated 3D streaming engine may be configured to selectively use predicted frames generated based on the trained interactive frame prediction model to encode the 3D streaming session, and to transmit the trained interactive frame prediction model and the encoded streaming data to the first and second client devices to generate the digital representation of the interactive 3D environment. The first and second client devices may be configured to receive the trained interactive frame prediction model and the encoded streaming data and decode the encoded streaming data based on the trained interactive frame prediction model to generate the digital representation of the interactive 3D environment.

[0016] In some embodiments, the video processing method includes configuring a server computer system to respond to a session handler receiving a request for a 3D streaming session at a first client device. The response to the session handler communicating with the first client device may include receiving from the session handler context verification information for the first client device, the context verification information including an IP location of the first client device. The response to the session handler may include allocating the first server from the plurality of servers of the server computer system to respond to the context verification information for the first client device and directing the 3D streaming session for the first client device based on the IP location of the first client device.

[0017] Responding to the session handler may include launching a first instance of the encapsulated 3D streaming engine located on the first server and initiating the 3D streaming session with the first client device via the WebRTC API. Responding to the session handler may include installing library files of the WebRTC API into an executable file of the encapsulated 3D streaming engine and generating a digital representation of an interactive 3D environment for the 3D streaming session for the first client device.

[0018] Responding to the session handler may include rendering the digital representation of the interactive 3D environment into a buffer of a GPU, encoding the rendered representation and transmitting the 3D streaming session to the first client device, and streaming the encoded representation of the interactive 3D environment to the first client device. Responding to the session handler may include dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device.

[0019] The response to the session handler may include maintaining the digital representation of the interactive 3D environment of the 3D streaming session by iteratively performing the rendering, encoding, streaming, and adjustment processes described above until the 3D streaming session is terminated at the first client device. In an embodiment of a video processing method, the method performs operations to implement any embodiment or combination of embodiments described herein.

[0020] The foregoing will be apparent from a more particular description of example embodiments in the accompanying drawings, in which like reference characters refer to like parts throughout the different views, and in which the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the embodiments. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a schematic block diagram of an exemplary video processing system according to the present disclosure. [Figure 2] 1 is a flow diagram of an exemplary video processing method according to this disclosure. [Figure 3] 1 is a schematic block diagram of an exemplary embodiment of a video processing system; [Figure 4] 1 is a schematic block diagram of an exemplary embodiment of a video processing system; [Figure 5A] 1 is a schematic and overview diagram of an exemplary use case of an embodiment of a video processing system; [Figure 5B] 5B illustrates various exemplary user interfaces in an example use of the exemplary video processing system of FIG. 5A. [Figure 6] FIG. 1 is a schematic diagram of an exemplary computer network environment in which embodiments of the present invention may be deployed. [Figure 7] FIG. 7 is a block diagram of a computer node in the network of FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0022] A description of exemplary embodiments follows.

[0023] This document describes a high performance, low cost streaming solution that can be easily implemented as a complete solution even by users without programming skills, and provides high visual quality and low latency.

[0024] [System Architecture] 1 is a schematic block diagram of an exemplary embodiment of a video processing system 100. In the exemplary embodiment, the video processing system 100 includes multiple servers 105-1, 105-n operating in a peer-to-peer (P2P) communication network 110 via a WebRTC application programming interface (API) 115 to generate 3D streaming data 120. The multiple servers 105-1, 105-n, may be responsive to a session handler 125 receiving a request 130 for a 3D streaming session at a first client device 135-1, including, in part, receiving a context validation 140 for the first client device 135-1 from the session handler 125 in communication with the first client device 135-1, the context validation 140 including the Internet Protocol (IP) location of the first client device 135-1. The plurality of servers 105-1, 105-n may respond to the context verification 140 for the first client device 135-1, assign the first server 105-1 from the plurality of servers 105-1, 105-n, and direct a 3D streaming session 145 for the first client device 135-1 based on the IP location of the first client device 135-1.

[0025] As an example, the context validation 140 may include a distance metric based on the distance between the respective server 105-1, 105-n and the first client device 135. In one embodiment, the context validation 140 for the first client device 135 may include the number of network hops. Intermediate routers or servers require time to process signals transmitted from the respective server 105-1, 105-n to the first client device 134. Synthetic monitoring tools may be deployed from the respective server 105-1, 105-n to perform artificial calls to the first client 145 and detect increased latency or performance degradation. Based on the context validation data analysis, a server from the servers 105-1, 105-n that provides the lowest latency and highest throughput capacity for the client may be selected.

[0026] The plurality of servers 105-1, 105-n may then launch a first instance of an encapsulated 3D streaming engine 150 located on the first server 105-1 and initiate the 3D streaming session 145 with the first client device 135-1 via the WebRTC API 115. The plurality of servers 105-1, 105-n may then install library files 155 of the WebRTC API 115 into executable files 160 of the encapsulated 3D streaming engine 145 and generate a digital representation of an interactive 3D environment 165 for the 3D streaming session 145 for the first client device 135-1.

[0027] The plurality of servers 105-1, 105-n may render a digital representation of the interactive 3D environment 165 into a buffer of a graphics processing unit (GPU) 170, encode the rendered representation, send the 3D streaming session 145 to the first client device 135-1, and continue streaming the encoded representation of the interactive 3D environment to the first client device 135-1. The plurality of servers 105-1, 105-n may dynamically adjust the 3D streaming session 145 of the interactive 3D environment in response to input 175 received from the first client device 135-1.

[0028] The multiple servers 105-1, 105-n may maintain the digital representation of the interactive 3D environment 165 of the 3D streaming session 145 by repeatedly performing the rendering, encoding, streaming, and adjustment processes described above until the 3D streaming session 145 is terminated at the first client device 135-1.

[0029] 2 is a flow diagram of an example embodiment of a video processing method 200. In the example embodiment, the video processing method 200 includes configuring a server computer system, such as a first server 105-1 of the plurality of servers 105-1, 105-n of the system 100, to respond to a session handler 125 receiving a request 130 for a 3D streaming session at a first client device 135-1. In the method 200, the response to the session handler 125 communicating with the first client device 135-1 may include receiving 204 a context validation 140 for the first client device 135-1 from the session handler, the context validation 140 including the IP location of the first client device 135-1. In the method 200, the response to the session handler 125 may include responding 209 to the context verification 140 for the first client device 135-1 and directing a 3D streaming session 145 for the first client device 135-1 based on the IP location of the first client device 135-1 by allocating the first server 105-1 from the plurality of servers 105-1, 105-n of the server computer system.

[0030] In the method 200, responding to the session handler 125 may include launching 214 a first instance of an encapsulated 3D streaming engine 150 located on the first server 105-1 and initiating the 3D streaming session 145 with the first client device 135-1 via the WebRTC API 115. In the method 200, responding to the session handler 125 may include installing 219 a library file 155 of the WebRTC API into an executable file 160 of the encapsulated 3D streaming engine 150 and generating a digital representation of an interactive 3D environment 165 for the 3D streaming session 145 for the first client device 135-1.

[0031] In the method 200, responding to the session handler 125 may include rendering 224 a digital representation of the interactive 3D environment 165 into a buffer of the GPU 170, encoding 229 the rendered representation, sending the 3D streaming session 145 to the first client device 135-1, and streaming 234 the encoded representation of the interactive 3D environment to the first client device 135-1. In the method 200, responding to the session handler 125 may include dynamically adjusting 239 the 3D streaming session 145 of the interactive 3D environment in response to input 175 received from the first client device 135-1.

[0032] In the method 200, responding to the session handler 125 may include maintaining 244-1 the digital representation of the interactive 3D environment 165 for the 3D streaming session 145 by iteratively performing the rendering 224, encoding 229, streaming 234, and adjustment 239 processes described above until the 3D streaming session 145 is terminated 244-2 at the first client device 244. In response to the 3D streaming session 145 being terminated 224-2, the method 200 may include terminating 249 streaming of the digital representation of the interactive 3D environment 165.

[0033] 1, the first server 135-1 of the system 100 may be configured to establish a predetermined number of virtual computing instances (VCIs) 180-1, 180-j of the 3D streaming session 145 on one or more client devices 135-1, 135-2, 135-i, and the first server 105-1 may be configured to automatically adjust the number of VCIs 180-1, 180-j provided based on changes in the number of client devices 135-1, 135-2, 135-i. At least one of the one or more client devices 135-1, 135-2, 135-i may be the first client device 135-1.

[0034] In the exemplary system 100 of FIG. 1, a first client device 135-1, a second client device 135-2, and an i-th client device 135-i are shown, but it should be noted that there may be only the first client device 135-1, only the first client device 135-1 and the second client device 135-2, or some other number i of client devices 135-1, 135-2, 135-i. 1, while a first virtual computing instance (VCI 1) 180-1, a second virtual computing instance (VCI 2) 180-2, and j virtual computing instances (VCI j) are shown, it should be noted that there may be only the first virtual computing instance (VCI 1) 180-1, only the first virtual computing instance (VCI 1) 180-1 and the second virtual computing instance (VCI 2) 180-2, or any other number j of virtual computing instances (VCI 1, VCI 2, ..., VCI j) 180-1, 180-2, 180-j. It should also be noted that there may be any number of servers 105-1, 105-n, including one server 105-1, or any other number n of servers 105-1, 105-n.

[0035] At least one of the plurality of servers 105-1, 105-n may be configured to respond to each request 130 for a 3D streaming session 145 from the client device 135-1, 135-i by assigning one server from the plurality of servers 105-1, 105-n to respond to the request 130 for a 3D streaming session 145. The first server 105-1 may be one of the plurality of servers 105-1, 105-n.

[0036] At least one of the plurality of servers 105-1, 105-n may calculate an analysis to determine an appropriate number of 3D streaming sessions 145 for the plurality of client devices 135-1, 135-i scaled based on the predetermined number of the VCIs 180-1, 180-j. The calculated analysis may include, for example, the number of concurrent users (CCU), session time, daily active users (DAU), monthly active users (MAU), and sessions. The encapsulated 3D streaming engine 150 may configure the 3D streaming sessions 130 as a video streaming cluster and cause at least one of the plurality of servers 105-1, 105-n to perform multicast transmission of packets including at least a portion of the digital representation of the interactive 3D environment 165 to the one or more client devices 135-1, 135-i.

[0037] Dynamically adjusting the 3D streaming session 145 of the interactive 3D environment in response to input 175 received from the first client device 135-1 may further include configuring the encapsulated 3D streaming engine 150 to capture events related to the interactive 3D environment. The events may be detected by one or more listeners on the one or more client devices 135-1, 135-i. The dynamic adjustment may further include configuring the encapsulated 3D streaming engine 150 to capture metadata, consumption information, and interaction data during the 3D streaming session 145 at the one or more client devices 135-1, 135-i. The dynamic adjustment may further include configuring the encapsulated 3D streaming engine 175 to maintain the digital representation of the interactive 3D environment 165 by re-rendering the interactive 3D environment at the one or more client devices 135-1, 135-i in response to the captured events, metadata, consumption information, and interaction data.

[0038] The Encapsulated 3D Streaming Engine 150 may compute composite 3D video data based on the captured events, metadata, consumption information, and interaction data, define a composite image layout based on properties derived from the 3D streaming session 145 at the one or more client devices 135-1, 135-i, configure the 3D streaming session to provide a composite video signal according to the defined composite image layout, and transmit the composite video signal to the one or more client devices 135-1, 135-i via a packetizer, thereby maintaining the digital representation of the interactive 3D environment 165. The Encapsulated 3D Streaming Engine 150 may interface with the one or more client devices 135-1, 135-i and facilitate control of a communication interface enabling audio, text, and video transmission between the one or more client devices 135-1, 135-i in a packet-switched communication system of the peer-to-peer network 110.

[0039] The encapsulated 3D streaming engine 150 may be configured as a container that includes bindings of software components. The software components may include at least one of a library file 155 of the WebRTC API 115, an event manager, a scene manager, a resource manager, a session manager, a physics manager, and an artificial intelligence system. The WebRTC API 115 may include one or more WebRTC API functions configured to invoke one or more of the software components. The encapsulated 3D streaming engine 150 may be other than a software plug-in.

[0040] The Encapsulated 3D Streaming Engine 150 can help improve the scalability of HTTP streaming delivery and the scalability of artificial intelligence systems, which may include HTTP Dynamic Adaptive Streaming (MPEG-DASH) and deep learning video streaming architectures (deep neural networks, deep belief networks, deep reinforcement learning, recurrent neural networks, convolutional neural networks, etc.). The Encapsulated 3D Streaming Engine 150 in communication with the artificial intelligence systems can help configure the client devices 135-1, 135-I into self-learning HTTP adaptive streaming clients, such as State-action-reward-state-action (SARSA) and Q-learning (a model-free reinforcement learning algorithm).

[0041] The input 175 may be received over a data channel established using an RTCDataChannel API. The encapsulated 3D streaming engine 150 may be configured to train an interactive frame prediction model for the digital representation of the interactive 3D environment 165 based on the encoded streaming data of the 3D streaming session 145 and the events, metadata, interaction data, and consumption data from the one or more client devices 135-1, 135-i.

[0042] In an exemplary embodiment, streaming via the encapsulated 3D streaming engine 150 can be implemented as a service for high-end 3D applications with ultra-low latency streaming technology. Any cloud provider (e.g., Amazon Web Services (AWS), Google Cloud Platform (GCP), Azure, OnPremise, etc.) can provide the servers 105-1, 105-n for the systems and methods described herein. Furthermore, the encapsulated 3D streaming engine 150 can be integrated with one or more game engines, such as Unreal Engine or Unity. An encoder can include an NVIDIA nvenc or similar for encoding the 3D streaming session 145. The encoder can encode the 3D streaming session 145 using a codec, such as the h264 codec. The API 115 can be configured as a WebRTC API or another API to provide an ultra-low latency video connection between the client devices 135-1, 135-2, 135-i and the GPU instances 170 provided by the cloud-based servers 105-1, 105-n.

[0043] FIG. 3 is a schematic block diagram of an example embodiment of a video processing system, particularly showing a server 305 providing a GPU 370 to support a 3D streaming session 145. In this example, the encapsulated 3D streaming engine 150 is provided by a 3D game application 350, which may be based on DirectX. The library file 155 of the WebRTC API 115 may be injected by a DirectX injector 319. Upon receiving video data, including images, from the 3D game application 350, the video data may be rendered 319 into a buffer of the GPU 370. The rendered video data may be passed to a streaming server 315, which may be based on WebRTC and configured to encode the video data according to the NVIDIA nvenc codec. The WebRTC server 315 may be configured to control 310 an interface for interacting with the 3D game application 350. The encoded video data may be streamed 320 over the Internet by the WebRTC server 315.

[0044] The encapsulated 3D streaming engine 150 may include multiple execution instances, which may be distributed across multiple servers 16, 105-1, 105-n. The encapsulated 3D streaming engine 150 may be configured to selectively use predicted frames generated based on the trained interactive frame prediction model to encode the 3D streaming session 145, and to transmit the trained interactive frame prediction model and the encoded streaming data to the first and second client devices to generate the digital representation of the interactive 3D environment 165. The first and second client devices 135-1, 135-2 may be configured to receive the trained interactive frame prediction model and the encoded streaming data and decode the encoded streaming data based on the trained interactive frame prediction model to generate the digital representation of the interactive 3D environment 165.

[0045] In an exemplary embodiment, the encapsulated 3D streaming engine 150 may be configured to train an interactive frame prediction model. The interactive frame prediction model may be constructed and trained based on the streaming session, a 3D model, user input, metadata, and consumption data associated with the client session. The encapsulated 3D streaming engine 150 interfaces with the encoder and decoder to predict frames of the 3D stream using the trained interactive frame prediction model. For example, the encoder may be configured to encode the 3D streaming data and transmit the trained interactive frame prediction model. The decoder may receive the 3D streaming data, transmit the trained interactive frame prediction model, decode the encoded streaming data based on the trained interactive frame prediction model, and provide reconstructed 3D streaming data at the client 135-i, 135-1, 135-2.

[0046] In an exemplary embodiment, the encapsulated 3D streaming engine 150 may be configured with deep neural network (DNN)-based interactive frame prediction for video coding. For example, the interactive frame prediction may be trained and configured using the DNN, which may help improve video coding and decoding efficiency. The DNN may be implemented in both the encoder and decoder. The DNN may predict a current frame using a previously decoded frame and the trained interactive frame prediction model. This may be configured as a separate interactive frame prediction mode that allows optimization to compete with other prediction modes. This may be implemented to avoid transmitting motion vectors. In this manner, the DNN may train the interactive frame prediction model and implement unidirectional and bidirectional prediction, thereby providing significant coding efficiency improvements compared to High Efficiency Video Coding (H.265 and MPEG-H Part 2). DNN-based frame prediction.

[0047] In one embodiment, the Encapsulated 3D Streaming Engine 150 can be configured to generate and process a scene graph for transmission of the 3D packet stream. The scene graph can be generated using machine learning and machine vision computational predictive systems to provide a powerful scene graph generation (SGG). For example, the Encapsulated 3D Streaming Engine 150 can perform scene graph generation (SGG) that can be extended with a powerful semantic representation that defines a scene. The scene graph generation (SGG) includes automatically mapping images or videos into a semantic structure scene graph. This process can include encoding position data for detected objects and their respective relationships, and this process can be scaled using deep learning techniques to improve system performance.

[0048] The scene graph generation (SGG) model can be processed by the Encapsulated 3D Streaming Engine 150 to render 3D video from a visually grounded scene graph. A scene graph can be a structural representation that can capture detailed semantics by explicitly modeling objects. The semantic structure of the scene graph can be processed using perceptual statistics to compute a map indicating which regions of a video frame are important to the human eye. This process can incorporate inter-frame prediction and motion estimation for motion vectors to generate an encoded bitstream. Quality metrics for the motion vectors can be used to construct a vector map that can be used to encode the bitstream using metrics such as block variance and block luminance. Applying the vector map and the scene graph generation (SGG) model to 3D video coding in a model-based compression framework such as continuous block tracking (CBT) can improve the compression quality and bitstream throughput capacity of the Encapsulated 3D Streaming Engine 150.

[0049] The scene graph generation (SGG) may involve analyzing an image or a sequence of images to generate a structured representation of a visual scene. The scene graph may be encoded with objects and their respective relationships, as well as contextual data that surrounds and connects those visual relationships. In this way, machine learning predictions of objects and relationships are predicted and modeled based on their surrounding context.

[0050] In one embodiment, the Encapsulated 3D Streaming Engine 150 running on the 305 GPU server with a 3D gaming application may be configured to recognize objects and scenes in still images using mathematical models or statistical learning, and then evolve to perform motion recognition, target tracking, action recognition, etc. in streaming videos. Object or feature detectors can capture the shapes of objects and their locations and model object attributes in three-dimensional space, which helps further realize object detection, recognition, tracking, interaction, and prediction.

[0051] As one example, the 3D streaming engine 150 may be configured for real-time 3D rendering capabilities that enable streaming of 3D "holographic" content, which in one embodiment includes point clouds, multi-view, streaming scene graphs (object-oriented representations of 3D video content), and dynamically animated meshes with texture streaming.

[0052] 4 is a schematic block diagram of an exemplary embodiment of a video processing system 400. In this example, a user 435 (e.g., via the first client device 135-1) requests 130 a 3D streaming session 145 using a library file 145, such as a Javascript library 455. Session management (i.e., session handler 125) may manage a connection to a streaming service 405 supported by a server, such as the first server 105-1. The Javascript library 455, or another library, may be used to handle input controls, such as mouse, touch, or key controls. The Javascript library 455 may define specifications specific to the current session. A publicDomainName parameter 418 may be used to connect to a WebSocket server 437 to communicate with the streaming service 405 and an associated encapsulating 3D streaming engine 150, such as an Unreal application 450. The connection may be secured by Secure Sockets Layer (SSL), Transport Layer Security (TLS), or similar encryption. The WebRTC API 115 may be configured as a proxy 417, routing incoming messages to a WebSocket server 437 and providing the aforementioned encryption. A turnserver 406 may interface between the proxy 417 and the streaming service 405. Additional mouse, touch, and keyboard controls may be handled between the proxy 417 and the streaming service 405. The WebSocket server 437 may receive incoming messages from a user 435 at a front-end interface, such as a website 436. The messages may include an identifier for the listening port of the WebSocket server 437. Incoming messages arriving at the WebSocket server 437 and outgoing messages issuing from the WebSocket server 437 may each use a separate communication path. The communication path may be manually selected and configured. Multiple sessions of the system 400 may be independently configured.Thus, GPU instance 370 can host services independently, without requiring middleware or additional proxies to be configured between services.

[0053] [Example Streaming Use Cases] 5A is a schematic diagram and overview of an exemplary use case 500a of an embodiment of a video processing system, such as system 100. A sales representative 535-1 can connect with a remotely located client 535-2 through a cloud-based server implementation 505 of system 100 to conduct a virtual guided sales tour. Benefits of such connectivity include enabling simulated face-to-face interactions even when a point of sale is closed or inaccessible to either client 535-2 or sales representative 535-1. The implementation 505 can host multiple clients on a single instance of the implementation 505. The implementation 505 can provide video and chat capabilities. Additionally, the implementation 505 provides a controlled environment for distributing content related to the products offered by sales representative 535-1, controlling the costs of providing and maintaining the environment.

[0054] Website-based streaming according to the present disclosure can provide a high-end 3D streaming experience on a public website for multiple users. This can provide a streaming solution for points of sale, such as dealerships. Access to the streaming solution can be restricted to specific users, improving platform security. Instead of purchasing and maintaining hardware such as dedicated streaming servers, points of sale can use the cloud streaming solution described herein, which saves resources and potentially increases revenue by applying them to other areas, such as improving customer experience. Use cases can be provided on a global scale to facilitate high-quality connections between users and GPU cloud instances.

[0055] In an exemplary embodiment, a digital sales lounge for an automobile dealership is provided. The dealership can initiate a streaming session and act as a moderator by inviting multiple clients (e.g., up to six clients or more) to individual streaming sessions. All participants in the streaming session, regardless of physical location, can view a common, albeit virtual, video presentation.

[0056] Figure 5B is a diagram 500b illustrating various example frames of a video presentation configured according to use case 500a of Figure 5A. Diagram 500b shows a streaming session being initiated 578 with an access code shared with client 535-2, client 535-2 using the access code to access 583 the streaming session, sales representative 535-1 and client 535-2 launching a collaboration application 588 to collaborate on configuring various options for a vehicle being offered for sale, and the streaming session ending 593 with custom video or images rendered for a personalized brochure according to the options selected in collaboration application 588.

[0057] [Challenges in existing real-time streaming applications] Other real-time streaming applications existing at the time of this disclosure require a high degree of technical expertise to develop a streaming solution or experience. Traditionally, establishing the streaming environment has required a highly manual process. Traditional approaches generally have difficulty creating a reliable and scalable solution at a global level. Furthermore, traditional approaches generally cannot provide sufficiently high visual quality and sufficiently low latency. Traditional approaches using Pixelstreaming require users to use the Unreal Engine as an encapsulated 3D streaming engine. Pixelstreaming offers users a limited feature set. Traditional approaches require multiple service providers to establish a streaming service. As a result, streaming using traditional approaches is expensive, and calculating actual costs can be a complex process. Existing cloud-based streaming solutions are plagued by complex IT environments within the companies that adopt them, making them difficult to set up. The aforementioned challenges result in an undesirably long time to market for streamed real-time solutions.

[0058] Aspects of the Real-Time Streaming Solution Disclosed Herein The embodiments disclosed herein provide fully developed streaming technology available from online marketplaces such as Google Marketplace. The streaming solution can be set up through a simple, highly automated process without requiring development expertise from the user. As such, the streaming solution can be tailored for content creators rather than software developers. Streaming can be effectively decoupled from the associated application engine, eliminating technical lock-in with respect to the specific application engine used. As such, a complete streaming solution can be provided, similar to a one-stop shop.

[0059] Furthermore, the streaming solution disclosed herein supports a variety of advanced technologies and provides a high-quality, high-performance streaming experience. The implementation can be used worldwide without restrictions. The setup is flexible and can meet diverse user requirements. The implementation uses configurations such as Software as a Service (SaaS) as blueprints, making it compatible with most corporate IT guidelines. The streaming solution also has an affordable and straightforward pricing structure.

[0060] [Digital processing environment] An exemplary implementation of a multimedia system 600 for streaming selected media content to user client devices 15 (e.g., client devices 135-1, 135-2, 135-i) may be implemented in a software, firmware, or hardware environment. FIG. 6 illustrates one example of such an environment. One or more client devices 15 (e.g., mobile phones) and a cloud 16 (or server computer or cluster thereof) provide processing, storage, and input / output devices for running application programs, etc. In this specification, client devices may be referred to interchangeably as client computers.

[0061] Client device 15 is connected to other computing devices, including other client devices / processes 15 and server computers 16, via communications network 17. Communications network 17 can be part of a remote access network, a global network (e.g., the Internet), an out-of-band network, a worldwide collection of computers, a local or wide area network, a cloud network, and gateways that currently communicate with each other using their respective protocols (TCP / IP, HTTP, Bluetooth, etc.). Other electronic device / computer network architectures are also suitable.

[0062] Server computers 16 (e.g., servers 105-1, 105-n) may be configured to implement a streaming media server for provisioning, formatting, and storing selected media content (audio, video, text, images / photos, etc.) for processing and playback on client devices 15. The server computers 16 are communicatively connected to client devices 15 that implement respective video encoders for capturing, encoding, loading, or providing selected media content transmitted to the server computers 16. In one exemplary embodiment, one or more of the server computers 16 are scalable Java application servers capable of handling increased loads, even in the event of traffic spikes.

[0063] FIG. 7 is a diagram illustrating the internal structure of a computer / computing node (e.g., client processor / device / mobile phone device / tablet 15, 135-i, 135-1, 135-2 or server computer 16, 105-1, 105-n) in the processing environment of FIG. 6, which may be used to facilitate the display of such audio, video, image, or data signal information. Each computer 15, 135-i, 135-1, 135-2, 16, 105-1, 105-n includes a system bus 11. A bus is a set of real or virtual hardware lines used to transfer data between components of a computer or processing system. The bus 11 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input / output ports, etc.) and enables data transfer between the elements. Attached to system bus 11 is an I / O device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, touch screen interface, display, printer, speakers, etc.) to computers 15, 16. A network interface 86 connects the computer to various other devices connected to a network (e.g., the network shown at 17 in FIG. 6). Memory 24 provides volatile storage of computer software instructions 25 and data 26 used in the software implementation of the present invention (e.g., capturing / loading, provisioning, formatting, acquiring, downloading, and / or saving selected streams of media content and streams of user-initiated commands).

[0064] Disk storage 95 provides non-volatile storage of computer software instructions 92 (equivalent to an "OS program") and data 94 used to implement an embodiment of the multimedia system 600 of the present invention. A central processing unit (CPU) 84 is also connected to system bus 11 and provides execution of computer instructions. The processor 84 may include one or more microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), programmable logic devices, state machines, gated logic, or discrete hardware circuits, and may handle load balancing for 3D streaming.

[0065] In one embodiment, the CPU 84 is a hybrid CPU / graphics processing unit (GPU) incorporating a Vector Packet Processor (VPP)-based hardware accelerator. The VPP-based hybrid CPU / GPU is optimized for improved packet processing of multiple 3D streaming sessions, including full H.264 decoding for multiple channels. In an exemplary embodiment, the Vector Packet Processor (VPP)-based hardware accelerator is configured to process hyperscale cloud workloads of 3D streaming sessions, including 5G transport processing, 5G RAN Intelligent Controller (RIC), and edge inference. In one preferred embodiment, the Vector Packet Processor (VPP)-based hardware accelerator includes an integrated 1-terabit switch, true inline streaming, and highly programmable 3D packet processing. The VPP accelerator can provide low latency and high throughput capacity, making it suitable for deploying multiple high-speed 3D streaming sessions.

[0066] In one embodiment, the hybrid CPU / graphics processing unit (GPU) includes an embedded hardware / firmware implementation of the 3D streaming engine 150 and is capable of generating multiple executable instances of the encapsulated 3D streaming engine 150. Each encapsulated execution instance of the 3D streaming engine 150 is preferably within a container. In this manner, the generated instance of the 3D streaming engine 150 and its respective libraries and dependencies run within a lightweight executable container, allowing for consistent execution and optimization as a portable computing unit that can be deployed more quickly and safely. Unlike traditional computing methods, running the 3D streaming engine 150 within a container makes it more resistant to bugs and errors when transferred to a new location. Containerization eliminates this issue by combining application code with the associated configuration files, libraries, and dependencies required for execution. This allows the container to potentially be abstracted from the host system, making it portable and capable of running on any platform or cloud without issues. In one embodiment, the encapsulated 3D streaming engine 150 may be implemented as a virtual machine running from a server 16, 105-1, 105-n, or via a Secure Sockets Layer deployed on a client device 135-i, 135-1, 135-2.

[0067] In one embodiment, the processor routines 92 and data 94 of the multimedia processing system (video processing system) may be implemented as a computer program product including a computer-readable medium storable on a storage device 95 or deployable as Software as a Service (SaaS), providing at least a portion of the software instructions for the multimedia processing system 600, the encapsulated 3D streaming engine 150, the GPU server 305 with 3D game application context validation 140, and the session handler 125. An instance of a software embodiment of the multimedia processing system 600 may be implemented as a computer program product 92 and installed by any suitable software installation procedure familiar to those skilled in the art. In another embodiment, at least a portion of the instructions for the multimedia processing system 600 may be downloaded via a cable, communication, and / or wireless connection. In other embodiments, the software components of the multimedia processing system 600 may be implemented as a computer program propagated signal product 77 embodied in a propagation medium (e.g., radio waves, infrared waves, laser waves, sound waves, or electrical waves propagated over the Internet or other global network). The carrier wave medium or signal provides at least a portion of the software instructions of the routines / programs 92 of the multimedia processing system 600 .

[0068] In alternative embodiments, the propagated signal is an analog carrier wave or a digital signal carried on a propagation medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a communications network, an out-of-band network, or other network. In one embodiment, the propagated signal is transmitted over a propagation medium over a period of time (e.g., instructions of a software application transmitted in packets over a network over milliseconds, seconds, minutes, or longer). In another embodiment, the computer-readable medium of the computer program product 92 is a propagation medium that can be received and read by the computer system 15, e.g., to receive the propagation medium and identify the propagated signal embodied in the propagation medium, as described above for computer program propagated signal products.

[0069] The multimedia processing system 600 described herein can be configured using any known programming language, including high-level, object-oriented programming languages. The client computer / device 15 of the multimedia system 600 can be implemented via a software implementation and can operate within a browser session. The multimedia processing system 600 can be developed using HTML, JavaScript, Flash, etc. The HTML code can be configured to embed the system into the client 15's web browsing session. The JavaScript can be configured to perform clickstream and session tracking on the client 15 and cache streaming media recording and editing data. In another embodiment, the system can be implemented in HTML5 for client devices 15 that do not have Flash installed and uses the HTTP Live Streaming (HLS) or MPEG-DASH protocols. The system can be implemented to transmit media streams using real-time streaming protocols, such as Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), and Web Real-Time Communications (WebRTC). Components of the multimedia processing system 600 may be configured to create and load XML, JSON, or CSV data files, or other structured metadata files (e.g., manifest files), containing information about where and how components of the multimedia processing system 600 are stored, hosted, or formatted (e.g., timing information, size, footnotes, attachments, interactive components, style sheets, etc.).

[0070] In an exemplary mobile implementation, the user interface framework of the components of the multimedia processing system 600 may be based on XHP, Javelin, and WURFL. In another exemplary mobile implementation for the OS X and iOS operating systems and their APIs, Cocoa and Cocoa Touch may be implemented using Objective-C or other high-level programming languages ​​that add Smalltalk-style messaging to the C programming language.

[0071] [Illustrative Benefits] The advantages of the streaming solution disclosed herein for users include: Embodiments offer low-cost streaming for users (e.g., 70% cheaper than some current offerings) and no additional development costs on the client side. The implementation provides a very fast and automated process, allowing for a rapid turnaround time for projects created with the implementation. Furthermore, the streaming solution is not restricted by IT regulations due to the SaaS approach used. Embodiments offer high-performance streaming with high visual quality and low latency. The solution avoids technology lock-in and is therefore open to future developments in the field of real-time engines and streaming. Cost control can be provided through automatic scaling of virtual compute instances based on actual demand. For example, thresholds can be used to ensure that a minimum number of virtual compute instances are always available. Embodiments offer full transparency through individual dashboards for each client, which can display information such as the configuration or the number of available runtime instances. The complete solution is provided as a service from one provider.

[0072] The high-fidelity streaming solution may be configured to support ray tracing. 3D applications may be streamed according to the methods and systems disclosed herein without additional software plug-ins. External interfaces may be used to control the streaming application. Implementations are cloud provider and device agnostic and may work with many such providers and user device types. Embodiments may include built-in maintenance and analysis tools.

[0073] Any game engine or streaming engine can be supported. Modern web browsers such as Chrome, Safari, and Firefox can be used in various implementations. Multiple applications can be hosted on a single graphics card or GPU. External protocols such as HTTP and WebSocket can be supported. The implementation enables analytics reporting such as session time, concurrent users (CCU), and other analytics. The implementation may include an automatic shutdown feature. The streaming solution enables cross-region support. Website integration can be achieved through an included Javascript library. Existing streaming applications can easily migrate to a platform configured according to the approach disclosed herein. Security can be provided globally, with source files and streaming applications protected on cloud servers. Automatic scaling of virtual compute instances based on the number of connected users allows, for example, more than 1,000 users to connect to the service. All common application features, such as animation and moving products, can be supported.

[0074] While exemplary embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the scope of the embodiments encompassed by the appended claims.

Claims

1. A video processing system, It includes multiple servers that operate over a peer-to-peer communication network via the WebRTC Application Programming Interface (API) to generate 3D streaming data. When the aforementioned multiple servers respond to the session handler receiving a request for a 3D streaming session on the first client device, (i) Receiving context validation information about the first client device, including the Internet Protocol (IP) location of the first client device, from the session handler that communicates with the first client device. (ii) Assigning a first server from the plurality of servers and, in response to the context verification information relating to the first client device, instructing a 3D streaming session on the first client device based on the Internet Protocol (IP) location of the first client device, (iii) Instructing the first instance of the encapsulated 3D streaming engine in the first server to start the 3D streaming session on the first client device via the WebRTC application programming interface (API), (iv) Importing the WebRTC Application Programming Interface (API) library files into the executable file of the encapsulated 3D streaming engine to generate a digital representation of the interactive 3D environment for the 3D streaming session on the first client device. (v) Rendering the digital representation of the interactive 3D environment into a buffer of a graphics processing unit (GPU), (vi) Encoding the rendered representation by the encoder and transmitting the 3D streaming session to the first client device. (vii) Streaming the encoded representation of the interactive 3D environment on the first client device, (viiii) Dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device, and (ix) Maintain the digital representation of the interactive 3D environment of the 3D streaming session by repeating the processes (v) through (viiii) until the 3D streaming session ends on the first client device. It responds by executing Video processing system.

2. The first server is configured to establish a predetermined number of virtual computing instances of the 3D streaming session for one or more client devices. The first server is configured to automatically adjust the number of virtual computing instances provided based on changes in the number of client devices. At least one of the one or more client devices is the first client device. The video processing system according to claim 1.

3. At least one of the plurality of servers is configured to respond to requests for 3D streaming sessions from the client device by assigning one server from the plurality of servers to respond to each request for a 3D streaming session. The aforementioned first server is one of the plurality of servers, The video processing system according to claim 1 or 2.

4. At least one of the plurality of servers computes an analysis to determine an appropriate number of 3D streaming sessions for the plurality of client devices scaled based on a predetermined number of virtual computing instances. The video processing system according to claim 1.

5. The calculated analysis includes concurrent users (CCU), session duration, daily active users (DAU), monthly active users (MAU), and sessions. The video processing system according to claim 1.

6. The encapsulated 3D streaming engine configures the 3D streaming session as a media streaming cluster that causes at least one of the plurality of servers to multicast packets containing at least a portion of the digital representation of the interactive 3D environment to one or more client devices. The video processing system according to claim 1.

7. Dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device is: The encapsulated 3D streaming engine is configured to capture events related to the interactive 3D environment, and these events are detected by one or more listeners on one or more client devices. The encapsulated 3D streaming engine is configured to capture metadata, consumption information, and interaction data during the 3D streaming session on one or more client devices. The encapsulated 3D streaming engine is configured to maintain the digital representation of the interactive 3D environment in response to captured events, metadata, consumption information, and interaction data by redrawing the interactive 3D environment on one or more client devices. The video processing system according to claim 1, further comprising:

8. The encapsulated 3D streaming engine provides the digital representation of the interactive 3D environment. Composite 3D media data is calculated based on the captured events, metadata, consumption information, and interaction data. Defining a composite image layout based on properties derived from the 3D streaming session on one or more client devices, Configuring the 3D streaming session to provide a composite media signal according to the defined composite image layout, and Transmitting the composite media signal in packets to one or more client devices via a packetizer. The video processing system according to claim 1, maintained by

9. The encapsulated 3D streaming engine further includes facilitating control of a communication interface that connects to one or more client devices via an interface, enabling voice, text, and video transmission between the one or more client devices in the peer-to-peer network packet-switched communication system. The video processing system according to claim 1.

10. The aforementioned encapsulated 3D streaming engine requires configuration files and libraries for execution. It is configured as a software container that includes bindings of software components, including dependencies, The video processing system according to claim 1.

11. The software component includes at least one of the following: the library files of the WebRTC application programming interface (API), an event manager, a scene manager, a resource manager, a session manager, a physical manager, and an artificial intelligence system. The WebRTC application programming interface (API) includes one or more WebRTC application programming interface (API) functions configured to call one or more of the software components. The video processing system according to claim 1.

12. The aforementioned encapsulated 3D streaming engine is not a software plugin. The video processing system according to claim 1.

13. The encapsulated 3D streaming engine is deployed as a virtual machine via Secure Sockets Layer (SSL), and the virtual machine is configured within the software container. The video processing system according to claim 1.

14. The aforementioned input is received via a data channel established using the RTC Data Channel Application Programming Interface (API). The video processing system according to claim 1.

15. The encapsulated 3D streaming engine is configured to train an interactive frame prediction model for the digital representation of the interactive 3D environment based on the encoded streaming data of the 3D streaming session and the events, metadata, interaction data, and consumption data from one or more client devices. The video processing system according to claim 1.

16. The encoder is configured to selectively use predicted frames generated based on the trained interactive frame prediction model to encode the 3D streaming session, and to transmit the trained interactive frame prediction model and the encoded streaming data to the first and second client devices to create the digital representation of the interactive 3D environment. The first and second client devices are configured to receive the trained interactive frame prediction model and the encoded streaming data, and to decode the encoded streaming data based on the trained interactive frame prediction model to create the digital representation of the interactive 3D environment. The video processing system according to claim 1.

17. A video processing method, This includes configuring a server computer system, and when a session handler receives a request for a 3D streaming session on a first client device, (i) Receiving context validation information about the first client device, including the Internet Protocol (IP) location of the first client device, from the session handler that communicates with the first client device. (ii) Assigning a first server from among multiple servers of the server computer system and, in response to the context verification information relating to the first client device, instructing a 3D streaming session on the first client device based on the Internet Protocol (IP) location of the first client device, (iii) Instructing the first instance of the encapsulated 3D streaming engine in the first server to start the 3D streaming session on the first client device via the WebRTC application programming interface (API), (iv) Importing the WebRTC Application Programming Interface (API) library files into the executable file of the encapsulated 3D streaming engine to generate a digital representation of the interactive 3D environment for the 3D streaming session on the first client device, (v) Rendering the digital representation of the interactive 3D environment into a buffer of a graphics processing unit (GPU), (vi) Encoding the representation rendered via the encoder and transmitting the 3D streaming session to the first client device. (vii) Streaming the encoded representation of the interactive 3D environment on the first client device, (viiii) Dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device, and (ix) Maintain the digital representation of the interactive 3D environment of the 3D streaming session by repeating the processes (v) through (viiii) until the 3D streaming session ends on the first client device. It responds by executing Video processing methods.

18. The first server is configured to establish a predetermined number of virtual computing instances of the 3D streaming session for one or more client devices, The first server is further configured to automatically adjust the number of virtual computing instances provided based on a change in the number of client devices, At least one of the one or more client devices is the first client device. The video processing method according to claim 17.

19. The further includes configuring at least one of the plurality of servers to respond to requests for 3D streaming sessions from the client device by assigning one server from the plurality of servers to respond to each request for a 3D streaming session, The aforementioned first server is one of the plurality of servers, The video processing method according to claim 17 or 18.

20. The further includes configuring at least one of the plurality of servers to compute an analysis to determine an appropriate number of 3D streaming sessions for the plurality of client devices scaled based on a predetermined number of virtual computing instances, The video processing method according to claim 17.

21. The calculated analysis further includes configuring it to include concurrent users (CCU), session duration, daily active users (DAU), monthly active users (MAU), and sessions. The video processing method according to claim 17.

22. The 3D streaming session is further configured as a media streaming cluster that causes at least one of the plurality of servers to multicast packets containing at least a portion of the digital representation of the interactive 3D environment to one or more client devices. The video processing method according to claim 17.

23. Dynamically adjusting the 3D streaming session of the interactive 3D environment in response to input received from the first client device is: The encapsulated 3D streaming engine is configured to capture events related to the interactive 3D environment, wherein the events are detected by one or more listeners on one or more client devices. The encapsulated 3D streaming engine is configured to capture metadata, consumption information, and interaction data during the 3D streaming session on one or more client devices. The encapsulated 3D streaming engine is configured to maintain the digital representation of the interactive 3D environment in response to the captured events, metadata, consumption information, and interaction data by redrawing the interactive 3D environment on one or more client devices. The video processing method according to claim 17, further comprising:

24. The encapsulated 3D streaming engine provides the digital representation of the interactive 3D environment. Composite 3D media data is calculated based on the captured events, metadata, consumption information, and interaction data. Defining a composite image layout based on properties derived from the 3D streaming session on one or more client devices, Configuring the 3D streaming session to provide a composite media signal according to the defined composite image layout, and Transmitting the composite media signal in packets to one or more client devices via a packetizer. The video processing method according to claim 17, maintained by

25. The encapsulated 3D streaming engine is further connected to one or more client devices via an interface to facilitate control of the communication interface that enables voice, text, and video transmission between the one or more client devices in the peer-to-peer network packet-switched communication system. The video processing method according to claim 17.

26. The encapsulated 3D streaming engine is further configured as a container including bindings for software components. The video processing method according to claim 17.

27. The software component includes at least one of the following: the library files of the WebRTC application programming interface (API), an event manager, a scene manager, a resource manager, a session manager, a physical manager, and an artificial intelligence system. The WebRTC application programming interface (API) includes one or more WebRTC application programming interface (API) functions configured to call one or more of the software components. The video processing method according to claim 17.

28. The aforementioned encapsulated 3D streaming engine is not a software plugin. The video processing method according to claim 17.

29. The encapsulated 3D streaming engine is further deployed as a virtual machine via Secure Sockets Layer (SSL), the virtual machine being configured within the software container. The video processing method according to claim 17.

30. The further includes receiving the input via a data channel established using the RTC Data Channel Application Programming Interface (API), The video processing method according to claim 17.

31. The encapsulated 3D streaming engine is further configured to train an interactive frame prediction model for the digital representation of the interactive 3D environment based on encoded streaming data of the 3D streaming session, and the events, metadata, interaction data, and consumption data from one or more client devices. The video processing method according to claim 17.

32. The encoder is configured to selectively use predicted frames generated based on the trained interactive frame prediction model to encode the 3D streaming session, and to transmit the trained interactive frame prediction model and the encoded streaming data to the first and second client devices to create the digital representation of the interactive 3D environment. The first and second client devices are configured to receive the trained interactive frame prediction model and the encoded streaming data, and to decode the encoded streaming data based on the trained interactive frame prediction model to create the digital representation of the interactive 3D environment. The video processing method according to claim 17.

33. One or more of the aforementioned servers include a hybrid CPU / graphics processing unit (GPU), the hybrid CPU / graphics processing unit (GPU) having an embedded encapsulated 3D streaming engine with a vector packet processor (VPP)-based hardware accelerator, the hardware accelerator being optimized to improve packet processing for multiple 3D streaming sessions, including full H.264 decoding of multiple channels. The video processing system according to claim 1.

34. One or more of the aforementioned servers include a hybrid CPU / graphics processing unit (GPU), the hybrid CPU / graphics processing unit (GPU) having an embedded encapsulated 3D streaming engine with a vector packet processor (VPP)-based hardware accelerator, the hardware accelerator being optimized to improve packet processing for multiple 3D streaming sessions, including full H.264 decoding of multiple channels. The video processing method according to claim 17.

Images