System and method for predicting states using a distributed game engine
The distributed game engine uses deep learning to predict and correct game states, addressing latency issues in video games by early processing and resource distribution, ensuring efficient and high-quality rendering.
Patent Information
- Application Number
- JP2023109581
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-03-15
- Filing Date
- 2023-07-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-02-19
AI Technical Summary
Modern video games accessed through computer networks experience increased latency due to the growing amount of generated information, making it difficult to separate and process components like virtual objects or characters effectively within game scenes.
A distributed game engine utilizing deep learning to predict game states, allowing for early processing of functions and course correcting errors, thereby reducing latency and distributing work across computing resources.
Reduces latency in game execution by predicting and processing game states ahead of user input, enabling timely and high-quality rendering of game scenes with reduced computational overhead.
Smart Images

Figure 0007590501000001 
Figure 0007590501000002 
Figure 0007590501000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a system and method for predicting states using a distributed game engine. [Background technology]
[0002] Modern video games are accessed through computer networks. For example, the Fortnite™ game is played by many players in different parts of the world. One player controls a first avatar and another player controls a second avatar. Each avatar collects weapons and cuts wood during the game. The avatars are then forced to stay inside a virtual circle. If an avatar is left outside the virtual circle, it dies virtually in the game. When both avatars are inside the circle, they find each other and fight each other with their weapons. Only one of the two avatars survives.
[0003] Because video games are accessed over computer networks, a lot of information related to the video games is generated, and the increased amount of information increases the latency of the execution of the video games. Summary of the Invention
[0004] Embodiments of the present disclosure provide a system and method for predicting states using a distributed game engine to reduce latency in running applications on a game cloud.
[0005] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the embodiments described in the present disclosure.
[0006] The systems and methods described herein help reduce latency for processing of games and / or help distribute work in a distributed game engine through the use of deep learning, a technique used to approximate functions that would otherwise be difficult to compute using an algorithm. The function is difficult to compute because the algorithm is unknown or the computation using the algorithm is complex.
[0007] Typically, many games run successfully, and the algorithms of these games are commonly known and automated. However, in the games described herein, due to the relationships between many states when processing a frame of a game scene, it may be difficult to separate components such as virtual objects or virtual characters in the game from each other. For example, a user input is used to manipulate a virtual character in the game. The action of that virtual character may cause an artificially intelligent non-playable character (AI NPC) in the game to behave in a certain way and trigger effects according to the laws of physics. To explain, upon receiving a user input, an explosive may be fired, an AI NPC may move, or a virtual object may move. Changes in the AI NPC, the laws of physics, and the user's viewpoint contribute to changes in the rendering of the game scene. This illustrates the interdependencies of various functions within the game engine.
[0008] Neural networks can be used to predict the output of various functions, allowing subsequent functions to be processed early and adaptively. If the neural network's prediction is incorrect, early processing based on the prediction can be stopped and correct processing based on the actual function output can be resumed.
[0009] In some embodiments, the systems and methods can "course correct" errors in the prediction. As an example, the predictor may erroneously determine that a virtual object or character does not need to be rendered and the scene will be rendered in time without the virtual object. Later during processing of the frame, the effects of physics determine that the virtual object will be pushed into the view. The systems and methods described herein provide two options at this point. In one option, the systems and methods can discard some or all of the pre-processing and re-render the scene with the virtual object present. In the other option, the systems and methods can render the virtual object independently and composite the virtual object into the scene.
[0010] In some embodiments, the systems and methods described herein aid in distributing work across computing resources. For example, a physics engine is typically divided into several components. Examples of these components include a collision detection system, a collision resolution system or solver, and an update system. Collision detection is an expensive process and can be further divided into subcomponents, such as a wide phase detection operation that provides a rough estimate of whether virtual objects are colliding, and a narrow phase detection operation that is much more computationally expensive and calculates the exact contact points between virtual objects. During the wide phase detection operation, a neural network trained on previous results of the wide phase detection operation can predict which virtual objects have a high probability of collision, a certain probability of collision, or no probability of collision at all. For virtual objects predicted to have a high probability, the narrow phase detection operation can be dispatched to other computing resources. Once the wide phase detection operation is completed, virtual objects that may collide to some extent are either resolved and dispatched or not dispatched.
[0011] In one embodiment, the system and method updates a complex backdrop, such as a complex background of a scene. For example, a neural network can be trained to determine when a reasonably static but computationally intensive high-resolution backdrop is likely to need to be re-rendered based on several factors, such as user input, AI state, physics state, and other game state. State values, such as intensity values, color values, shading values, texture values, etc., for rendering the high-resolution backdrop can be predicted via the neural network so that a remote rendering job can be dispatched in anticipation of the high-resolution backdrop.
[0012] While it is important for a game to be able to update all these things in a timely manner, neural networks provide the system with the freedom to move tasks around. Consider a game running on a distributed game engine, which is a cloud-based distributed game engine. An individual node of the distributed game engine can handle reasonable quality output for one user, but not high quality output. This may be due to insufficient resources on the node, for example. Some nodes of the distributed game engine may have very fast graphic processing units (GPUs) capable of high quality graphics rendering of computationally intensive components, other nodes of the distributed game engine may have relatively low computational power GPUs with large amounts of memory capable of rendering items with very high resolution textures, and other nodes of the distributed game engine may have specialized physics or AI components. The game may run with reasonable quality on one general-purpose node of the distributed game engine, or, if predicted early enough, high quality jobs may be delegated to less busy or more specialized nodes of the distributed game engine, which will return high quality output. A deep learning predictor can decide when to dispatch these high quality jobs. If the prediction is late, a reasonable quality is calculated locally at the generic node. If the prediction is early, the high quality output will be outdated by the time it is applied and will be discarded. However, if the prediction is correct, e.g., not too late or too early, but on time, the high quality output will be returned in time and combined with other intermediate results to provide a higher quality experience. Even if all nodes of a distributed game engine are similarly configured, in one embodiment, in some cases, load balancing is applied between the nodes. With load balancing, not all resources are used in the same amount on all nodes.
[0013] In one embodiment, a method for reducing latency in the execution of a game is described. The method includes receiving, via a computer network, a user input associated with the game, and determining a current state of the game from the user input. While the current state of the game is being determined, the method includes predicting a next state of the game based on the user input and one or more predicted user inputs. The method further includes generating one or more predicted image frames from the next state, determining whether the one or more predicted user inputs have been received via the computer network, and transmitting the one or more predicted image frames in response to receiving the one or more predicted user inputs to reduce latency in the execution of the game.
[0014] In an embodiment, a system for reducing latency in the execution of a game is described. The system includes a first node and a second node. The first node receives a user input related to the game via a computer network and determines a current state of the game from the user input. The second node is coupled to the first node. The second node receives the user input related to the game from the first node and predicts a next state of the game based on the user input and one or more predicted user inputs while the current state of the game is being determined. The second node further generates one or more predicted image frames from the next state, determines whether the one or more predicted user inputs have been received via the computer network, and transmits the one or more predicted image frames in response to receiving the one or more predicted user inputs to reduce latency in the execution of the game.
[0015] In one embodiment, a computer-readable medium is described that includes program instructions for reducing latency in game execution. Execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform a number of operations of a method for reducing latency in game execution, as described above.
[0016] Some advantages of the system and method described herein for predicting states using a distributed game engine are described below: A state is predicted and one or more frames of the predicted state are generated before user input for the predicted state is received by the distributed game engine. Once user input is received, the frames are transmitted over a computer network to a client device for displaying images and generating sound according to the frames. The latency of frame generation after user input is received is reduced or eliminated. The speed at which images are provided to client devices and the speed at which applications, such as game applications, run is increased.
[0017] The various embodiments of the present disclosure are best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]
[0018] [Figure 1] FIG. 1 is a diagram of an embodiment of a system for illustrating an application such as a video game or video conferencing application. [Figure 2A-1] A diagram of an embodiment of a system showing that while state 1 is being generated by node A, predicted state 2 is being generated by another node 1. [Figure 2A-2] A diagram of an embodiment of a system showing that while state 1 is being generated by node A, predicted state 2 is being generated by another node 1. [Figure 2B] FIG. 2B is a diagram of the remainder of the system of FIG. 2A. [Figure 3A] FIG. 1 is a diagram of an embodiment of a system showing a distributed game engine and a distributed predictor engine. [Figure 3B] A diagram of an embodiment of a system showing that image and audio frames of prediction state 2 are generated during a period when a virtual object controlled by user A via a client device is at position c. [Figure 3C]An embodiment of a distributed game engine system showing that while predicted state 2 is being generated, user input 1' received by the distributed game engine system from a client device over a computer network serves as a trigger for the generation of state 1'. [Figure 3D] FIG. 13 illustrates that when user input 1' is received instead of user input 2 in predicted state 2, a virtual object or artificial intelligence (AI) entity or AI virtual object is rendered and added to the virtual background. [Figure 4A] FIG. 1 is a diagram of an embodiment of a system showing using a node 1 to perform wide phase detection where virtual objects are likely to collide in a virtual scene, and using another node 2 to perform narrow phase detection to determine contact points of virtual object collisions. [Figure 4B] FIG. 1 is a diagram of an embodiment of a system showing a high quality graphics job being dispatched from node 1 to node 2. [Figure 4C] FIG. 2 is a diagram of an embodiment of a system illustrating the allocation of one or more tasks associated with a physics prediction engine from node 1 to node 2. [Figure 4D] FIG. 1 is a diagram of an embodiment of a system illustrating the allocation of one or more tasks associated with an AI prediction engine from node 1 to node 2. [Diagram 5] FIG. 2 is a diagram of an embodiment of a client device. [Figure 6] 6 is a flow diagram conceptually illustrating various operations performed to stream a cloud video game to the client device of FIG. 5 in accordance with an embodiment of the present disclosure. [Figure 7] FIG. 6 is a block diagram of an embodiment of a game console compatible for interfacing with the display device of the client device of FIG. 5 and capable of communicating with a distributed game engine system over a computer network. [Figure 8] A diagram showing components of a head-mounted display (HMD), which is an example of a client device. [Figure 9] 1 illustrates an embodiment of an Information Service Provider (INSP) architecture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] A system and method for predicting states using a distributed game engine is described. It should be noted that various embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail so as not to unnecessarily obscure various embodiments of the present disclosure.
[0020] FIG. 1 is a diagram of an embodiment of the system 100 to illustrate an application, such as a video game or a video conferencing application. Examples of applications include multiplayer games and single player games. In a multiplayer game, multiple players access the video game from a server system 104. For example, a first avatar is controlled by a user A playing the multiplayer game, and a second car is controlled by another user B playing the multiplayer game. Furthermore, in a multiplayer game, there may be non-playable characters (NPCs) that are controlled by artificial intelligence (AI). The AI is performed and controlled by the server system 104, not by the user. An AI NPC is a type of virtual object that is not controlled by a user through a client device. In a single player game, one or more players access the video game from the server system 104. As an example, an avatar is controlled by a user A playing the single player game, and the same avatar is controlled by a user B playing the single player game.
[0021] The system 100 includes a server system 104, a computer network 102, and multiple client devices 1, 2, and 3. The server system 104 includes one or more servers, which may be server blades or game consoles. Each server includes one or more processors and one or more memory devices. The one or more processors of the server are coupled to one or more memory devices. Examples of processors used herein include a central processing unit (CPU), a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), and a programmable logic device (PLD). In one embodiment, a memory device is a device from which data is read or to which data is written. Examples of memory devices include a read only memory (ROM) device, or a random access memory (RAM) device, or a combination thereof. Illustratively, memory devices include flash memory, cache, or a redundant array of independent disks (RAID).
[0022] The computer network 102 is used to transfer data, such as video and audio data, between client devices and a server, or between client devices and a node, or between multiple nodes, or between multiple servers, or between multiple client devices, to facilitate operation of a distributed game engine system (DGES) 106, which includes a distributed game engine and a distributed predictor engine. Examples of the computer network 102 include a wide area network (WAN) such as the Internet or a local area network (LAN) such as the Internet, or a combination thereof.
[0023] As used herein, a client device is a device that is operated by a user to access applications executed by the distributed game engine system 106. Examples of client devices include computers, tablets, game consoles, smartphones, handheld controllers, head mounted displays (HMDs), combinations of televisions, game consoles, and handheld controllers, and combinations of HMDs, handheld controllers, and game consoles. Examples of handheld controllers include the DualShock® controller and the Move® motion controller, both available from Sony®. In combinations that include a game console and a handheld controller, the game console is coupled to the handheld controller via a wired or wireless connection. Furthermore, in combinations of an HMD, a handheld controller, and a game console, the HMD is coupled to the game console via a wired or wireless connection. Examples of wired connections include a serial or parallel transfer cable or a Universal Serial Bus (USB) cable. Examples of wireless connections include a Bluetooth® connection and a Wi-Fi® connection.
[0024] The game console, or HMD, or handheld controller, or combination thereof, includes one or more cameras for capturing gestures such as, for example, position or orientation changes or combinations of input devices of the client device, such as, for example, handheld controller, HMD, mouse, keypad, joystick, touchpad, touch screen, etc., or movements of body parts of user A. Examples of body parts include arms, hands, legs, wrists, one or more fingers, legs, knees, eyes, head, etc. As used herein, an HMD is a display device worn by a user to view virtual scenes, such as virtual reality (VR) scenes and augmented reality (AR) scenes. Also, a television is an example of a display device and includes a display screen. Examples of display devices used herein include liquid crystal display (LDC) devices, light emitting diode display (LED) devices, and plasma display devices. The VR or AR scene is generated upon execution of the application by the distributed game engine system 106.
[0025] The server system 104 includes multiple nodes that collectively operate as a distributed game engine system 106. For example, the distributed game engine system 106 includes multiple AI engines AI1-AIn, where n is an integer greater than 1. Additionally, the distributed game engine system 106 includes multiple audio engines (AEs) AE1-AEn, multiple physics engines PE1-PEn, and multiple graphics engines (GEs) GE1-GEn, where n is an integer greater than 1. In one embodiment, the distributed game engine system 106 further includes multiple animation engines, multiple network engines, multiple memory management engines, multiple screaming engines, and / or multiple script engines. Each of the AI engines AI1-AIn, audio engines AE1-AEn, physics engines PE1-PEn, and graphics engines GE1-GEn are part of the distributed game engine system 106. In one embodiment, an engine as used herein is a software engine. A software engine is an object that represents a library, a software development kit (SDK), or a functional block. The software engines are executed by one or more processors, such as a graphics processing unit (GPU) or a central processing unit (CPU). In one embodiment, each engine as used herein is a virtual machine (VM). In some embodiments, each engine is a processor or hardware device. In one embodiment, each engine is a neural network or part of a neural network.
[0026] The distributed game engine system 106 stores applications, e.g., game computer programs, computer programs for generating VR scenes, computer programs for generating augmented reality AR scenes, physics software programs for applying physical laws to generate VR or AR scenes, rendering computer programs for applying rendering operations to generate VR or AR scenes. As an example, a part of the application is stored and executed by node A of the distributed game engine system 106, another part of the application is stored and executed by another node 1 of the distributed game engine system 106, and the remaining part of the application is stored and executed by yet another node 2 of the distributed game engine system 106. A node as used herein is a hardware server or a game console, or a hardware server of the distributed game engine system 106 for executing at least a part of the application. As an example, a node has a housing separate from the housing of another node. As another example, a node is located in a rack in a data center that is different from the rack in which another node is located in the data center. As yet another example, a node of the distributed game engine system 106 is located in a data center that is different from another node of the distributed game engine system 106.
[0027] In some embodiments, multiple nodes are located within a single housing. For example, in the case of a PlayStation Now server, a single housing is shared by multiple nodes. When multiple nodes are housed in a single housing, each node has its own network connectivity to the computer network 102 via a network communication device. However, alternatively, the single housing includes a network communication device and the nodes are coupled to the computer network 102 via the same network communication device. Having multiple nodes in a single housing allows for improved connectivity in terms of throughput and latency.
[0028] In one embodiment, the node is a virtual machine that is an emulation of a computer system. In a virtual machine, a hypervisor is computer software or hardware, or a combination thereof, that shares and manages hardware resources, such as a processor and memory devices, to run applications on one or more operating systems. By way of example, a virtual machine includes an operating system, one or more application computer programs that run on the operating system, and one or more hardware resources, such as a central processing unit, a graphic processing unit, a video encoder, an audio encoder, a video codec, an audio codec, a video decoder, an audio decoder, a network communication device, a memory device, an internal communication device, etc., that are accessed by the one or more application computer programs through the operating system and the hypervisor to perform the functions described herein as being performed by the node. The applications listed above are examples of application computer programs.
[0029] Further, in an embodiment, the physics engine applies the laws of physics to determine the position and orientation of the virtual objects or virtual background of the virtual scene, which may be a VR scene or an AR scene. Examples of virtual objects used herein include a virtual gun, a virtual spear, a virtual boulder, a virtual weapon, an avatar, a virtual character, etc. Examples of virtual backgrounds used herein include a virtual room, a virtual natural environment, a virtual tree, a virtual ocean, and a virtual environment in which the virtual objects are located. In an embodiment, the terms virtual backdrop and virtual background are used interchangeably herein. The virtual objects or virtual background are controlled by user A via client device 1. In an embodiment, the virtual objects or virtual background are controlled by an AI on behalf of user A. The multiple positions and multiple orientations define the movement of the virtual objects or virtual background. The physics engine is a computer program that is executed to determine the physical relationships between various parts in the virtual scene and between various virtual scenes. The physical relationships are determined based on the laws of physics, such as the laws of gravity, motion, and friction.
[0030] In one embodiment, the AI engine determines the movement and functionality of virtual objects or characters within a video game, which virtual objects or characters cannot be controlled by a user via a client device.
[0031] In some embodiments, the audio engine determines and provides audio data for a corresponding virtual scene of the game, for example, when a portion of the virtual scene produces sound, the audio engine determines audio data for outputting the sound and other variables of the sound, such as pitch, tone, amplitude, etc., and links the audio data with that portion of the virtual scene.
[0032] In an embodiment, the graphics engine is a rendering engine or renderer or rendering operation. The rendering engine or renderer or rendering operation applies graphics, such as color, texture, shading, intensity, lighting, or a combination thereof, to a two-dimensional (2D) or three-dimensional (3D) model of a virtual object to create a 2D or 3D representation of the virtual object, applies the graphics to a 2D or 3D model of a virtual background to create a representation of the virtual background, and outputs one or more image frames. As used herein, a model of a virtual object has a position and orientation of the virtual object, and a model of a virtual background has a position and orientation of the virtual background. For example, the model is a grid with vertices and defines a shape of the virtual object or virtual background. The shape is defined according to a position and orientation calculated for the virtual object or virtual background. The rendering engine generates an image from a 2D or 3D model of one or more parts, such as the virtual object or virtual background, of the virtual scene. For example, the rendering engine defines the color, texturing, shading, and light intensity to be applied to one or more parts of the virtual scene.
[0033] The server system 104 includes a video codec 108 that encodes one or more image frames to output one or more encoded image frames and decodes one or more encoded image frames to output one or more decoded image frames. For example, the video codec 108 is implemented as one or more processors or computer software to compress or decompress one or more image frames using a video encoding protocol such as H.264. A video encoding protocol is also a video decoding protocol. As another example, the video codec 108 is a hardware device, e.g., an integrated circuit, a processor, etc., or a software module, e.g., a computer program, etc., or a combination thereof, that compresses or decompresses image frames according to a video file format or a streaming video format or a video encoding format, e.g., H.264, H.265 / MPEG-H, H.263 / MPEG-4, H.262 / MPEG-2a, a customized protocol, etc. In an embodiment, the terms compressing and encoding are used interchangeably herein, and the terms decompressing and decoding are used interchangeably herein.
[0034] In one embodiment, in addition to the video codec 108, an audio codec is used in the system 100 to apply an audio encoding protocol to encode audio data to output one or more encoded audio frames, and to apply an audio decoding protocol to decode the audio frames to output one or more decoded audio data. The audio encoding protocol is also an audio decoding protocol. The audio codec is a hardware device, e.g., an integrated circuit, a processor, etc., or a software module, e.g., a computer program, etc., or a combination thereof, that compresses or decompresses audio data according to an audio file format or a streaming audio format.
[0035] User A operates the client device 1 to access an application stored in the server system 104. For example, user A logs into his / her user account stored in the server system 104 via a website or a user application to access the application stored in the server system 104. When accessing the application, user A uses the client device 1 to provide a user input 1. Examples of user input 1 are the selection of a button on the client device 1, the touch of the client device 1, and a gesture made by user A with his / her body or the client device 1. In one embodiment. The user input includes one or more movements of one or more joysticks on the client device 1, or one or more gestures made by user A, or one or more selections of one or more buttons on the client device 1.
[0036] A user input 1 is sent from a client device 1 to the server system 104 via a computer network 102 to change the state of an application. For example, one or more of the physics engines PE1 to PEn output one or more positions of a virtual object in a virtual scene, one or more orientations of the virtual object, one or more positions of a virtual background in the virtual scene, and one or more orientations of the virtual background. Also, one or more of the audio engines AE1 to AEn output audio data emitted by a virtual object, output audio data output by an AI in the virtual scene, and output audio data output with the virtual background. Furthermore, one or more of the AI engines AI1 to AIn output one or more positions and one or more orientations of an AI in the virtual scene. One or more of the graphics engines GE1 to GEn provide graphics, such as color, texture, intensity, shading, lighting, etc., to the models of the virtual objects in the virtual scene, the models of the AI, and the models of the virtual background to output one or more image frames. The virtual objects in the virtual scene are controlled by the user input 1 and may be avatars or representations of a user A.
[0037] The video codec 108 receives one or more image frames generated based on the user input 1 from one or more of the graphics engines GE1 to GEn, encodes the one or more image frames, and outputs a video stream of encoded video frames 1. Furthermore, the audio codec receives audio data generated based on the user input 1 output from one or more of the audio engines AE1 to AEn, encodes the audio data, and outputs one or more encoded audio frames. The encoded video frames 1 and the encoded audio frames are packetized by the server system 104 by applying an external communication protocol, such as Transmission Control Protocol (TCP) over Internet Protocol (IP) to generate packets. The packets are transmitted from the server system 104 to the client device 1 via the computer network 102.
[0038] The client device 1 receives packets from the server system 104, applies an external communication protocol to depacketize the packets, and extracts the encoded video frames 1 and the encoded audio frames from the packets. Furthermore, the client device 1 applies a video decoding protocol to decompress or decode the encoded video frames 1 to obtain image frames, and applies an audio decoding protocol to decompress or decode the encoded audio frames to obtain audio frames. The audio data of the audio frames are output as voice by the client device 1. Furthermore, the image frames are displayed as images on a display device of the client device 1. Examples of display devices used herein include liquid crystal display (LCD) devices, light emitting diode (LED) display devices, and plasma display devices.
[0039] Similarly, user input is received from client device 2 by server system 104 via computer network 102 to output encoded frame 2 including encoded video frames and encoded audio frames. The encoded frame 2 is transmitted from server system 104 via computer network 102 to client device 2 to display one or more images on a display device of client device 2 and output audio on client device 2. Client device 2 is operated by user 2 and can be used to control an avatar different from the avatar controlled by user A via client device 1. Similarly, user input is received from client device 3 by server system 104 via computer network 102 to output encoded frame 3 including encoded video frames and encoded audio frames. The encoded frame 3 is transmitted from server system 104 via computer network 102 to client device 3 to display one or more images on a display device of client device 3 and output audio on client device 3. Client device 3 is operated by user 3 and can be used to control an avatar different from the avatar controlled by user A via client device 1. Also, client device 3 may be used to control an avatar that is different from the avatar controlled by user B via client device 2.
[0040] 2A is a diagram of an embodiment of system 200 illustrating that when state 1 is being generated by node A, predicted state 2 is being generated by node 1. State 1 may also be referred to herein as the current state, and predicted state 2 may also be referred to herein as the next state. FIG. 2B is a diagram of the remainder of system 200. System 200 includes node A and node 1. Additionally, system 200 includes computer network 102, node assembly server 202, cloud gaming server 204, and client device 1. Additionally, system 200 includes switch system 206.
[0041] The switch system 206 includes one or more switches that facilitate the transfer of data between the node assembly server 202 and two or more of the nodes A, B (not shown), and C (not shown) described herein. For example, the switch system 206 is a switch fabric. The switch fabric provides a large amount of bandwidth between the two or more nodes A, B, and C, and is dynamically reconfigured frequently to enable quality of service (QoS). To explain, the QoS helps to alleviate congestion on a link when there is insufficient capacity between the two or more nodes and the QoS retries sending data. Eventually, some of the two or more nodes A, B, and C start processing the remaining data of the two or more nodes A, B, and C that lack capacity. As another example, the switch system 206 includes a multiplexer that selects a node from among the two or more nodes A, B, and C (FIG. 1) that forms the distributed game engine system 106 and forwards data from the node assembly server 206 and forwards the data to one or more of the client devices 1, 2, and 3 via the computer network 102. As another example, the switch system 206 includes one or more transistors that facilitate the transfer of data between the node assembly server 202 and two or more nodes A, B, and C. As yet another example, the switch system 206 includes one or more switches, each switch shifting between an open position and a closed position. An open position of the switch disconnects the node assembly server 202 from the node coupled to the switch. A closed position of the switch couples the node assembly server 202 to the node coupled to the switch. In one embodiment, nodes A, B, and C are nodes of a distributed game engine, and nodes 1 and 2 described herein are nodes of a distributed predictor engine.
[0042] The cloud gaming server 204 and the client devices 1, 2, and 3 are coupled to the computer network 102. Further, the node assembly server 202 is coupled to the cloud gaming server 204.
[0043] Node A includes a central processing unit A (CPU A), a memory device A, another central processing unit A1 (CPU A1), a memory device A1, a graphics processing unit A (GPU A), a GPU memory device A, an internal communication device A, a network communication device A, an audio encoder A, and a video encoder A. The components of node A, such as CPU A, memory device A, CPU A1, memory device A1, GPU A, GPU memory device A, internal communication device A, network communication device A, audio encoder A, and video encoder A, are coupled to each other via a bus A.
[0044] As used herein, a GPU executes a rendering computer program to generate video frames that include state information such as color, texture, intensity, shading, and lighting of an AR or VR scene. Examples of GPUs include processors, ASICs, and PLDs. In one embodiment, the terms "video frame" and "image frame" are used interchangeably herein.
[0045] An internal communication device as used herein is used to communicate data between one node and another node. The internal communication device applies an internal communication protocol, such as a direct memory access (DMA) protocol, a remote DMA (RDMA) protocol, RDMA over converged Ethernet, InfiniBand, Ethernet protocol, a customized protocol, a serial transfer protocol, a parallel transfer protocol, a USB protocol, a wireless protocol, a Bluetooth protocol, a wired protocol, a universal datagram protocol (UDP), UDP over Internet protocol, a transmission control protocol (TCP) over IP protocol, an Ethernet over TCP / IP protocol, etc., to communicate data between two nodes. As an example of DMA, an internal communication chip of a node, such as a PCI Express opaque switch chip, an RDMA chip, an RDMA over converged Ethernet chip, or an InfiniBand chip, communicates via a Peripheral Component Interconnect Express (PCIe) communication bus to directly write to or read from a memory device of one or more other nodes. Moreover, in a communication bus like PCIe, peripherals like GPUs and other devices are memory-based since each peripheral has an assigned memory address space on the bus. To explain, the GPU of one node applies an internal communication protocol to write to or read from the registers or buffers of the GPU of another node. In this way, a node communicates with another node through a shared mailbox register. When the other node reads or writes to that node, there is an interrupt to the part or application running on the node's CPU. The other node sends an interrupt signal before reading to or writing from the node.
[0046] Examples of internal communication devices as used herein include processors, ASICs, and PLDs. Explained, the internal communication device is a PCI Express opaque switch chip or an RDMA chip, or an RDMA over converged Ethernet chip, or an InfiniBand chip. As another example, the internal communication device is a network interface controller or a network interface card (NIC), a device that communicates using serial transfer of data, a device that communicates using parallel transfer of data, or a device that communicates using a USB protocol.
[0047] It should be noted that PCI-Express and RDMA technologies have significantly lower latency and higher performance compared to Ethernet protocol or TCP protocol or UDP protocol because they eliminate a protocol layer that generates overhead in the operating system executed by the CPU. A DMA engine in a node that runs a DMA protocol bypasses the operating system in the node and directly reads or writes from the memory of the other node when the node is allowed to access a block of data in the other node. There is no network protocol such as Ethernet protocol, TCP protocol, or UDP protocol, and the DMA engine of the node decides how to organize the memory and its internal structure. When a memory transfer operation between the node and the other node is required, the internal communication chip of the node runs a DMA engine to read and write data from the other node without using the CPU of the node.
[0048] In one embodiment, a node described herein is coupled to another node described herein via a cable or computer network 102. For example, node A is coupled to node 1 via a coaxial cable, a USB cable, or the Internet. As another example, node 1 is coupled to another node 2 via a cable or computer network 102.
[0049] The network communication device is used to transfer data packets between the nodes and the client devices over the computer network 102. For example, the network communication device transmits and receives data packets by applying an external communication protocol such as TCP / IP, UDP / IP, etc. Examples of the network communication device include a processor, an ASIC, and a PLD. Explaining, the network communication device is a network interface controller or NIC.
[0050] Similarly, node 1 includes a central processing unit 1 (CPU1), a memory device 1, another central processing unit 1a (CPU1a), a memory device 1a, a graphics processing unit 1 (GPU1), a GPU memory device 1, an internal communication device 1, a network communication device 1, an audio encoder 1, and a video encoder 1. The components of node 1, such as CPU1, memory device 1, CPU1a, memory device 1a, GPU1, GPU memory device 1, internal communication device 1, network communication device 1, audio encoder 1, and video encoder 1, are coupled to each other via a bus 1.
[0051] The client device 1 generates a game request 210 and transmits it to the cloud gaming server 204 via the computer network 102. For example, the user A uses an input device of the client device 1 to select one or more buttons on the input device to generate the game request 210. The cloud gaming server 204 determines, based on the game request 210, whether the user account that the user A of the client device 1 accesses to generate the game request 210 is authorized to access the distributed game engine system 106. The user A of the client device 1 provides login information, e.g., username, password, etc., to access the user account via the input device of the client device 1. Once the login information is authenticated by the cloud gaming server 204, the user A of the client device 1 is provided with access to the user account. Upon determining that the user A's user account is authorized to access the distributed game engine system 106, the cloud gaming server 204 transmits a signal to the node assembly server 202 to enable access to the execution of the application via the client device 1 and the computer network 102. The application is executed by one or more of nodes A, B, C, 1, and 2 of the distributed game engine system 106 (FIG. 1).
[0052] In one embodiment, in addition to authenticating the login information, there are additional operations that are performed before allowing the client device 1 to couple to the node assembly server 202 to access the application executed by the distributed game engine system 106. For example, a network test server (not shown) coupled to the computer network 102 receives a signal from the cloud game server 204 to access the application executed by the distributed game engine system 106 and performs a bandwidth ping to multiple data centers having the distributed game engine system 106. The results of the test are provided by the network test server to a cloud resource manager (not shown). The cloud resource manager is a server coupled to the computer network 102. The cloud resource manager determines which data center to connect to the client device 1 to access the application. This determination is based on the test results and other information such as the availability of a sufficient number of nodes and the data center where the application is stored. The cloud resource manager selects one or more of the data centers having one or more of the nodes A, B, C, 1, and 2 and sends a signal to the node assembly server 202 to select one or more of the nodes A, B, C, 1, and 2.
[0053] Upon receiving a signal from the cloud resource manager, the node assembly server 202 selects one or more of nodes A, B, C, 1, and 2 of the distributed game engine system 106 to execute the application, via the switch system 206, to initialize one or more of the nodes A, B, C, 1, and 2. For example, the node assembly server 202 sends a signal to a control input of the switch system 206 to couple nodes A and 1, or nodes A and B, or nodes 1 and 2, or nodes A, 1, and 2. Upon receiving the signal at the control input, the switch system 206 closes one or more positions of the switch to connect the node assembly server 202 to a corresponding one or more of the nodes of the distributed game engine system 106 coupled to one or more of the switches, and opens the remaining positions of the switch to disconnect the remaining nodes of the distributed game engine system 106 from the node assembly server 202. When connected to the node assembly server 202 via the switch system 206, one or more of nodes A, B, C, 1, and 2 of the distributed game engine system 106 execute an application to transfer encoded frames from one or more of nodes A, B, C, 1, and 2 to the client device 1 via the computer network 102.
[0054] After one or more of nodes A, B, C, 1, and 2 are selected by the switch system 206, user A provides user input 1 to node A via the computer network 102 using the input device of client device 1. For example, user input 1 is generated when user A selects one or more buttons on the input device of client device 1, moves input device 1, or a combination thereof. As another example, user input 1 includes one or more image frames of the movement of the input device or user A's body part, or a combination thereof. The image frames are captured by one or more cameras of client device 1. User input 1 is packetized by client device 1 using an external communication protocol to generate one or more packets, and the packets are transmitted from client device 1 to network communication device A of node A via the computer network 102. Network communication device A applies the external communication protocol to depacketize the one or more packets with user input 1 to extract user input 1, and provides user input 1 to CPU A.
[0055] CPU A analyzes user input 1 to determine a change in position and orientation of the input device of client device 1, to determine whether there is a button selection on the input device, to determine whether there is a change in position and orientation of a body part of user A, or a combination thereof. CPU A executes a physics engine to determine one or more positions and one or more orientations of a virtual object in a virtual scene controlled by user A using an input device of client device 1. The one or more positions and one or more orientations of the virtual object correspond to a change in position and orientation of the input device of client device 1, or a change in position and orientation of a body part of user A, or a button selection on the input device, or a combination thereof. CPU A further applies an audio engine to determine audio data corresponding to one or more positions and one or more orientations of the virtual object in state 1. The one or more positions of the virtual object in state 1, the audio data output by the virtual object in state 1, and the one or more orientations of the virtual object in state 1 are stored in memory device A by CPU A. In an embodiment, the virtual background is an example of a virtual object. For example, the virtual background is a virtual object in the virtual scene, and another virtual object is also present in the virtual scene. One or both of these virtual objects are controlled by user A via client device 1.
[0056] Further, CPU A1 accesses from memory device A one or more positions of the virtual object in state 1, audio data output by the virtual object in state 1, and one or more orientations of the virtual object in state 1. CPU A1 applies an AI engine to the one or more positions of the virtual object in state 1, audio data output by the virtual object in state 1, and one or more orientations of the virtual object in state 1. The AI engine is applied by CPU A1 to determine one or more positions of one or more AIs in a virtual scene having a virtual object controlled by user A, one or more orientations of the one or more AIs, and audio data corresponding to the one or more positions and one or more orientations. The one or more positions of one or more AIs in a virtual scene having a virtual object controlled by user A, one or more orientations of the one or more AIs, and audio data corresponding to the one or more positions and one or more orientations are related to state 1. Also, audio data corresponding to the one or more positions and one or more orientations of the one or more AIs are output by the one or more AIs. One or more positions of one or more AIs in state 1, one or more orientations of one or more AIs in state 1, as well as audio data output by one or more AIs in state 1 are stored in memory device A1 by CPU A1.
[0057] GPU A accesses one or more positions of virtual objects and one or more orientations of the virtual objects from memory device A, accesses one or more positions of one or more AIs and one or more orientations of the one or more AIs from memory device A1, and applies a rendering engine to the positions and orientations of the virtual objects and one or more AIs to generate one or more image frames of State 1. The image frames of State 1 are stored by GPU A in GPU memory device A.
[0058] Audio encoder A accesses audio data of the virtual object from memory device A and audio data of one or more AIs from memory device A1 to generate one or more audio frames, encodes the audio frames in state 1, and outputs one or more encoded audio frames in state 1. Similarly, video encoder A accesses image frames in state 1 from GPU memory device A, encodes the image frames, and outputs one or more encoded image frames in state 1.
[0059] Network communication device A receives encoded image frames of state 1 from video encoder A and encoded audio frames of state 1 from audio encoder A, and applies an external communication protocol to the encoded frames of state 1 to generate one or more packets of state 1. The encoded frames of state 1 include encoded image frames of state 1 and encoded audio frames of state 1. Network communication device A transmits the packets of state 1 via computer network 102 to client device 1 for displaying one or more images of state 1 on a display device of client device 1 and outputting audio via client device 1. The images are generated based on the encoded image frames of state 1, and the audio is output based on the encoded audio frames of state 1.
[0060] CPU A stores user input 1 in memory device A. While the physics engine is applied by CPU A, the audio engine is applied by CPU A, the AI engine is applied by CPU A1, and / or the graphics engine is applied by GPU A, node 1 applies its physics prediction engine, audio prediction engine, AI prediction engine, and / or graphics prediction engine to determine a predicted state 2. Predicted state 2 is an example of a next state or next state of state 1. For example, during the period when the physics engine is applied by CPU A, the audio engine is applied by CPU A, the AI engine is applied by CPU A1, and / or the graphics engine is applied by GPU A, CPU A determines that predicted state 2 should be generated. Upon determining that predicted state 2 should be generated, CPU A instructs internal communication device A of node A to access user input 1 from memory device A and provide user input 1 to node 1 for access by CPU 1 of node 1. The internal communication device A of node A applies the internal communication protocol to the user input 1 to generate one or more transfer units having the user input 1 and transmits the transfer units to node 1. The internal communication device 1 of node 1 receives the one or more transfer units having the user input 1 from the internal communication device A of node A, applies the internal communication protocol to parse the one or more transfer units, and obtains the user input 1 from the one or more transfer units. The user input 1 is stored in the memory device 1 of node 1 by the internal communication device 1. For example, the internal communication device 1 receives an instruction from the CPU A of node A, via the internal communication device A, to store the user input 1 in the memory device 1 for access by the CPU 1.
[0061] The CPU1 of the node1 accesses the user input1 from the memory device1 and determines a movement, such as one or more positions, one or more orientations, or a combination thereof, of a virtual object controlled by the user A via the client device1. A movement of the predicted state2 is determined from the predicted user input2 and the also predicted user input3. For example, the CPU1 applies the laws of physics to predict an additional movement of the virtual object when the user input2 is received from the client device1 via the computer network102, and predicts a further movement of the virtual object when the user input3 is received from the client device1 via the computer network102. The user inputs2 and3 are examples of predicted user inputs, which have not yet actually been received at the time when the movement of the virtual object in the predicted state2 is determined. After the user input1 is received, the CPU1 predicts that the user inputs2 and3 will be received, or that the user inputs2 and3 are likely to be received, or that the user inputs2 and3 may be received. User input 2 is predicted by CPU 1 to be received sequentially from client device 1 after receiving user input 1, and user input 3 is predicted by CPU 1 to be received sequentially after receiving user input 2. As an example, predicted state 2 includes a position and orientation of a virtual object based on or corresponding to user input 2, and further includes a position and orientation of a virtual object based on or corresponding to user input 3. The virtual object is controlled by user A via client device 1. The physical laws applied by CPU 1 are part of the physics prediction engine. CPU 1 also determines audio data corresponding to the position and orientation of the virtual object in predicted state 2. CPU 1 stores the position and orientation of the virtual object in predicted state 2 and the audio data output by the virtual object in predicted state 2 in the memory device 1 of node 1. In one embodiment, the movement for predicted state 2 is determined from user input 2 or any other number of user inputs, such as three, four, or five user inputs.
[0062] Further, the CPU1a accesses the position and orientation of the virtual object and the audio data output by the virtual object from the memory device 1 for the predicted state 2, and applies the AI prediction engine to determine one or more positions and one or more orientations of the one or more AIs in one or more virtual scenes in which the position and orientation of the virtual object in the predicted state 2 is determined. For example, the CPU1a determines the position and orientation of the AI in a virtual scene in which a virtual object controlled by the user A via the client device 1 has a position and orientation determined based on the user input 2 or based on the user inputs 2 and 3. As another example, the CPU1a determines that the AI or the virtual object controlled by the CPU1a moves or stands to the right in the virtual scene when another virtual object controlled by the user A via the client device 1 runs or sits to the left in the virtual scene.
[0063] Furthermore, CPU 1a applies an audio prediction engine to generate audio data to be output by one or more AIs in prediction state 2 based on the audio data to be output by the virtual object in prediction state 2. CPU 1a stores, for prediction state 2, one or more positions and one or more orientations of the one or more AIs and the audio data to be output by the one or more AIs in memory device 1a.
[0064] GPU1 of node 1 accesses positions of virtual objects and orientations of virtual objects in prediction state 2 from memory device 1, accesses one or more positions and one or more orientations of one or more AIs in prediction state 2 from memory device 1a, and applies a rendering engine, a graphics prediction engine, to the positions and orientations to generate one or more image frames in prediction state 2. The image frames in prediction state 2 are stored by GPU1 in GPU memory device 1. For example, the image frames in prediction state 2 are stored in one or more caches in GPU memory device 1.
[0065] The audio encoder 1 accesses audio data of the virtual object from the memory device 1 and accesses audio data of one or more AIs from the memory device 1a to generate one or more audio frames, encodes the audio frames of the prediction state 2, and outputs one or more encoded audio frames of the prediction state 2. Similarly, the video encoder 1 accesses image frames of the prediction state 2 from the GPU memory device 1, encodes the image frames, and outputs one or more encoded image frames. The audio encoder 1 stores the encoded audio frames of the prediction state 2 in the memory device of the audio encoder 1, and the video encoder 1 stores the encoded image frames of the prediction state 2 in the memory device of the video encoder 1.
[0066] When the user inputs 2 and 3 are actually received by node A from client device 1 via computer network 102, CPU A of node A transmits the user inputs 2 and 3 to internal communication device A. For example, network communication device A of node A receives user inputs 2 and 3 from client device 1 via computer network 102. Internal communication device A applies an internal communication protocol to the user inputs 2 and 3 to generate one or more transfer units having the user inputs 2 and 3, and transmits the transfer units to internal communication device 1 of node 1. Internal communication device 1 receives the transfer units, applies an internal communication protocol to the transfer units to obtain user inputs 2 and 3 from the transfer units, and provides user inputs 2 and 3 to CPU 1.
[0067] The CPU 1 determines that the actually received user inputs 2 and 3 match the user inputs 2 and 3 whose encoded image frames and encoded audio frames for the prediction state 2 are generated by the node 1. Upon so determining, the CPU 1 transmits a signal to the network communication device 1 of the node 1. The network communication device 1 accesses the encoded image frames of the prediction state 2 from the memory device of the video encoder 1, accesses the encoded audio frames of the prediction state 2 from the memory device of the audio encoder 1, and applies an external communication protocol to the encoded image frames and encoded audio frames of the prediction state 2 to generate one or more packets. The network communication device 1 transmits the packets to the client device 1 via the computer network 102 for displaying one or more images according to the encoded image frames of the prediction state 2 on the client device 1 and outputting audio by the client device 1 according to the encoded audio frames of the prediction state 2.
[0068] By generating the encoded image frames and encoded audio frames of predicted state 2 before the user inputs 2 and 3 are actually received by node 1 or node A, latency in the execution of the application by the distributed game engine system 106 is reduced. For example, when the user inputs 2 and 3 are received by node 1 or node A, there is no delay in analyzing the user inputs 2 and 3 to determine one or more positions and one or more orientations of a virtual object controlled by user A via client device 1, to determine audio data to be output by the virtual object, to determine one or more positions and one or more orientations of one or more AIs in a virtual scene having the virtual object, to determine audio data to be output by the one or more AIs, to generate image frames from the positions and orientations of the virtual object and the AI, to generate encoded image frames of predicted state 2 from the image frames, and to generate encoded audio frames of predicted state 2 from the audio data of the AI and the virtual object. One or more of the operations of determining one or more positions and one or more orientations of a virtual object in the virtual scene, determining audio data to be output by the virtual object, determining one or more positions and one or more orientations of one or more AIs in the virtual scene, determining audio data to be output by the one or more AIs, generating an image frame from the positions and orientations of the virtual objects and AIs, generating an encoded image frame of prediction state 2 from the image frame, and generating an encoded audio frame of prediction state 2 from the audio data of the AI and the virtual object have already been performed in advance by node 1 before user inputs 2 and 3 are received by node 1.When user inputs 2 and 3 are received by node 1, one or more packets are generated from the encoded image frame of predicted state 2 and the encoded audio data of predicted state 2 and transmitted from node 1 to client device 1 via computer network 102 to reduce latency in application execution.
[0069] It should be noted that one or more of the application of a physics prediction engine by CPU1, application of an audio prediction engine by CPU1, application of an AI prediction engine by CPU1a, and application of a graphics prediction engine by GPU1 of node 1 to generate predicted state 2 occur simultaneously with one or more of the application of a physics engine by CPU A, application of an audio engine by CPU A, application of an AI engine by CPU A1, and application of a graphics engine by GPU A of node A to generate state 1. For example, during the period when the physics engine of node A is applied by CPU A, the audio engine is applied by CPU A, the AI engine is applied by CPU A1, and / or the graphics engine is applied by GPU A to generate predicted state 1, one or more of the physics prediction engine, audio prediction engine, AI prediction engine, and graphics prediction engine are applied by node 1 to generate predicted state 2.
[0070] Furthermore, while node 1 is applying a physics prediction engine, an AI prediction engine, an audio prediction engine, or a graphics prediction engine, or a combination thereof, to generate image and audio frames of predicted state 2, if user input 1' is received instead of user input 2, one or more of nodes A and 1 apply high priority processing based on user input 1' to determine state 1'. For example, CPU A of node A receives user input 1' from client device 1 via computer network 102. CPU A determines that instead of receiving user input 2 of predicted state 2, it has received user input 1' following receipt of user input 1. CPU A further determines that user input 1' does not match user input 2 for which predicted state 2 is determined.
[0071] The user inputs 2 and 3 for the predicted state 2 are pre-stored in the memory device A to determine whether there is a match between the user input 2 and the user input 1′ for the predicted state 2, and between the user input 3 and the user input 1′ for the predicted state 2. For example, the CPU 1 accesses the user inputs 2 and 3 for the predicted state 2 from the memory device 1 of the node 1, and provides the user inputs 2 and 3 for the predicted state 2 to the internal communication device 1 of the node 1. The internal communication device 1 applies an internal communication protocol to the user inputs 2 and 3 for the predicted state 2 to generate one or more transfer units, and transmits the transfer units to the internal communication device A of the node A. The internal communication device A applies an internal communication protocol to the transfer units to obtain the user inputs 2 and 3 for the predicted state 2, and transmits the user inputs 2 and 3 for the predicted state 2 to the memory device A for storage.
[0072] CPU A provides user input 1' to internal communication device A. Internal communication device A applies an internal communication protocol to the user input 1' to generate one or more transfer units having the user input 1' and transmits the transfer units to internal communication device 1 of node 1. Upon receiving the transfer unit having the user input 1', internal communication device A applies an internal communication protocol to the transfer unit to extract user input 1' from the transfer unit and provides the user input 1' to CPU 1 of node 1.
[0073] When user input 1′ is received by node 1 instead of receiving user input 2, node 1 ceases applying one or more of the physics prediction engine, AI prediction engine, audio prediction engine, and graphics prediction engine used to generate predicted state 2 from user inputs 2 and 3 of predicted state 2, and gives higher priority to generating state 1′ compared to generating predicted state 2. For example, user input 1′ is received by node 1 or node A while or during which node 1 is in the process of applying one or more of its physics prediction engine, its AI prediction engine, its audio prediction engine, or its graphics prediction engine to generate predicted state 2. CPU1 stops applying a physics prediction engine to determine a movement of a virtual object controlled by user A for the prediction state 2, or CPU1 stops applying an audio prediction engine to determine audio data to be output by a virtual object for the prediction state 2, or CPU1a stops applying an AI prediction engine to determine one or more AI movements for the prediction state 2, or CPU1a stops applying an AI prediction engine or an audio prediction engine to determine audio data to be output by an AI for the prediction state 2, or GPU1 stops applying a graphics prediction engine to generate one or more image frames from the movement of the virtual object in the prediction state 2 and the movement of the AI, or video encoder 1 stops encoding the image frames in the prediction state 2, or audio encoder 1 stops encoding the audio data in the prediction state 2, or a combination thereof.
[0074] Also, when user input 1' is received instead of user inputs 2 and 3, instead of generating predicted state 2, one or more of nodes A and 1 immediately start generating state 1' from user input 1', giving higher priority to generating state 1' than generating predicted state 2. For example, during or during the process of node 1 generating an image frame or encoded image frame or encoded audio frame or combination thereof for predicted state 2, user input 1' is received by node 1 or node A. Node 1 ceases generating an image frame or encoded image frame or encoded audio frame for predicted state 2. Instead, CPU 1 of node 1 or CPU A of node A applies its physics engine to state 1' to determine the movement of a virtual object controlled by user A via client device 1. Furthermore, CPU 1 of node 1 or CPU A of node A applies its audio engine to determine audio data to be output by the virtual object in state 1'. Moreover, the CPU A1 of the node A or the CPU1a of the node 1 applies its AI engine to determine the movement of one or more AIs in the state 1' based on the movement of the virtual object in the state 1', and to determine the audio data output by one or more AIs in the state 1' based on the audio data output by the virtual object in the state 1'. Furthermore, the GPU A of the node A or the GPU1 of the node 1 applies its graphics engine to generate one or more image frames of the virtual object, a virtual background of the virtual scene in which the virtual object is displayed, and graphics of the one or more AIs in the state 1'. Furthermore, the audio encoder A or the audio encoder 1 encodes the audio data output by the virtual object in the state 1' and the audio data output by the one or more AIs in the state 1' to generate one or more encoded audio frames. Furthermore, the video encoder A or the video encoder 1 encodes the image frame of the state 1' to output one or more encoded image frames.Either network communication device A or network communication device 1 applies an external communication protocol to the encoded audio frames and encoded image frames of state 1' to generate one or more packets of state 1' and transmits the packets via computer network 102 to client device 1. Client device 1 processes the encoded image frames of state 1' to display one or more images of state 1' on a display device of client device 1, and processes the encoded audio frames of state 1' to output audio associated with the one or more images.
[0075] In one embodiment, if the CPU 1 determines that the actually received user inputs 2 and 3 match the user inputs 2 and 3 of the image frames and audio frames of the prediction state 2 generated by the node 1, the CPU 1 sends an encoding signal to the video encoder 1 of the node 1 and the audio encoder 1 of the node 1. The image frames of the prediction state 2 and the audio data of the prediction state 2 are not encoded until the user inputs 2 and 3 are actually received. Upon receiving the encoding signal, the video encoder 1 performs encoding of the image frames of the prediction state 2 as described above, and the audio encoder 1 performs encoding of the audio data of the prediction state 2. The network communication device 1 of the node 1 applies an external communication protocol to the encoded frames, including the encoded audio frames and the encoded image frames, to generate packets in the manner described above, and transmits the packets to the client device 1.
[0076] In one embodiment, a portion of state 1' is generated by node A, and another portion of state 1' is generated by node 1 based on user input 1'. For example, node A applies its physics engine to determine the movement of virtual objects in the virtual scene in state 1', and node 1 applies its audio engine to determine audio data output by the virtual objects in state 1'. As another example, node A applies its physics engine to determine the movement of virtual objects in the virtual scene in state 1', and node 1 applies its AI engine to determine the movement of one or more AIs in the virtual scene based on the movement of the virtual objects in state 1'.
[0077] In this embodiment, a portion of state 1' generated by node A is transferred from node A to node 1, or a portion of state 1' generated by node 1 is transferred from node 1 to node A for transmission to client device 1 via computer network 102. For example, data including positions and orientations defining the movement of virtual objects in the virtual scene in state 1' is accessed by CPU A from memory device A and transmitted to internal communication device A. Internal communication device A generates one or more transfer units by applying an internal communication protocol to the data defining the movement of virtual objects in state 1' and transmits the transfer units to internal communication device 1. Internal communication device 1 applies the internal communication protocol to the transfer units to obtain the data defining the movement of virtual objects in state 1' and transmits the data to CPU 1a to determine positions and orientations defining the movement of one or more AIs in the virtual scene. As another example, one or more image frames of one or more AIs in state 1' are accessed by GPU 1 from GPU memory device 1 of node 1 and transmitted to internal communication device 1 for generating one or more transfer units according to the internal communication protocol. The transfer unit is sent from the internal communication device 1 to the internal communication device A, which applies an internal communication protocol to obtain image frames of one or more AIs in state 1' and send the image frames to the video encoder A. The video encoder A of the node A encodes the image frames of one or more AIs in state 1' and the image frames of the virtual objects in state 1' received from the node 1. The image frames of the virtual objects in state 1' are stored in the GPU memory device A of the node A. Furthermore, the video encoder A encodes any remaining image frames of the virtual scene in state 1' to output encoded image frames in state 1'. Furthermore, the audio data in state 1' is encoded by the audio encoder A to output one or more encoded audio frames in state 1'.Network communication device A applies an external communication protocol to the encoded audio and image frames in state 1' to generate one or more packets and transmits the packets to client device 1 via computer network 102.
[0078] In an embodiment, the high priority processing is performed by one of nodes 1 and A having a faster processing speed and / or a larger memory compared to the other nodes. For example, CPU 1 having a faster processing speed is selected by nodes 1 and A to apply its physics engine to generate data regarding the movement of a virtual object controlled by user A in state 1', such as one or more positions and one or more orientations. CPU A determines that CPU 1 has a faster processing speed and transmits user input 1' in state 1' to CPU 1 for processing user input 1' via internal communication devices A and 1. As another example, GPU memory device A has a larger amount of memory space compared to the memory space in GPU memory device 1. In this example, the data regarding the movement of the virtual object in state 1' determined by CPU 1 is accessed by CPU 1 from memory device 1 and transmitted to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the data regarding the movement of the virtual object in state 1' to generate one or more transfer units and transmits the transfer units to internal communication device A. The internal communication device A applies an internal communication protocol to the transfer unit to obtain data regarding the movement of the virtual object and transmits the data to the memory device A for storage. The GPU A accesses the data regarding the movement of the virtual object in state 1′ from the memory device A, generates one or more image frames from the data, and stores the image frames in the GPU memory device A.
[0079] In an embodiment, one or more of nodes 1 and A, in response to receiving user input 1', determine or identify differences between one or more elements of predicted state 2 and one or more elements of state 1', and generate audio data or image frames based on the differences for providing to client device 1 via computer network 102. For example, GPU 1 generates one or more image frames of a virtual background of predicted state 2 of the virtual scene. After the image frames of the virtual background of predicted state 2 are generated, user input 1' is received by node A. User input 1' is provided from node A to node 1 for storage in GPU memory device 1. GPU 1 determines that user input 1' is received from node A, and upon receiving user input 1', an image frame of the virtual background of predicted state 2 is provided by GPU 1 of node 1 from GPU memory device 1 to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the image frames of the virtual background of predicted state 2 to generate one or more transfer units, and transmits the transfer units to internal communication device A of node A. The internal communication device A of the node A applies the internal communication protocol to the transfer unit to obtain image frames of the virtual background of the predicted state 2, and provides the image frames to the CPU A and CPU A1 of the node A. The CPU A analyzes the image frames of the virtual background of the predicted state 2 to determine or identify differences between the image frames generated for the virtual scene of the state 1' and the image frames of the virtual background of the virtual scene of the predicted state 2. The CPU A determines the differences as differences of virtual objects to be superimposed on the virtual background of the predicted state 2. The differences are determined based on the user input 1'. The virtual object to be superimposed, or the position and orientation of the virtual object, are examples of one or more elements of the state 1', and the virtual background of the predicted state 2 is an example of one or more elements of the predicted state 2. The CPU A determines the position and orientation of the virtual object of the state 1' and audio data output by the virtual object, and stores the position, orientation, and audio data in the memory device A of the node A. The position and orientation of the virtual object of the state 1' is determined from the user input 1'.
[0080] Further, in the embodiment, the CPU A1 of the node A analyzes the image frames of the virtual background of the predicted state 2 to determine or identify that the difference between the image frames generated for the virtual scene of the state 1' and the image frames of the virtual background of the virtual scene of the predicted state 2 is the difference of the AI superimposed on the virtual background of the predicted state 2. The difference is determined based on the position and orientation of the virtual object of the state 1'. The superimposed AI, or the position and orientation of the AI, are examples of one or more elements of the state 1'. The CPU A1 determines the position and orientation of the AI and the audio data output by the AI for the state 1', and stores the position, orientation, and audio data in the memory device A1. The position and orientation of the AI of the state 1' is determined from the position and orientation of the virtual object of the state 1'. To explain, if the virtual object of the state 1' jumps, the AI of the state 1' jumps higher than the virtual object, or if the virtual object of the state 1' shoots in the virtual scene, the AI also shoots in the virtual scene.
[0081] In that embodiment, GPU A generates image frames having a virtual object in state 1' based on the position and orientation of the virtual object and an AI in state 1' based on the position and orientation of the AI, and superimposes the image frames of the virtual object in state 1' on the image frames of the virtual background in predicted state 2 to generate superimposed image frames of the virtual scene in state 1'. GPU A sends the superimposed image frames to video encoder A. Video encoder A encodes the superimposed image frames to generate encoded image frames of the virtual scene in state 1'.
[0082] Also in this embodiment, audio encoder A encodes audio data of the AI and virtual objects in state 1' to generate encoded audio frames. The encoded image frames of the virtual scene in state 1' and the encoded audio frames of the virtual scene in state 1' are converted into one or more packets by network communication device A and transmitted to client device 1 by computer network 102 for displaying one or more composite images of the virtual scene in state 1' and the virtual objects and AI in state 1' with the virtual background in predicted state 2 on a display device of client device 1.
[0083] In one embodiment, instead of the image frames of the virtual object and the image frames of the AI being superimposed on the image frames of the virtual background as described in the previous embodiment, the image frames having the virtual object or the AI, or both, and the virtual background of the virtual scene in state 1' are generated by node A or node 1 based on the user input 1', encoded, and transmitted to the client device 1 via the computer network 102 for display. In this embodiment, the GPU 1 discards, such as by deleting or erasing, the image frames having the virtual background in the predicted state 2 from the GPU memory device 1. The image frames having the virtual background in the predicted state 2 match the image frames having the virtual background in state 1'. For example, the virtual background in the predicted state 2 is the same as the virtual background in state 1'.
[0084] It should be noted that in one embodiment, system 200 includes a number of nodes other than those shown in Figure 2A. For example, system 200 includes 50 nodes, or 25 nodes, or 5 nodes.
[0085] It is further noted that some of the above embodiments are described with respect to two user inputs 2 and 3. However, in one embodiment, the embodiments apply equally to one user input, such as user input 2 or user input 3, but not both user inputs 2 and 3. For example, predicted state 2 is generated based on user input 2, but not both user inputs 2 and 3.
[0086] In an embodiment, instead of using user input 1 to generate state 1 or predicted state 2, any number of user inputs are used to generate state 1 or predicted state 2. For example, CPU A of node A stores one or more previous user inputs in memory device A. The one or more previous user inputs are user inputs that node A received from client device 1 via computer network 102 before user input 1 was received by node A from client device 1. The one or more previous user inputs are generated in a similar manner to how user input 1 is generated. For example, the one or more previous user inputs are generated when user A selects one or more buttons on an input device of client device 1, moves one or more joysticks on client device 1, or makes one or more gestures that are captured by a camera of client device 1. As an example, the one or more previous user inputs are generated by client device 1 when user A controls the input device of client device 1 to move an avatar or a virtual object in a virtual scene. The internal communication device A accesses the user input 1 and one or more previous user inputs of state 1 stored in the GPU memory device A of node A, applies an internal application protocol to the user input 1 and the one or more previous user inputs to generate one or more transfer units having the user input 1 and the previous user input, and transmits the transfer units to the internal communication device 1 of node 1. The predicted state 2 is generated by node 1 based on the one or more previous user inputs and the user input 1 in a similar manner as state 1 is generated by node 1 based on the user input 1.
[0087] In one embodiment, the output of state 1 is used to generate a predicted state 2. For example, CPU1 determines a position of a virtual object controlled by user A via client device 1 for predicted state 2 based on the position of the virtual object in state 1. Explained, for predicted state 2, the virtual object moves right or left or up or down from the position in state 1. As another example, CPU1 determines an orientation of a virtual object controlled by user A via client device 1 for predicted state 2 based on the orientation of the virtual object in state 1. Explained, for predicted state 2, the virtual object rotates clockwise or counterclockwise from the orientation of the virtual object in state 1. As yet another example, CPU1a determines a position of an AI for predicted state 2 based on the position of the virtual object in state 1, and determines an orientation of the AI for predicted state 2 based on the orientation of the AI in state 1.
[0088] In one embodiment, any control of an input device by a user to change a state described herein is performed via a user account assigned to the user by the server system 104. For example, User A controls a virtual object using client device 1 via his / her user account after logging into the user account.
[0089] It should be noted that, in one embodiment, functions described herein as being performed by CPU A of node A are instead performed by CPU A1 of node A, or functions described as being performed by CPU A1 are instead performed by CPU A. Similarly, in an embodiment, functions described herein as being performed by CPU 1 of node 1 are instead performed by CPU 1a of node 1, or functions described as being performed by CPU 1a are instead performed by CPU 1.
[0090] 3A is a diagram of an embodiment of a system 300 showing a distributed game engine 302 and a distributed predictor engine 304. The distributed game engine 302 and the distributed predictor engine 304 are part of the distributed game engine system 106 (FIG. 1). For example, one or more nodes A, B, and C of the distributed game engine system 106 are nodes of the distributed game engine 302, and one or more of nodes 1 and 2 of the distributed game engine system 106 are nodes of the distributed predictor engine 304. Examples of the distributed game engine 302 and the distributed predictor engine 304, respectively, include a machine learning engine and a neural network.
[0091] As shown in graph 306, a virtual object, such as an avatar or representation of user A, controlled by user A via client device 1 moves from position a to position b in the virtual scene. Graph 306 illustrates the positions of virtual objects in the virtual scene. User Input 1 is received and processed by distributed game engine 302 to generate State 1, which includes position c of a virtual object controlled by user A via client device 1 (FIG. 1). Position c is an example of an output of State 1. User Input 1 is received to move a virtual object controlled by user A via client device 1 from position b to position c.
[0092] During the period when user input 1 is being processed to determine position c or to generate an image frame showing a virtual object moving from position b to position c, the distributed predictor engine 304 receives user input 1, predicts that user input 2 is received from client device 1 via computer network 102 (FIG. 1), processes user input 2, and outputs predicted state 2. Predicted state 2 includes position d1 of a virtual object controlled by user A and a movement from position c to position d1. Furthermore, during the period when user input 1 is being processed, after receiving user input 2, the distributed predictor engine 304 further predicts that user input 3 is received, and outputs predicted state 2. Predicted state 2 includes position f of a virtual object controlled by user A and a movement from position d1 to position f.
[0093] Furthermore, during the period when user inputs 2 and 3 are processed to output predicted state 2, if the distributed predictor engine 304 or the distributed game engine 302 determines that user input 1' is received, one or more of the distributed game engine 302 and the distributed predictor engine 304 determine state 1' based on the user input 1'. For example, one or more of the distributed game engine 302 and the distributed predictor engine 304 determine state 1' instead of continuing to determine predicted state 2. State 1' includes a position d of a virtual object controlled by user A via client device 1 and a movement of the virtual object from position c to position d. Positions c, d1, and f are located along path I, and positions c, d, and f are located along predicted path II.
[0094] In one embodiment, the distributed predictor engine 304 determines a predicted state 2 of the virtual scene based on the history of user A of the virtual scene, or the history of other users B and C of the virtual scene (FIG. 1), or one or more positions and one or more orientations of one or more virtual objects in the virtual scene, or state 1 of the virtual scene, or a combination thereof. For example, CPU1 (FIG. 2A) determines that user A is more likely to control the input device of client device 1 to generate user input 2 that moves a virtual object from position c to position d1 instead of position d. In this example, CPU1 accesses a database in memory device 1 (FIG. 2A) or memory device A (FIG. 2A) to determine that user A has previously controlled, via his / her user account, to move a virtual object from position c to position d1 more than a predetermined number of times instead of moving it from position c to position d. CPU1 accesses the database in memory device A via internal communication devices A and 1 (FIG. 2A). As another example, CPU 1 determines that user A is more likely to control the input device of client device 1 to generate user input 2 that moves the virtual object from position c to position d1. In this example, CPU 1 accesses memory device 1 (FIG. 2A) or a database in memory device A to determine that more than a predetermined number of users B and C have previously controlled the virtual object, via their corresponding user accounts and their corresponding client devices 2 and 3 (FIG. 1), to move it from position c to position d1 rather than from position c to position d, and that users B and C have each controlled the virtual object more than a predetermined number of times.
[0095] As yet another example, the CPU 1 determines that the user A is more likely to control the input device of the client device 1 to generate a user input 2 that moves the virtual object from the position c to the position d1 rather than the position d. In this example, the CPU 1 accesses a database in the memory device 1 (FIG. 2A) or in the memory device A to determine that the user A has previously controlled the virtual object through his / her user account to move the virtual object from the position c to the position d1 more than a predetermined number of times rather than from the position c to the position d. Also, in this example, the CPU 1 accesses a database in the memory device 1 (FIG. 2A) or in the memory device A to determine that a greater number of users B and C have previously controlled the virtual object through their corresponding user accounts and their corresponding client devices 2 and 3 to move the virtual object from the position c to the position d1 rather than from the position c to the position d, and that each of the users B and C has controlled the virtual object more than a predetermined number of times.
[0096] As another example, CPU1 determines that user A is more likely to control the input device of client device 1 to generate user input 2 that moves a virtual object from position c to position d1 instead of position d. In this example, CPU1 accesses the position of an AI in a virtual scene having a virtual object controlled by user A via client device A. The AI's position is accessed from memory device 1a (FIG. 2A) or memory device A1 (FIG. 2A). The virtual object is at position c in the virtual scene. CPU1 accesses memory device A1 via internal communication devices A and 1 (FIG. 2A). CPU1 determines that, from the AI's position in the virtual scene, the virtual object is more likely to move from position c to position d1 instead of position d. To explain, CPU1 determines that a virtual enemy of the virtual object at position c is close to the virtual object, and therefore the virtual object will run and move to position d1 instead of position d. Position d1 is safer than position d because position d1 provides cover for the virtual object. Position d1 may be behind a virtual rock or a virtual tree in the virtual scene, providing cover compared to position d, where there is no cover.
[0097] As yet another example, CPU1 determines that user A is more likely to control the input device of client device 1 to generate user input 2 that would move the avatar from position c to position d1 rather than position d. In this example, CPU1 accesses the position of another virtual object, such as a slider of a health level bar, from memory device 1 or memory device A (FIG. 2A) to determine that the avatar's health is below a predetermined level. CPU1 further determines that if the avatar's health state is below the predetermined level, instead of receiving user input 1', user input 2 is more likely to be received from client device 1 via computer network 102.
[0098] Note that in graph 306, instead of positions a, b, c, d, d1, and f, orientations a, b, c, d, d1, and f can be used, or a, b, c, d, d1, and f are each combinations of positions and orientations.
[0099] 3B is a diagram of an embodiment of a system 310 illustrating that image and audio frames of predicted state 2 are generated during a time period when a virtual object controlled by user A via client device 1 (FIG. 1) is at position c. System 310 includes a distributed predictor engine 304, which further includes a distributed physics prediction engine 312. Distributed physics prediction engine 312 is executed by CPU A of node A, or a CPU of node B, or CPU 1 of node 1, or CPU 2 of node 2, or a combination thereof.
[0100] During a period when the virtual object controlled by user A through client device 1 is at position c, the distributed physics prediction engine 312 predicts or determines that the virtual object is highly likely or somewhat likely to collide with another virtual object 314 located between positions c and d1. The distributed physics prediction engine 312 predicts or determines that the virtual object is highly likely or somewhat likely to collide with another virtual object 314 when the virtual object moves from position c to position d1 in prediction state 2. Upon so predicting, the distributed physics prediction engine 312 determines one or more positions and one or more orientations of the virtual object controlled by user A and one or more positions and one or more orientations of the virtual object 314 during the collision, and stores the positions and orientations in one or more memory devices of nodes 1 (FIG. 2A) and 2. Additionally, the distributed audio prediction engine of the distributed predictor engine 304 generates audio data of the collision. The distributed graphics engine of the distributed predictor engine 304 also accesses the positions and orientations of the virtual object controlled by user A and the virtual object 314 from one or more memory devices of the nodes 1 (FIG. 2A) and 2, and generates one or more image frames from the positions and orientations during the collision. The one or more video encoders of the nodes 1 and 2 also obtain image frames, encode the image frames, and output one or more encoded image frames of the collision. The one or more audio encoders of the nodes 1 and 2 also obtain audio data of the collision, output one or more audio frames, encode the audio frames, and output one or more encoded audio frames of the collision.
[0101] Upon actually receiving user input 2 from client device 1 via computer network 102, one or more network communication devices of nodes 1 and 2 apply an external communication protocol to the encoded audio and video frames to generate one or more packets and transmit the packets via computer network 102 to client device 1 for displaying one or more images of the collision at client device 1 and for outputting audio of the collision at client device 1.
[0102] 3C is an embodiment of the distributed game engine system 106 illustrating that while the predicted state 2 is being generated, a user input 1′ received by the distributed game engine system 106 from the client device 1 via the computer network 102 (FIG. 1) acts as a trigger for the generation of state 1′. Immediately after receiving the user input 1, when the user input 1′ is received by the distributed game engine system 106 from the client device 1 via the computer network 102, the distributed game engine system 106 stops applying the physics prediction engine, the AI prediction engine, the graphics prediction engine, and the audio prediction engine to generate image frames and audio frames of the predicted state 2, and starts applying the graphics engine, the AI engine, the graphics engine, and the audio engine to generate image frames and audio frames of state 1′. As an example, the CPU 1 of node 1 (FIG. 2A) stops determining the position, orientation, and audio data of the virtual object controlled by user A for the predicted state 2. Instead, CPU1 of node 1 (FIG. 1) or CPU A of node A (FIG. 2A) starts to determine the position, orientation, and audio data of the virtual object in state 1' based on user input 1'. As another example, CPU1a of node 1 (FIG. 2A) stops generating the position, orientation, and audio data of the AI in predicted state 2. Instead, CPU1a of node 1 (FIG. 2A) or CPU A1 of node A (FIG. 2A) starts to determine the position, orientation, and audio data of the AI in state 1' based on user input 1'. As yet another example, GPU1 of node 1 (FIG. 2A) stops generating one or more image frames from one or more positions and orientations of the virtual object in predicted state 2 and one or more positions and orientations of the AI in predicted state 2. The virtual object is controlled by user A via client device 1 (FIG. 1). Instead, GPU 1 of node 1 or GPU A of node A (Figure 2A) begins generating one or more image frames from one or more positions and orientations of the virtual object in state 1' and one or more positions and orientations of the AI in state 1'.Also, the audio encoder 1 (FIG. 2A) of the node 1 stops encoding the audio data output by the virtual object in the prediction state 2, and the video encoder 1 (FIG. 2A) of the node 1 stops encoding the image frame in the prediction state 2. Instead, the audio encoder 1 or the audio encoder A starts encoding the audio data in the state 1' to output the encoded audio frame in the state 1'. Also, the video encoder 1 or the video encoder A starts encoding the image frame in the state 1' to output the encoded image frame in the state 1'. The encoded audio frame and the encoded image frame in the state 1' are packetized by the network communication device A of the node A or the network communication device 1 of the node 1 (FIG. 2A) and transmitted to the client device 1 via the computer network 102 (FIG. 2B) for displaying one or more images of the virtual scene at the client device 1 based on the encoded image frames and for outputting the audio of the virtual scene based on the encoded audio frames.
[0103] FIG. 3D illustrates that a virtual object or AI is rendered and added to the virtual background when a user input 1' in state 1' is received instead of a user input 2 in predicted state 2. The GPU 1 of node 1 (FIG. 2A) generates an image frame of the virtual background in predicted state 2, where the image frame does not include a virtual object controlled by user A via client device 1 in predicted state 2 and / or does not include an AI in predicted state 2. After receiving user input 1', the CPU A of node A (FIG. 2A) determines a position and orientation of the virtual object controlled by user A for state 1', and further determines audio data to be output by the virtual object based on the user input 1'. The CPU A of node A determines that the virtual object in state 1' is added to the position and orientation of the virtual background in predicted state 2 at its position and orientation.
[0104] It should be noted that CPU A accesses the position and orientation of the virtual background of predicted state 2 from memory device 1 of node 1. For example, CPU A sends a request for the position and orientation of the virtual background of predicted state 2 to node 1 via internal communication devices A and 1 (FIG. 2A). Upon receiving the request, CPU 1 of node 1 accesses the position and orientation of the virtual background of predicted state 2 from memory device 1 of node 1 and provides the position and orientation to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the position and orientation of the virtual background of predicted state 2 to generate one or more transmission units and transmits the transmission units to internal communication device A of node A. Internal communication device A applies an internal communication protocol to the transmission units to obtain the position and orientation of the virtual background of predicted state 2 and provides the position and orientation of the virtual background to CPU A of node A.
[0105] Further, CPU A1 of node A (FIG. 2A) determines the position and orientation of the AI of state 1′, and further determines the audio data to be output by the AI based on user input 1′. CPU A1 determines to add AI of state 1′ at its position and orientation to the position and orientation of the virtual background of predicted state 2. CPU A1 accesses the position and orientation of the virtual background of predicted state 2 from node 1 in the same manner as described above in which CPU A accessed the position and orientation of the virtual background of predicted state 2 from node 1. GPU A of node A (FIG. 2A) accesses the position and orientation of the virtual object stored in memory device A of node A for state 1′, and superimposes the virtual object at its position and orientation onto one or more image frames of the virtual background of predicted state 2, and outputs one or more superimposed image frames of the virtual scene of state 1′. Further, GPU A of node A accesses the position and orientation of the AI stored in memory device A1 of node A for state 1', overlays the AI at that position and orientation onto the image frame of the virtual background in predicted state 2, and outputs the overlaid image frame of the virtual scene in state 1'.
[0106] The video encoder A of the node A obtains the superimposed image frames and encodes the superimposed image frames to output one or more encoded superimposed image frames. Furthermore, the audio encoder A of the node A encodes audio data output by the virtual objects in state 1' to generate one or more encoded audio frames, and encodes audio data output by the AI in state 1' to generate one or more encoded audio frames. The network communication device A of the node A generates one or more packets from the encoded superimposed image frames and the encoded audio frames in state 1', and transmits the packets to the client device 1 via the computer network 102 to display one or more composite images having the virtual background, the virtual objects, and the AI in the virtual scene on the client device 1.
[0107] In one embodiment, instead of generating a superimposed image frame in state 1' based on the virtual background of predicted state 2, one or more image frames with the same virtual background in state 1' are re-rendered by GPU A of node A to generate one or more image frames in state 1' (Figure 2A).
[0108] Also, in one embodiment, instead of adding both the AI and virtual objects of state 1' to the virtual background of predicted state 2, either the AI or the virtual objects of state 1' are added to the virtual background of predicted state 2.
[0109] 4A is a diagram of an embodiment of a system 400 illustrating using node 1 to perform wide phase detection where virtual objects are likely to collide in a virtual scene and using node 2 to perform narrow phase detection to determine contact points of collision of virtual objects. Node 2 is also used to perform collision resolution. System 400 includes node 1, node 2, node A, node B, computer network 102, node assembly server 202, cloud gaming server 204, and client device 1. Node B is coupled to a switch system 206.
[0110] The node 2 includes a central processing unit 2 (CPU2), a memory device 2, another central processing unit 2a (CPU2a), a memory device 2a, a graphics processing unit 2 (GPU2), a GPU memory device 2, an internal communication device 2, a network communication device 2, an audio encoder (AE) 2, and a video encoder (VE) 2. The components of the node 2, such as the CPU2, the memory device 2, the CPU2a, the memory device 2a, the GPU2, the GPU memory device 2, the internal communication device 2, the network communication device 2, the audio encoder 2, and the video encoder 2, are coupled to each other via a bus 2.
[0111] Upon receiving user input 1, CPU 1 of node 1 sends the computationally intensive job of executing a physics prediction engine to CPU 2 to perform broad detection, narrow phase detection, and / or collision resolution. For example, CPU 1 executes a physics prediction engine to determine whether a first virtual object of the virtual scene will collide, is likely to collide, or has some degree of collision with a second virtual object of the virtual scene based on the predicted user input 2, if the predicted user input 2 is received. Upon such determination, CPU 1 does not determine the position and orientation of the first virtual object or the position and orientation of the second virtual object during the collision. Also, upon such determination, CPU 1 does not determine the position and orientation of the first virtual object or the position and orientation of the second virtual object after the collision. Rather, CPU 1 provides the determination of whether the first virtual object will collide, is likely to collide, or has some degree of collision with the second virtual object to the internal communication device 1 together with instructions to perform narrow phase detection and / or collision resolution. Determining the position and orientation of the first and second virtual objects during and after the collision is more computationally intensive compared to determining whether the first and second virtual objects will collide with each other, or are likely to collide, or are likely to collide to some extent. The position and orientation of the first and second virtual objects during the collision provides a collision point where the first and second virtual objects collide, and is an example of a complex physical state. Similarly, the position and orientation of the first and second virtual objects after the collision is an additional example of a complex physical state. Furthermore, determining whether the first and second virtual objects will collide, or are likely to collide, or are likely to collide to some extent is an example of a simple physical state determination. Determining that the first and second virtual objects have a collision probability greater than a predetermined percentage, such as 70% or 65%, is an example of determining that the first and second objects are likely to collide.Additionally, a determination that the first and second virtual objects have a likelihood of collision that is greater than another predetermined percentage, such as 40% or 50%, and less than a predetermined percentage, such as 70% or 65%, is an example of a determination that there is some likelihood that the first and second virtual objects will collide.
[0112] The internal communication device 1 receives instructions for performing narrow phase detection and / or collision resolution and a determination regarding the collision of the first and second virtual objects, and applies an internal communication protocol to the instructions and the determination to generate one or more transfer units having the instructions and the determination. The determination regarding the collision of the first and second virtual objects includes a determination that the first and second virtual objects will collide, or a determination that the first and second virtual objects are likely to collide, or a determination that the first and second virtual objects are likely to collide to some extent. The internal communication device 1 transmits the transfer units to the internal communication device 2 of the node 2. The internal communication device 2 executes the internal communication protocol to obtain instructions for performing narrow phase detection and / or collision resolution and the determination regarding the collision of the first and second virtual objects from the transfer units, and transmits the instructions and the determination to the CPU 2 of the node 2.
[0113] The CPU 2 executes instructions for narrow phase detection and / or collision resolution based on the determination regarding the collision of the first and second virtual objects. For example, the CPU 2 applies a physics prediction engine to determine one or more positions and one or more orientations of the first virtual object during the collision, one or more positions and one or more orientations of the second virtual object, and audio data during the collision, and stores the positions and orientations and the audio data of the first and second virtual objects in the memory device 2. As another example, to resolve the collision, the CPU 2 applies a physics prediction engine to determine how far the first virtual object should be moved away from the second virtual object after the collision, and applies a law of physics to generate a force to move the first and second virtual objects away from each other. The CPU 2 also determines audio data corresponding to the first and second virtual objects after the collision, and stores the audio data in the memory device 2. The separation between the first and second virtual objects after the collision is defined by one or more positions and one or more orientations of the first virtual object and one or more positions and one or more orientations of the second virtual object.
[0114] The GPU2 accesses from the memory device 2 the position and orientation of the first virtual object during and after the collision and the position and orientation of the second virtual object during and after the collision, applies a graphics prediction engine to render one or more image frames based on the positions and orientations of the first and second virtual objects, and stores the image frames in the GPU memory device 2. The video encoder VE2 accesses the image frames stored in the memory device 2, encodes the image frames, and outputs one or more encoded image frames, which are stored in the video encoder VE2. Additionally, the audio encoder AE2 accesses audio data stored in the memory device 2, encodes the audio data, and outputs one or more encoded audio frames, which are stored in the audio encoder AE2.
[0115] A user input 2 for a predicted state 2 is received by node A over computer network 102 from client device 1. Node A provides the user input 2 to a CPU 1 of node 1 via internal communication devices A and 1. The user input 2 is provided by CPU 1 to internal communication device 1. The internal communication device 1 applies an internal communication protocol to the user input 2 to generate one or more transfer units and transmits the transfer units to internal communication device 2.
[0116] The internal communication device 2 applies the internal communication protocol to the forwarding unit to obtain the user input 2 and transmits the user input 2 to the network communication device 2. In response to receiving the user input 2, the network communication device 2 obtains encoded audio frames of the collision from the audio encoder AE2 and encoded video frames of the collision from the video encoder VE2, applies the external communication protocol to the encoded audio frames and encoded video frames to generate one or more packets, and transmits the packets via the computer network 102 to the client device 1 for displaying one or more images of the collision of the first and second virtual objects on a display device of the client device 1 and for outputting audio of the collision via the client device 1.
[0117] In one embodiment, the positions and orientations of the first and second virtual objects during and after the collision and the audio data during and after the collision are of prediction state 2.
[0118] In one embodiment, CPU2 has higher processor performance than CPU1. For example, CPU2 has a lower response time, or a higher throughput, or a lower utilization of components of node2, or a higher processing power, or a lower load, or executes a smaller amount of jobs, executes a smaller number of tasks, or a combination thereof, compared to CPU1.
[0119] Note that in one embodiment, functions described herein as being performed by CPU2 of node 2 are instead performed by CPU2a of node 2, and vice versa.
[0120] In one embodiment, user input 1 is provided by node A to node 1. For example, user input 1 is received by node 1 from client device 1 via computer network 102. CPU 1 accesses the user input stored in memory device 1 and provides user input 1 to internal communication device 1. Internal communication device 1 applies an internal communication protocol to user input 1 to generate one or more transfer units and transmits the transfer units to internal communication device A. Internal communication device A applies an internal communication protocol to the transfer units to extract user input 1 and provides user input 1 for storage in memory device 1 of node 1 to CPU 1 for access from memory device 1.
[0121] In one embodiment, user input 1, or any other user input described herein, such as user input 2, user input 3, and user input 1′, is transmitted from client device 1 to one or more of the nodes selected by switch system 206 for execution of the application.
[0122] In some embodiments, CPU2 determines a contact surface where the first and second virtual objects collide and a penetration distance from the contact point. In one embodiment, CPU2 analyzes the instructions from CPU1 in detail and determines that there is no collision between the first and second virtual objects and there is no need to determine the positions and orientations of the first and second virtual objects during and after the collision.
[0123] FIG. 4B is a diagram of an embodiment of a system 410 illustrating a high quality graphics job being dispatched from node 1 to node 2. GPU 1 generates one or more image frames or portions of image frames in predicted state 2 and stores the image frames or portions of image frames in GPU memory device 1. The one or more image frames or portions of image frames in predicted state 2 are examples of simple graphics states, such as a virtual background of a virtual scene. GPU 1 generates instructions to generate the remaining image frames or portions of image frames in predicted state 2. The remaining image frames or portions of image frames are examples of complex graphics states, such as a virtual object displayed in the virtual scene. The virtual scene, when displayed, includes the virtual background of the simple graphics state. GPU 1 transmits instructions to internal communication device 1 to provide node 2 with the task of determining the complex graphics state. In response to receiving the instructions, internal communication device 1 accesses one or more positions and one or more orientations of the virtual object from memory device 1, or one or more positions and one or more orientations of the AI from memory device 1a, or a combination thereof, to facilitate generation of the remaining image frames or portions of image frames in predicted state 2. The position and orientation of the virtual object in predicted state 2 stored in memory device 1 and the position and orientation of the AI in predicted state 2 stored in memory device 1a are used by GPU 1 of node 1 to generate an image frame or portion of an image frame in predicted state 2.
[0124] The internal communication device 1 applies an internal communication protocol to the instructions received from the GPU 1 and the positions and orientations of the virtual objects and AI in the predicted state 2 to generate one or more transfer units and transmits the transfer units to the internal communication device 2. The internal communication device 2 receives the transfer units and applies an internal communication protocol to the transfer units to obtain the instructions and the positions and orientations of the virtual objects and AI in the predicted state 2. The internal communication device 2 transmits the instructions along with the positions and orientations of the virtual objects and AI to the GPU 2.
[0125] GPU2 generates a remaining image frame, or a remaining portion of the image frame, from the positions and orientations of the virtual objects and AI in prediction state 2 and stores the remaining image frame, or a remaining portion of the image frame, in GPU memory device 2. Internal communication device 2 accesses the remaining image frame, or a remaining portion of the image frame, in prediction state 2 from GPU memory device 2 and applies an internal communication protocol to the remaining image frame, or a remaining portion of the image frame, to generate one or more transfer units.
[0126] The transfer unit of the predicted state 2 is sent from the internal communication device 2 to the internal communication device 1 of the node 1. The internal communication device 1 applies an internal communication protocol to the transfer unit to obtain the remaining image frame or remaining portion of the image frame of the predicted state 2 and provides the remaining image frame or remaining portion to the GPU memory device 1 for storage.
[0127] GPU1 combines the portion of the image frame in prediction state 2 with the remaining portion of the image frame in prediction state 2, or combines the image frame in prediction state 2 with the remaining portion of the image frame in prediction state 2. For example, GPU1 identifies an image frame in which the portion and the remaining portion are the same from the portion of the image frame in prediction state 2 and the remaining portion of the image frame in prediction state 2, and identifies a location of the portion relative to a location of the remaining portion. By combining the portion and the remaining portion into an image frame in prediction state 2 and identifying a location of the portion and the remaining portion within the image frame, GPU1 outputs an image frame in which the portion is at that location of the image frame in prediction state 2 and the remaining portion is at that location. As another example, GPU1 identifies a sequence in which the image frames are to be displayed on client device 1 from the image frames generated by GPU1 for prediction state 2 and the remaining image frames generated by GPU2 for prediction state 2. GPU1 arranges the image frames generated by GPU1 for prediction state 2 and the remaining image frames generated by GPU2 for prediction state 2 in the sequence, and outputs a presentation or display order of the image frames generated by GPU1 and the remaining image frames generated by GPU2. The image frames generated by GPU1 and the remaining image frames generated by GPU2 are combined, such as arranged, by GPU1 to be in that order.
[0128] The video encoder VE1 encodes a combination of the image frame generated by GPU1 for the prediction state 2 and the remaining image frame generated by GPU2 for the prediction state 2 to output one or more encoded image frames. The video encoder VE1 encodes the image frame generated by GPU1 and the remaining image frame generated by GPU2 according to the order in which the image frames and the remaining image frames are combined. In one embodiment, the video encoder VE1 encodes an image frame in the prediction state 2 having a portion generated by GPU1 for the prediction state 2 and a remaining portion generated by GPU2 for the prediction state 2 to output the encoded image frame.
[0129] When user input 2 for predicted state 2 is received by node 1 from node A, network communication device 1 of node 1 accesses encoded image frames for predicted state 2 from video encoder VE1 in addition to accessing encoded audio frames for predicted state 2 from audio encoder 1 of node 1, applies an external communication protocol to the encoded image frames and encoded audio frames to generate one or more packets, and transmits the packets to client device 1 via computer network 102. Upon receiving the packets, client device 1 displays one or more images based on the encoded image frames for predicted state 2 and outputs audio based on the encoded audio frames for predicted state 2.
[0130] The image frames generated by GPU1 for prediction state 2 are of lower quality compared to the remaining image frames generated by GPU2 for prediction state 2. For example, the image frames generated by GPU1 have a lower resolution than the remaining image frames generated by GPU2. As another example, the image frames generated by GPU1 are displayed at a lower frame rate compared to the display of the remaining image frames generated by GPU2.
[0131] Similarly, the portion of the image frame generated by GPU1 for prediction state 2 is of lower quality compared to the remainder of the image frame generated by GPU2 for prediction state 2. For example, the portion of the image frame generated by GPU1 has a lower resolution than the resolution of the remainder of the image frame generated by GPU2.
[0132] In one embodiment, GPU2 is of higher quality than GPU1. For example, GPU2 has higher computational performance than GPU1. To explain, GPU2 has a lower response time, or a higher throughput, or a lower utilization of the components of node2, or a higher processing power, or a lower load, or executes a smaller amount of jobs, or executes a smaller number of tasks, or a combination thereof, compared to GPU1.
[0133] In one embodiment, when user input 2 in prediction state 2 is received by node 1 from node A, video encoder VE1 encodes a combination of the image frame generated by GPU1 for prediction state 2 and the remaining image frame generated by GPU2 for prediction state 2 to output one or more encoded image frames, or encodes an image frame in prediction state 2 having a portion generated by GPU1 for prediction state 2 and a remaining portion generated by GPU2 for prediction state 2 to output an encoded image frame. The encoding of the combination of the image frame generated by GPU1 and the remaining image frame generated by GPU2, or the encoding of the image frame generated by combining the portion of the image frame generated by GPU1 for prediction state 2 and the remaining portion of the image frame generated by GPU2 for prediction state 2, occurs after user input 2 is received, rather than before. The network communication device 1 of node 1 accesses encoded audio frames of predicted state 2 from the audio encoder 1 of node 1 as well as encoded image frames of predicted state 2 from the video encoder VE1, applies an external communication protocol to the encoded image frames and encoded audio frames to generate one or more packets, and transmits the packets to the client device 1 via the computer network 102.
[0134] FIG. 4C is a diagram of an embodiment of a system 420 illustrating an allocation of one or more tasks associated with a physics prediction engine from node 1 to node 2. System 420 includes nodes 1 and 2. One or more processors, such as CPU1, CPU1a, and GPU1, of node 1 have reduced processor performance compared to one or more processors of node 2. The one or more processors of node 2 include CPU2, CPU A1, and GPU2. Also, one or more memory devices, such as memory device 1, memory device 1a, and GPU memory device 1, of node 1 have a reduced amount of storage space compared to that of one or more memory devices of node 2. The one or more memory devices of node 2 include memory device 2, memory device 2a, and GPU memory device 2.
[0135] Upon receiving user input 1, CPU1 determines, based on user input 1 and user inputs 2 and 3 (FIG. 3A), that its processing performance is insufficient to determine a portion of predicted state 2. For example, CPU1 determines that more than a threshold amount of processing power is used to determine one or more positions and one or more orientations of predicted state 2 of a virtual object controlled by user A via client device 1 (FIG. 1). As another example, CPU1 determines that more than a predetermined number of components of node 1 are used to determine one or more positions and one or more orientations of predicted state 2 of a virtual object controlled by user A via client device 1.
[0136] Upon so determining, CPU1 transmits user inputs 1, 2, and 3 to CPU2 via internal communication devices 1 and 2. For example, internal communication device 1 applies an internal communication protocol to user inputs 1, 2, and 3 to generate one or more transfer units having user inputs 1, 2, and 3, and transmits the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to obtain user inputs 1, 2, and 3, and transmits user inputs 1, 2, and 3 to CPU2.
[0137] The CPU2 applies a physics prediction engine to determine one or more positions and one or more orientations of a virtual object controlled by the user A via the client device 1 based on the user inputs 1, 2, and 3. The position and orientation of the virtual object are of a predicted state 2. Furthermore, the CPU2 applies an audio prediction engine to determine audio data of a predicted state 2 to be output by the virtual object controlled by the user A via the client device 1 based on the user inputs 1, 2, and 3. The CPU2 stores the position and orientation of the predicted state 2 of the virtual object in the memory device 2 of the node 2.
[0138] Further, CPU 2a of node 2 accesses the predicted state 2 position and orientation of the virtual object from memory device 2 and applies a physics prediction engine to the position and orientation of the virtual object to determine one or more positions and orientations of one or more AIs in a virtual scene having the virtual object. The virtual scene is in predicted state 2. CPU 2a stores the position and orientation of the AI in memory device 2a. Further, CPU 2a applies an audio prediction engine to determine audio data to be output by the AI in the virtual scene based on the position and orientation of the AI. The audio data output by the AI is in predicted state 2.
[0139] GPU2 of node 2 accesses positions and orientations of virtual objects from memory device 2 and positions and orientations of the AI from memory device 2a, and applies a graphics prediction engine, such as a rendering code, to the positions and orientations to generate one or more image frames of the virtual scene in prediction state 2. Video encoder VE2 of node 2 encodes the image frames of the virtual scene to output one or more encoded image frames of the virtual scene in prediction state 2. Additionally, audio encoder 2 of node 2 accesses audio data output by the virtual objects from memory device 2 and audio data output by the AI from memory device 2a, and encodes the audio data to output encoded audio frames of prediction state 2.
[0140] User inputs 2 and 3 are received by node 2 from an input device of client device 1 via computer network 102 (FIG. 1), node A, and node 1. For example, CPU A (FIG. 2A) of node A receives user inputs 2 and 3 from client device 1 via computer network 102, and CPU A provides user inputs 2 and 3 to internal communication device A. Internal communication device A of node A applies an internal communication protocol to user inputs 2 and 3 to generate one or more transfer units and transmits the transfer units to internal communication device 1 of node 1. Internal communication device 1 applies an internal communication protocol to the transfer units to extract user inputs 2 and 3 from the transfer units and transmits user inputs 2 and 3 to CPU 1 of node 1. CPU 1 of node 1 instructs internal communication device 1 to transmit user inputs 2 and 3 to node 2. Internal communication device 1 of node 1 applies an internal communication protocol to user inputs 2 and 3 to generate one or more transfer units and transmits the transfer units to internal communication device 2 of node 2. The internal communication device 2 applies an internal communication protocol to the transfer unit received from the node 1 to extract the user inputs 2 and 3, and transmits the user inputs 2 and 3 to the CPU 2 a of the node 2.
[0141] Upon receiving user inputs 2 and 3 from the input devices of client device 1 via computer network 102 ( FIG. 1 ), node A, and node 1, network communication device 2 of node 2 applies an external communication protocol to the encoded image frames and encoded audio frames of the virtual scene in predicted state 2 to generate one or more packets, and transmits the packets to client device 1 via computer network 102. Upon receiving the packets, client device 1 displays one or more images of the virtual scene in predicted state 2 based on the encoded video frames, and outputs audio based on the encoded audio frames of predicted state 2.
[0142] In one embodiment, upon receiving user input 1, CPU1 applies a physics prediction engine to determine a portion of predicted state 2 that requires low processing performance based on user input 1 and user inputs 2 and 3 (FIG. 3A). For example, CPU1 determines a position and orientation of a virtual object controlled by user A via client device 1 (FIG. 1). Furthermore, CPU1 applies an audio prediction engine to determine audio data to be output by a virtual object controlled by user A via client device 1 (FIG. 1) based on user inputs 1, 2, and 3.
[0143] In the embodiment, CPU1 transmits user inputs 1, 2, and 3 and the position, orientation, and audio data of the virtual object to CPU2 via internal communication devices 1 and 2. For example, internal communication device 1 applies an internal communication protocol to user inputs 1, 2, and 3 and the position, orientation, and audio data of the virtual object to generate one or more transfer units having user inputs 1, 2, and 3, and transmits the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to obtain user inputs 1, 2, and 3 and the position, orientation, and audio data of the virtual object, and transmits user inputs 1, 2, and 3 and the position, orientation, and audio data of the virtual object to CPU2 and CPU2a. CPU2 stores the position and orientation of the virtual object in the predicted state 2 and the audio data output by the virtual object in memory device 2.
[0144] In the same embodiment, CPU2a applies a physics prediction engine to user inputs 1, 2, and 3 and the positions and orientations of virtual objects to determine one or more positions and orientations of one or more AIs in a virtual scene having virtual objects whose positions and orientations are determined by CPU1. The virtual scene is in a predicted state 2. Furthermore, CPU2a applies an audio prediction engine to determine audio data to be output by an AI in the virtual scene in predicted state 2 based on user inputs 1, 2, and 3. The positions and orientations of the AIs in predicted state 2 and the audio data to be output by the AIs in predicted state 2 are stored in memory device 2a. The task of determining the position and orientation of the AIs in predicted state 2 is more computationally intensive compared to the task of determining the position and orientation of the virtual objects in predicted state 2 in the virtual scene. Furthermore, the task of determining the audio data to be output by the AIs in predicted state 2 of the virtual scene is also more computationally intensive than the task of determining the audio data to be output by the virtual objects in predicted state 2 of the virtual scene.
[0145] In that embodiment, GPU2 of node 2 accesses positions and orientations of virtual objects from memory device 2 and positions and orientations of AIs from memory device 2a, and applies a graphics prediction engine, such as rendering code, to the positions and orientations to generate one or more image frames of the virtual scene in prediction state 2. Video encoder VE2 of node 2 encodes the image frames of the virtual scene to output one or more encoded image frames of the virtual scene in prediction state 2. Additionally, audio encoder 2 of node 2 accesses audio data output by the virtual objects from memory device 2 and audio data output by the AI from memory device 2a, and encodes the audio data to output encoded audio frames of prediction state 2.
[0146] In that embodiment, upon receiving user inputs 2 and 3 from input devices of client device 1 via computer network 102 ( FIG. 1 ), node A, and node 1, network communication device 2 of node 2 applies an external communication protocol to the encoded image frames and encoded audio frames of the virtual scene in predicted state 2 to generate one or more packets, and transmits the packets to client device 1 via computer network 102. Upon receiving the packets, client device 1 displays one or more images of the virtual scene in predicted state 2 based on the encoded video frames, and outputs audio based on the encoded audio frames of predicted state 2.
[0147] In one embodiment, any encoding described herein occurs after one or more of user inputs 2 and 3 for predicted state 2 are received. For example, upon receiving user inputs 2 and 3 from the input devices of client device 1 via computer network 102 (FIG. 1), node A, and node 1, video encoder VE2 of node 2 encodes image frames of the virtual scene to output one or more encoded image frames of the virtual scene for predicted state 2. Furthermore, upon receiving user inputs 2 and 3, audio encoder 2 of node 2 accesses audio data output by the virtual object from memory device 2 and audio data output by AI from memory device 2a, encodes the audio data, and outputs encoded audio frames for predicted state 2. Network communication device 2 of node 2 applies an external communication protocol to the encoded image frames and encoded audio frames of the virtual scene for predicted state 2 to generate one or more packets, and transmits the packets to client device 1 via computer network 102. Upon receiving the packets, client device 1 displays one or more images of the virtual scene in prediction state 2 based on the encoded video frames and outputs audio based on the encoded audio frames in prediction state 2.
[0148] In one embodiment, CPU1a has higher processor performance compared to CPU2a. In this embodiment, the position and orientation of the virtual object controlled by user A via client device 1 is transmitted from node 2 to node 1 to determine one or more positions and one or more orientations of one or more AIs in a virtual scene having the virtual object in predicted state 2. Also, audio data output by the virtual object is transmitted from node 2 to node 1 to determine audio data output by the AI in predicted state 2. For example, internal communication device 2 accesses the position, orientation, and audio data of the virtual object in predicted state 2 from memory device 2, applies an internal communication protocol to generate one or more transfer units, and transfers the transfer units to internal communication device 1 of node 1. Internal communication device 1 applies an internal communication protocol to analyze the transfer unit, obtains the position, orientation, and audio data of the virtual object in predicted state 2, and provides the position and orientation to CPU1a of node 1. CPU1a of node 1 determines one or more positions and orientations of one or more AIs based on the position and orientation of the virtual object in the virtual scene in predicted state 2. Furthermore, the CPU 1a determines the audio data to be output by the AI based on the audio data to be output by the virtual object in the virtual scene in the predicted state.
[0149] In one embodiment, GPU1 has higher processor performance than GPU2 of node 2. In this embodiment, instead of GPU2 generating one or more image frames from the position and orientation of a virtual object controlled by user A via client device 1 and the position and orientation of an AI, GPU1 generates an image frame of predicted state 2. GPU1 accesses the position and orientation of a virtual object in predicted state 2 either from memory device 1 if the position and orientation are determined by CPU1, or from memory device 2 if the position and orientation are determined by CPU2.
[0150] In the embodiment, GPU1 accesses the position and orientation of the virtual object in the predicted state 2 from memory device 2 via internal communication devices 1 and 2. For example, GPU1 generates an instruction to retrieve the position and orientation of the virtual object from memory device 2 and sends the instruction to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the instruction to generate one or more transfer units and sends the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to retrieve the instruction and sends the instruction to CPU2. CPU2 accesses the position and orientation of the virtual object from memory device 2 and sends the position and orientation to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the position and orientation of the virtual object to generate one or more transfer units and sends the transfer units to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the transfer units to extract the position and orientation of the virtual object and sends the position and orientation to GPU1.
[0151] Continuing with the embodiment, GPU1 accesses the AI's position and orientation either from memory device 1a if the position and orientation are determined by CPU1a, or from memory device 2a if the position and orientation are determined by CPU2a. GPU1 accesses the AI's position and orientation in predicted state 2 from memory device 2a via internal communication devices 1 and 2. For example, GPU1 generates an instruction to retrieve the AI's position and orientation in predicted state 2 from memory device 2a and sends the instruction to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the instruction to generate one or more transfer units and sends the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to extract the instruction and sends the instruction to CPU2a. CPU2a accesses the AI's position and orientation in predicted state 2 from memory device 2a and sends the position and orientation to internal communication device 2. The internal communication device 2 applies an internal communication protocol to the position and orientation of the AI in the predicted state 2 to generate one or more transfer units and sends the transfer units to the internal communication device 1. The internal communication device 1 applies an internal communication protocol to the transfer units to obtain the position and orientation of the AI in the predicted state 2 and sends the position and orientation to the GPU1.
[0152] In one embodiment, audio encoder 1 has a higher processor performance than audio encoder 2 of node 2. In this embodiment, when audio data output by a virtual object in prediction state 2 is generated by CPU 2, audio encoder 1 generates an instruction to obtain audio data from node 2 and provides the instruction to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the instruction to generate one or more transfer units having the instruction and transmits the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to obtain the instruction and transmits the instruction to CPU 2. CPU 2, in response to the instruction, accesses audio data from memory device 2 and transmits the audio data to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the audio data output by a virtual object in prediction state 2 to generate one or more transfer units and transmits the transfer units to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the transfer units to obtain audio data output by a virtual object in prediction state 2 and transmits the audio data to audio encoder 1. Upon receiving the audio data, audio encoder 1 encodes the audio data to output encoded audio frames. Similarly, if audio data output by one or more AIs for prediction state 2 is generated by CPU 2a, audio encoder 1 generates instructions to retrieve the audio data from memory device 2a of node 2. Upon receiving the audio data from memory device 2a via internal communication devices 1 and 2, audio encoder 1 encodes the audio data output by the AIs to generate one or more encoded audio frames for prediction state 2.
[0153] In one embodiment, audio encoder 1 accesses audio data output by virtual objects controlled by user A via client device 1 from memory device 1 and accesses audio data output by one or more AIs from memory device 2 of node 2. The audio data output by the virtual objects and the audio data output by the AIs are of a virtual scene in predicted state 2.
[0154] In one embodiment, audio encoder 1 accesses audio data output by one or more AIs from memory device 1 and accesses audio data output by virtual objects controlled by user A using client device 1 from memory device 2 of node 2. The audio data output by the virtual objects and the audio data output by the AIs are of a virtual scene in predicted state 2.
[0155] In an embodiment, video encoder 1 has a higher processor performance than video encoder 2 of node 2. In this embodiment, when one or more image frames having a virtual object in prediction state 2 are generated by CPU 2, video encoder 1 generates an instruction to obtain the image frame from node 2 and provides the instruction to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the instruction to generate one or more transfer units having the instruction and transmits the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to extract the instruction and transmits the instruction to GPU 2. GPU 2, in response to the instruction, accesses the image frame from GPU memory device 2 and transmits the image frame to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the image frame output by the virtual object in prediction state 2 to generate one or more transfer units and transmits the transfer units to internal communication device 1. Internal communication device 1 applies an internal communication protocol to the transfer units to extract the image frame of the virtual object in prediction state 2 and transmits the image frame to video encoder 1. Upon receiving an image frame, video encoder 1 encodes the image frame to output an encoded image frame. Similarly, if an image frame with one or more AIs in prediction state 2 is generated by GPU 2, video encoder 1 generates an instruction to retrieve the image frame from GPU memory device 2 of node 2. Upon receiving the image frame from GPU memory device 2 via internal communication devices 1 and 2, video encoder 1 encodes the image frame with AIs to generate one or more encoded image frames.
[0156] In one embodiment, video encoder 1 uses client device 1 to access image frames having a virtual object controlled by user A from GPU memory device 1 and access image frames having one or more AIs from GPU memory device 2 of node 2. The image frames having the virtual objects and AIs are of a virtual scene in prediction state 2.
[0157] In one embodiment, video encoder 1 accesses image frames having one or more AIs from GPU memory device 1 and accesses image frames having virtual objects controlled by user A using client device 1 from GPU memory device 2 of node 2. The image frames having the virtual objects and AIs are of a virtual scene in prediction state 2.
[0158] In an embodiment, the network communication device 1 has a higher processor performance than the network communication device 2 of the node 2. In this embodiment, when one or more encoded image frames of the prediction state 2 are output from the video encoder VE2, the network communication device 1 generates an instruction to obtain the encoded image frames from the node 2 and provides the instruction to the internal communication device 1. The internal communication device 1 applies an internal communication protocol to the instruction to generate one or more transfer units having the instructions and transmits the transfer units to the internal communication device 2. The internal communication device 2 applies an internal communication protocol to the transfer units to extract the instructions and transmits the instructions to the video encoder VE2. In response to the instruction, the video encoder VE2 accesses the encoded image frames from the memory device of the video encoder VE2 and transmits the encoded image frames to the internal communication device 2. The internal communication device 2 applies an internal communication protocol to the encoded image frames of the prediction state 2 to generate one or more transfer units and transmits the transfer units to the internal communication device 1. The internal communication device 1 applies the internal communication protocol to the transfer unit to extract the encoded image frame of the predicted state 2 and transmits the encoded image frame to the network communication device 1. Upon receiving the encoded image frame and after or in response to receiving user inputs 2 and 3, the network communication device 1 applies the external communication protocol to the encoded image frame to generate one or more packets and transmits the packets via the computer network 102 to the client device 1 for displaying one or more images based on the encoded image frame on a display device of the client device 1.
[0159] Similarly, in one embodiment, when one or more encoded audio frames of prediction state 2 are output from audio encoder VE2, network communication device 1 of node 1 generates instructions to obtain encoded audio frames of prediction state 2 from audio encoder AE2 of node 2 via internal communication devices 1 and 2. The instructions are transmitted from node 1 to node 2. Upon receiving the encoded audio frames, and after or in response to receiving user inputs 2 and 3, network communication device 1 applies an external communication protocol to the encoded audio frames to generate one or more packets, and transmits the packets via computer network 102 to client device 1 for outputting audio based on the encoded audio frames.
[0160] FIG. 4D is a diagram of an embodiment of a system 420 illustrating the allocation of one or more tasks related to an AI prediction engine from node 1 to node 2. Upon receiving user input 1, CPU 1a determines to determine a complex AI state for predicted state 2 based on user input 1. As an example, CPU 1a determines to determine multiple collision points of multiple AIs for predicted state 2 based on user inputs 2 and 3 not yet received at node 1. CPU 1a performs the task of determining that multiple AIs will collide with each other based on user inputs 2 and 3 for predicted state 2. The collision of multiple AIs is an example of a simple AI state. As another example, upon receiving user input 1, CPU 1a determines user inputs 2 and 3 for predicted state 2 based on user input 1. CPU 1a further determines that the processing performance of CPU 1a is insufficient to determine a portion of predicted state 2 based on user inputs 2 and 3 (FIG. 3A). To explain, CPU 1a determines that more than a threshold amount of processing power is used to determine one or more positions and one or more orientations of one or more AIs in the virtual scene for predicted state 2. As another example, CPU 1a determines that more than a predetermined number of node 1 components are used to determine one or more positions and one or more orientations of the AI in the virtual scene in predicted state 2.
[0161] Upon determining that a complex AI state should be determined, CPU1a sends user inputs 2 and 3 to CPU2a via internal communication devices 1 and 2 to determine a complex AI state of predicted state 2. For example, internal communication device 1 applies an internal communication protocol to user inputs 1, 2, and 3 to generate one or more transfer units having user inputs 2 and 3, and sends the transfer units to internal communication device 2. Internal communication device 2 applies an internal communication protocol to the transfer units to extract user inputs 2 and 3, and sends user inputs 2 and 3 to CPU2a.
[0162] CPU 2a applies an AI prediction engine to determine a position and orientation of the AI based on user inputs 2 and 3. The position and orientation of the AI is in predicted state 2. Furthermore, CPU 2a applies an audio prediction engine to determine audio data to be output by the AI in the virtual scene in predicted state 2 based on user inputs 2 and 3. CPU 2a stores the position and orientation of the AI in predicted state 2 and the audio data to be output by the AI in memory device 2a of node 2.
[0163] The remaining operations are similar to those described above with reference to FIG. 4C. For example, the GPU2 of the node 2 generates an image frame based on the position and orientation of the virtual object in the virtual scene of the prediction state 2 and the position and orientation of the AI in the virtual scene. Furthermore, the video encoder 2 outputs an encoded image frame by encoding the image frame generated by the GPU2. Also, the audio encoder AE2 encodes the audio data output by the AI and the audio data output by the virtual object in the virtual scene of the prediction state 2 to generate an encoded audio frame. Upon receiving the user inputs 2 and 3, the network communication device 2 generates a packet having the encoded audio frame and the encoded image frame generated by the node 2, and transmits the packet to the client device 1 via the computer network 102 (FIG. 4C).
[0164] In one embodiment, node A (FIG. 2A) may be referred to herein as the first node, node 1 may be referred to herein as the second node, and node 2 may be referred to herein as the third node.
[0165] 5 is a diagram of an embodiment of a client device 500 that is an example of client device 1, 2, or 3 (FIG. 1). Examples of client device 500 include a mobile phone and an HMD. Client device 500 includes multiple components such as a video audio separator 502, a network communication device (NCD) 504, a stream buffer 506, one or more speakers 508, an input device 510, a display device 512, an audio buffer 514, an external digital camera 516, an internal digital camera 518, a video buffer 520, a video audio synchronizer 522, a microphone 524, an inertial sensor 526, a frame encoder 528, a frame decoder 530, an audio digital-to-analog converter (DAC) 532, a video DAC 534, an audio analog-to-digital converter (ADC) 536, a video ADC 538, and a haptic feedback device 540. The haptic feedback device 540 includes a haptic feedback processor 542, a driver 544, and a mechanical element 546. The components of the client device 500 are coupled together via a bus 536. A buffer, as used herein, is used for short-term or temporary storage of data, such as video or audio data. As another example, a buffer is an area in a memory device for temporarily storing data while the data is in motion.
[0166] The external digital camera 516 faces towards the real-world environment of a user, such as user A, B, or C (FIG. 1), and the internal digital camera 518 faces towards the user's eye. Examples of real-world environments include a room, a warehouse, a cubicle, and an enclosed space. Examples of cameras used herein include a wide-angle camera, an infrared sensor camera, a panoramic camera, and the like. A camera is an optical instrument for recording still images or video, which is moving images, for storage in a digital memory device. A camera includes a lens that focuses light reflected from the real-world environment or the user's eye, and includes a camera body that encloses an image capture mechanism for capturing an image of the real-world environment or the user's eye. Examples of external digital cameras 516 include the Playstation Eye® camera manufactured by Sony Computer Entertainment®. Examples of display devices 512 include light-emitting diode (LED) display devices, liquid crystal display (LDC) display devices, liquid crystal on silicon display devices, organic LED (OLED) display devices, plasma screens, and the like.
[0167] The display device 512 includes a microcontroller including a microprocessor, drivers, and memory devices to display a game on a display screen of the display device 512. By way of example, the microprocessor of the display device 512 obtains decoded image frames from the frame decoder 530 and sends signals to the drivers to drive elements, such as LEDs, of the display screen of the display device 512 to display one or more images of the game or other application on the display screen. A driver, as used herein, includes one or more transistors coupled together.
[0168] Examples of inertial sensors 526 include gyroscopes, magnetometers, and accelerometers. Examples of NCDs 504 include network interface cards (NICs) and network interface controllers. Examples of input devices 510 include a keypad, a keyboard, a touch screen, a touch pad, an inertial sensor, one or more buttons, one or more switches, one or more joysticks, etc. In one embodiment, the inertial sensors 526 are part of the input devices 510.
[0169] The frame encoder 528 includes a video encoder that compresses or encodes a plurality of image frames or data within an image frame to output one or more encoded image frames. For example, the video encoder of the frame encoder 528 applies a video interframe compression protocol, such as H.264 and Motion Joint Photographic Experts Group (MJPEG), to compress a plurality of image frames, or applies an intraframe compression protocol to compress data within an image frame to output one or more encoded image frames. As an example, the video encoder of the frame encoder 528 includes a processor and a buffer for storing image frames to be compressed or already compressed by the processor. As another example, the video encoder of the frame encoder 528 includes a processor without including a buffer.
[0170] Furthermore, the frame encoder 528 includes an audio encoder that encodes the audio data and outputs encoded audio frames. For example, the audio encoder of the frame encoder 528 applies an audio interframe compression protocol, such as Advanced Audio Coding (AAC), to the audio data and outputs the encoded audio frames.
[0171] Similarly, the frame decoder 530 includes a video decoder that decodes an encoded image frame or encoded data in an encoded image frame to output one or more decoded image frames or one or more image frames. As an example, the video decoder of the frame decoder 530 applies an inter-frame decompression protocol, such as H.264 and Motion JPEG (MJPEG), to decompress multiple image frames, or applies an intra-frame decompression protocol to decompress data in an image frame to output one or more decoded image frames. As an example, the video decoder of the frame decoder 530 includes a processor and a buffer for storing image frames to be decompressed by the processor or that have already been decompressed. As another example, the frame decoder 530 includes a processor without a buffer.
[0172] The frame decoder 530 also includes an audio decoder that decodes one or more encoded audio frames to output an audio frame or decoded audio frames. For example, the audio decoder of the frame decoder 530 applies an audio interframe decompression protocol, such as AAC, to the encoded audio frames to output decoded audio data or audio data.
[0173] In one embodiment, the video audio separator 502 is a computer program that separates audio data from image frames. In another embodiment, the video audio separator 502 is a hardware circuit, such as an ASIC or PLD, designed and manufactured to distinguish between audio data and image frames.
[0174] In one embodiment, the video audio synchronizer 522 is a computer program that synchronizes video frames with audio data such that any one of the video frames is displayed with an audio output based on the corresponding audio data. In one embodiment, the video audio synchronizer 522 is a hardware circuit such as an ASIC or PLD designed and manufactured to synchronize the display of any one of the video frames with an audio output according to the corresponding audio data.
[0175] In one embodiment, a DAC as used herein is an electronic device that converts digital data, such as video or image frames or digital audio data, into an analog signal, such as a voltage, current, or charge. For example, a video DAC is a hardware circuit that converts image frames into an analog signal for displaying a game on the display device 512, and an audio DAC 532 is an electronic circuit that converts audio data into an analog audio signal for outputting sound through the speaker 508.
[0176] In one embodiment, an ADC as used herein is an electronic device that converts an analog signal into digital data, such as a video frame or image frame or audio data. For example, a video ADC 538 is a hardware circuit that converts an analog signal into an image frame, and an audio ADC 536 is an electronic circuit that converts an analog audio signal into audio data.
[0177] An example of the driver 544 is one or more transistors coupled together. Further, an example of the mechanical part 546 is a vibrating metal plate, or a motor and a metal plate. The motor is coupled to the metal plate via a connection mechanism such as one or more rods, or rods coupled together via gears.
[0178] The stream of data packets is received into the stream buffer 506 via the computer network 102 (FIG. 1) or via a router (not shown) coupled to the computer network 102 and the client device 500. As an example, a router is coupled between the computer network 102 and the client device 500. The NCD 504 accesses the stream of data packets from the stream buffer 506 and depacketizes the stream. For example, the NCD 504 applies an external communication protocol to extract or obtain encoded frames, such as encoded image frames or encoded audio frames, from the packets. To further illustrate, the NCD 504 determines the payload of the packet from the header of the packet and accesses the encoded frame from the payload. The NCD 504 sends the encoded frame to the frame decoder 530 or the frame decoder 530 accesses the encoded frame from the NCD 504. The frame decoder 530 applies a decoding protocol, such as H.264, H.265 / MPEG-H, H.263 / MPEG-4, H.262 / MPEG-2a, or a customized protocol, to decode the encoded frames and output decoded image frames and audio data.
[0179] The decoded image frames are accessed by the video audio separator 502 from the decoder 530 or sent from the decoder 530 to the video audio separator 502. The video audio separator 502 separates the audio data from the decoded image frames. For example, the video audio separator 502 separates the audio data from the decoded image frames according to an identifier provided in the audio data and an identifier provided in the decoded image frames. The video audio separator 502 sends the audio data to the audio buffer 514 and sends the decoded image frames to the video buffer 520. The video audio synchronizer 522 synchronizes the decoded image frames stored in the video buffer 520 with the audio data stored in the audio buffer 514. For example, the video audio synchronizer 522 uses the time when one of the decoded image frames was generated and the time when the corresponding audio data was generated to determine to display the decoded image frame with the sound output according to the corresponding audio data.
[0180] The audio DAC 532 converts the synchronized audio data from digital to analog format to generate an audio signal, which is converted into sound by the speaker 508. The video DAC 534 converts the synchronized image frames from digital to analog format to generate an analog video signal. The display device 512 displays images of the game on a display screen of the display device based on the analog video signal. The images are displayed in synchronization with the sound output by the speaker 508.
[0181] Further, user A, B, or C (FIG. 1) speaks into microphone 524, which converts audio, such as the user's voice, into an audio signal, such as a voltage, current, or electrical signal. Audio ADC 538 converts the audio signal from an analog format to a digital format to generate audio data, which is stored in audio buffer 514. The audio data is accessed from audio buffer 514 by NCD 504 and transmitted over computer network 102 (FIG. 1) to distributed game engine system 106 (FIG. 1). For example, NCD 504 applies an external communication protocol to generate one or more packets that embed the audio data and transmits the packets over computer network 102 to distributed game engine system 106.
[0182] In one embodiment, the frame encoder 528 applies an audio interframe compression protocol to the audio data generated by the audio ADC 538 to output encoded audio frames and provides the encoded audio frames to the NCD 504. The NCD 504 applies an external communication protocol to the encoded audio frames to generate one or more packets and transmits the packets over the computer network 102 to the distributed game engine system 106.
[0183] The internal digital camera 518 captures one or more images of the eyes or face or head or body of the user A, B, or C (FIG. 1). The internal digital camera 518 faces toward the user A, B, or C. For example, when the client device 500 is an HMD worn by the user A, the internal digital camera 518 faces toward the eyes of the user A. As another example, when the client device 500 is a smartphone, the internal digital camera 518 is a front camera on the same side or surface of the client device 500 on which the display screen of the display device 512 is located. As yet another example, the internal digital camera 518 is a front camera that faces toward the hand of the user A or the arm of the user A to capture the gesture made by the user A. Similarly, the external digital camera 516 captures one or more images of the real-world environment surrounding the user. As an example, when the client device 500 is a smartphone, the external digital camera 516 is a rear camera on the opposite side or surface of the client device 500 compared to the side on which the display screen of the display device 512 is located. The images captured by the digital cameras 516 and 518 are stored in a video buffer 520. In one embodiment, the terms image and image frame are used interchangeably herein. In some embodiments, the images captured by the digital cameras 516 and 518 are converted from digital to analog format by a video DAC 534 and sent to display the images on the display device 512.
[0184] Images captured by the digital cameras 516 and 518 are accessed from the video buffer 520 by the NCD 504 to generate one or more data packets having the images. The NCD 504 transmits the data packets over the computer network 102 to the distributed game engine system 106 (FIG. 1). In one embodiment, images captured by the digital cameras 516 and 518 are accessed by a frame encoder 530. The frame encoder 530 applies an inter-frame or intra-frame compression protocol to the images to generate one or more encoded image frames and provides the encoded image frames to the NCD 504 over the bus 536. The NCD 504 applies an external communication protocol to the encoded image frames to generate one or more data packets from the encoded image frames and transmits the data packets over the computer network 102 to the distributed game engine system 106.
[0185] The inertial sensor 526 generates inertial sensor data, such as the acceleration of user A's body part, the rotational velocity of user A's body part, the orientation of user A's body part, or a combination thereof. If the client device 500 includes a handheld controller, the inertial sensor 526 generates inertial sensor data related to the movement of the handheld controller. The inertial sensor data is provided by the inertial sensor 526 to the NCD 504 via the bus 536. The NCD 504 generates one or more packets embedding the inertial sensor data and transmits the packets via the computer network 102 to the distributed game engine system 106 (FIG. 1).
[0186] Additionally, NCD 504 receives one or more selections made by users A, B, or C by selecting one or more buttons, or by making one or more gestures, or by moving one or more joysticks of input device 510 to generate input data and provide the input data to NCD 504 via bus 536. NCD 504 applies an external communications protocol to the input data to generate one or more data packets having the input data and transmits the data packets over computer network 102 to distributed game engine system 106.
[0187] In one embodiment, data packets generated by the NCD 504 are transmitted through a router (not shown) coupled to the NCD 504 via a wireless connection, such as a Wi-Fi® or Bluetooth® connection. The router receives the data packets and determines a destination address from the data packets. The router routes the data packets through the computer network 102 (FIG. 1) to one or more nodes of the distributed game engine system 106 to which the destination address is assigned.
[0188] Similarly, in one embodiment, data packets generated by the distributed game engine system 106 are transmitted through a router (not shown) coupled between the computer network 102 and the client device 500. The router is coupled to the NCD 504 of the client device 500 via a wireless connection. The router receives the data packets from the distributed game engine system 106 and determines a destination address from the data packets. The router routes the data packets via the wireless connection to the NCD 514 of the client device 500 to which the destination address is assigned.
[0189] In one embodiment, the input device 510 is part of a device separate from the client device 500. For example, the input device 510 is part of a handheld controller, such as a Sony® Move® controller or a Sony® DualShock® controller or a game controller or a joystick controller. The handheld controller is coupled to an internal communication device (not shown) of the client device 500 via a wireless or wired connection, such as a parallel data transfer cable or a serial data transfer cable or a USB cable. An example of the internal communication device is a data transfer device that communicates data with the input device 510 by applying a wireless protocol, such as Bluetooth® or Wi-Fi®. As an example, the internal communication device of the client device 500 is coupled to the NCD 504 via a bus 536. The NCD 504 accesses input data received by the internal communication device from the handheld controller, applies an external communication protocol to generate one or more packets, and transmits the packets to the distributed game engine system 106 via the computer network 102.
[0190] In one embodiment, the client device 500 excludes an internal digital camera 518 .
[0191] In one embodiment, external digital camera 516 is positioned outside client device 500 to capture images of movements of body parts of user A, B, or C, such as arms, hands, head, or combinations thereof, or movements of an HMD worn by the user, or movements of a handheld controller held by the user.
[0192] In one embodiment, the terms packet and data packet are used interchangeably herein.
[0193] In one embodiment, the haptic feedback device 540, the microphone 524, the input device 510, and the inertial sensor 526 are components of the handheld controller. Also, in this embodiment, the display device 512 and the speaker 508 are components of the television. Furthermore, the external digital camera 516 is a separate device from the handheld controller, the game console, and the television. This embodiment excludes the internal digital camera 518. Furthermore, in this embodiment, the remaining components of the client device 500, such as the frame encoder 528, the video audio separator 502, the NCD 504, the frame decoder 530, the audio DAC 532, the stream buffer 506, the video DAC 534, the audio buffer 514, the video ADC 538, the audio ADC 536, the video audio synchronizer 522, and the video buffer 520, are components of the game console. The game console is coupled to the handheld controller via a wired or wireless connection. The game console is also coupled to a display device via a wireless connection or a High Definition Multimedia Interface (HDMI®) connection.
[0194] In one embodiment, the haptic feedback device 540, the microphone 524, the input device 510, the inertial sensor 526, the display device 512, the speaker 508, the external digital camera 516, and the internal digital camera 518 are components of an HMD worn on the head of a user A, B, or C. Furthermore, in this embodiment, the remaining components of the client device 500, such as the frame encoder 528, the video audio separator 502, the NCD 504, the frame decoder 530, the audio DAC 532, the stream buffer 506, the video DAC 534, the audio buffer 514, the video ADC 538, the audio ADC 536, the video audio synchronizer 522, and the video buffer 520, are components of a game console. The game console is coupled to the HMD via a wired or wireless connection.
[0195] FIG. 6 is a flow diagram conceptually illustrating various operations performed to stream cloud video games to client device 500 (FIG. 5) according to an embodiment of the present disclosure. One or more game servers 602 of distributed game engine system 106 run video games and generate raw (uncompressed) video 604 and audio 606. Servers are sometimes referred to herein as nodes. An example of a server is a game console or a server blade. Video 604 and audio 606 are encoded by one or more game servers 602 for streaming, as shown by reference number 608 in the figure. Encoding compresses the video and audio streams, reducing bandwidth usage and optimizing the gaming experience. Example encoding formats include H.265 / MPEG-H, H.264 / MPEG-4, H.263 / MPEG-4, H.262 / MPEG-2, WMV, VP6 / 7 / 8 / 9, etc.
[0196] The encoded audio 610 and encoded video 612 are further packetized by a streaming engine, such as NCD 504 (FIG. 5), into data packets or network packets, as indicated by reference numeral 614, for transmission over a computer network 102, such as the Internet. In some embodiments, the network packet encoding process also employs a data encryption process to enhance data security. In the illustrated implementation, the audio packets 616 and video packets 618 are generated by applying an external communication protocol for transport over the computer network 102.
[0197] The one or more game servers 602 further generate haptic feedback data 622, which is also packetized into data packets for network transmission. In the illustrated implementation, haptic feedback packets 624 are generated for transport over the computer network 102.
[0198] The aforementioned operations of generating raw video and audio and haptic feedback data are performed by one or more game servers 602 in one or more data centers of the distributed game engine system 106, and the operations of encoding the video and audio and packetizing the encoded audio / video and haptic feedback data for transport are performed by a streaming engine in one or more data centers. As indicated by reference numeral 620, the audio, video, and haptic feedback packets are transported over the computer network 102. As indicated by reference numeral 626, the audio packets 616, the video packets 618, and the haptic feedback packets 624 are decomposed by the client device 500, e.g., parsed, to extract the encoded audio 628, the encoded video 630, and the haptic feedback data 632 from the data packets at the client device 500. If the data is encrypted, decryption of the data is also performed. The encoded audio 628 and encoded video 630 are then decoded by the client device 500 to generate client-side raw audio and video data for display on the display device 512 of the client device 500, as indicated by reference numeral 634. The haptic feedback data 632 is processed by a haptic feedback processor 542 (FIG. 5) of the client device 500 to generate haptic feedback effects at a handheld controller device 642 or at an interface device capable of rendering haptic effects, such as an HMD. One example of a haptic effect is a vibration or rumble at the controller device 642 of an HMD.
[0199] It will be appreciated that, since the video game responds to user input, a similar procedural flow as described above is performed in the reverse direction from the client device 500 to the one or more servers 502 for transmitting and processing the user input. As shown, a controller device 642 or another input device or combination thereof generates input data 644. The input data 644 is packetized at the client device 500 to generate input data packets 646 for transport over the computer network 102 to the one or more servers 602. The input data packets 646 are decompressed and reconstructed by the one or more servers 602 to define input data 648 at the data center. The input data 648 is provided to the one or more servers 602 which process the input data 648 to generate one or more states, as described above.
[0200] During transport of the audio packets 616, video packets 618, and haptic feedback packets 624 over the computer network 102, in some embodiments, the transmission of data over the computer network 102 is monitored to ensure quality of service. For example, as indicated by reference numeral 650, network conditions of the computer network 102, including both upstream and downstream network bandwidth, are monitored and the game streaming is adjusted in response to changes in available bandwidth. That is, as indicated by reference numeral 652, the encoding and decoding of network packets is controlled based on the current network conditions.
[0201] FIG. 7 is a block diagram of an embodiment of a game console 700 compatible for interfacing with the display device 512 of the client device 500 and capable of communicating with the distributed engine system 106 (FIG. 1) via the computer network 102. The game console 700 is located in a data center A or at a location where a user A is located. In some embodiments, the game console 700 is used to run a game that is displayed on an HMD. The game console 700 includes various peripherals that can be connected to the game console 700. The game console 700 includes a Cell processor 728, a dynamic random access memory (XDRAM) unit 726, a Reality Synthesizer® graphics processor unit 730 with a dedicated video random access memory (VRAM) unit 732, and an input / output (I / O) bridge 734. The game console 700 also has a Blu Ray® disc read-only memory (BD-ROM) optical disc reader 740 for reading from a disc 740a, and a removable slot-in hard disk drive (HDD) 736, accessible through an I / O bridge 734. Optionally, the game console 700 also includes a memory card reader 738 for reading CompactFlash® memory cards, Memory Stick® memory cards, and the like, which is also accessible through the I / O bridge 734. The I / O bridge 734 also connects to a Universal Serial Bus (USB) 2.0 port 724, a Gigabit Ethernet port 722, an IEEE 802.11b / g wireless network (Wi-Fi®) port 720, and a Bluetooth® wireless link port 718 capable of supporting Bluetooth® connectivity.
[0202] In operation, the I / O bridge 734 handles all wireless, USB, and Ethernet data, including data from the game controllers 642 and / or 703, and from the HMD 705. For example, when User A is playing a game generated by execution of a portion of the game code, the I / O bridge 734 receives input data from the game controllers 642 and / or 703 and / or from the HMD 705 over the Bluetooth link and directs the input data to the Cell processor 728, which updates the current state of the game accordingly. As an example, a camera in the HMD 705 captures User A's gestures and generates an image representing the gestures. The image is an example of input data. Each game controller 642 and 703 is an example of a handheld controller (HHC).
[0203] Wireless, USB, and Ethernet ports provide connectivity to game controllers 642 and 703 and HMD 705 as well as other peripheral devices such as, for example, a remote control 704, a keyboard 706, a mouse 708, a portable entertainment device 710 such as, for example, a Sony Playstation Portable entertainment device, a video camera such as, for example, an EyeToy video camera 712, a microphone headset 714, and a microphone 715. In some embodiments, such peripherals are wirelessly connected to game console 700, for example, portable entertainment device 710 communicating via a Wi-Fi ad-hoc connection and microphone headset 714 communicating via a Bluetooth link.
[0204] The provision of these interfaces means that the game console 700 may also support other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, voice over Internet Protocol (IP) telephones, mobile phones, printers, and scanners.
[0205] Additionally, a legacy memory card reader 716 is connected to the game console 700 via a USB port 724 to allow for reading of memory cards 748 of the type used by the game console 700. The game controllers 642 and 703 and HMD 705 are operable to communicate wirelessly with the game console 700 via a Bluetooth® link 718 or to be connected to the USB port 724 to also receive power to charge the batteries of the game controllers 642 and 703 and HMD 705. In some embodiments, the game controllers 642 and 703 and HMD 705 each include memory, a processor, a memory card reader, permanent memory such as, for example, flash memory, light emitters such as, for example, illuminated spherical portions, light emitting diodes (LEDs), or infrared lights, microphones and speakers for ultrasonic communication, an acoustic chamber, a digital camera, an internal clock, a recognizable shape such as, for example, a spherical portion facing the game console 700, and wireless devices using protocols such as, for example, Bluetooth®, Wi-Fi®, and the like.
[0206] Game controller 642 is a controller designed to be used with both hands of user A, and game controller 703 is a one-handed controller with an attachment. HMD 705 is designed to fit on top of user A's head and / or in front of the eyes. In addition to one or more analog joysticks and traditional control buttons, each game controller 642 and 703 is subject to three-dimensional positioning. Similarly, HMD 705 is subject to three-dimensional positioning. As a result, in some embodiments, gestures and movements made by user A using game controllers 642 and 703 and HMD 705 are translated as inputs to the game in addition to or instead of traditional button or joystick commands. Optionally, other wireless-enabled peripheral devices, such as a Playstation® portable device, are used as controllers. In the case of a Playstation® portable device, additional game or control information, such as control instructions or remaining lives, is provided on the display screen of the device. In some embodiments, other alternative or supplemental control devices may be used, such as a dance mat (not shown), a light gun (not shown), a handle and pedals (not shown), or a custom controller, an example of which includes one or several large buttons (also not shown) for a quick-answer quiz game.
[0207] The remote control 704 is also operable to communicate wirelessly with the game console 700 via a Bluetooth® link 718. The remote control 704 includes controls suitable for operating a Blu Ray® disc BD-ROM reader 740, as well as controls suitable for navigating the disc contents.
[0208] The Blu Ray® disc BD-ROM reader 740 is operable to read conventional recorded and recordable CDs, and so-called Super Audio CDs, as well as CD-ROMs compatible with the game console 700. The Blu Ray® disc BD-ROM reader 740 is also operable to read conventional recorded and recordable DVDs, as well as Digital Video Disc ROMs (DVD-ROMs) compatible with the game console 700. The Blu Ray® disc BD-ROM reader 740 is further operable to read BD-ROMs compatible with the game console 700, as well as conventional recorded and recordable Blu-Ray® discs.
[0209] The game console 700 is operable to provide audio and video generated or decoded via the reality synthesiser graphics unit 730 to a display and sound output device 742, such as a monitor or television set having a display screen 744 and one or more loudspeakers 746, through an audio connector 750 and a video connector 752, or to provide audio and video to a display device of the HMD 705 via a Bluetooth wireless link port 718. The audio connector 750, in various embodiments, includes conventional analog and digital outputs, while the video connector 752 variously includes component video, S-video, composite video, and one or more High Definition Multimedia Interface (HDMI) outputs. As a result, the video output may be in a format such as Phase Shift Keying (PAL) or National Television Standards Committee (NTSC), or in high definition at 2220p, 1080i, or 1080p. Audio processing, e.g., generation, decoding, etc., is performed by the cell processor 708. The gaming console 700 operating system supports decoding of Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and 7.1 surround sound from Blu-Ray® discs.
[0210] In some embodiments, the video camera, such as video camera 712, includes a single charge-coupled device (CCD), an LED indicator, and a hardware-based real-time data compression and encoding device, whereby compressed video data is transmitted in a suitable format, such as an in-picture based motion picture expert group (MPEG) standard, for decoding by the game console 700. The LED indicator of the video camera 712 is arranged to illuminate in response to suitable control data from the game console 700, for example to indicate adverse lighting conditions. Some embodiments of the video camera 712 variously connect to the game console 700 via a USB, Bluetooth®, or Wi-Fi® communication port. Various embodiments of the video camera include one or more associated microphones and can also transmit audio data. In some embodiments of the video camera, the CCD has a resolution suitable for high-definition video capture. In use, images captured by the video camera are incorporated into the game or interpreted as game control inputs. In another embodiment, the video camera is an infrared camera suitable for detecting infrared light.
[0211] In various embodiments, appropriate software, such as device drivers, are provided to enable successful data communication with a peripheral device, such as a video camera or remote control, through one of the communication ports of the game console 700.
[0212] In some embodiments, the aforementioned system devices including the game console 700, the handheld controller, and the HMD 705 enable the HMD 705 to display and capture video of an interactive session of the game. The system device initiates an interactive session of the game, where the interactive session defines an interaction between User A and the game. The system device further determines an initial position and orientation of the handheld controller and / or the HMD 705 operated by User A. The game console 700 determines a current state of the game based on User A's interaction with the game. The system device tracks the position and orientation of the handheld controller and / or the HMD 705 during User A's interactive session with the game. The system device generates a spectator video stream of the interactive session based on the current state of the game and the tracked position and orientation of the handheld controller and / or the HMD 705. In some embodiments, the handheld controller renders the spectator video stream on a display screen of the handheld controller. In various embodiments, the HMD 705 renders the spectator video stream on a display screen of the HMD 705.
[0213] Referring to FIG. 8, a diagram showing components of an HMD 802 is shown. The HMD 802 is an example of the HMD 705 (FIG. 7). The HMD 802 includes a processor 800 for executing program instructions. A memory device 802 is provided for storage purposes. Examples of the memory device 802 include volatile memory, non-volatile memory, or a combination thereof. A display device 804 is included that provides a visual interface for user A (FIG. 1) to view, such as displaying image frames. A battery 806 is provided as a power source for the HMD 802. A motion detection module 808 includes any of various types of motion sensing hardware, such as a magnetometer 810, an accelerometer 88, and a gyroscope 814.
[0214] Accelerometers are devices that measure acceleration and gravity-induced reaction forces. Single-axis and multi-axis models are available to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 812 are used to provide the direction of gravity, which provides an absolute reference for two angles, such as world-space pitch and world-space roll.
[0215] The magnetometer measures the strength and direction of the magnetic field near the HMD 802. In some embodiments, three magnetometers 810 are used in the HMD 802 to ensure an absolute reference for the world space yaw angle. In various embodiments, the magnetometers are designed to span the Earth's magnetic field of ±80 microstellar. The magnetometers are affected by metal and provide a yaw measurement that is monotonic with respect to the actual yaw. In some embodiments, the magnetic field is distorted by metal in the real world environment, which causes distortion in the yaw measurement. In various embodiments, this distortion is calibrated out using information from other sensors, such as gyroscope 814, camera 816, etc. In one embodiment, an accelerometer 812 is used in conjunction with magnetometer 810 to obtain the tilt and azimuth of the HMD 802.
[0216] A gyroscope is a device that measures or maintains orientation based on the principles of angular momentum. In one embodiment, three gyroscopes, instead of gyroscope 814, provide information about motion about the x, y, and z axes based on inertial sensing. The gyroscopes aid in the detection of high speed rotations. However, in some embodiments, the gyroscopes drift over time in the absence of an absolute reference. This triggers a periodic gyroscope reset, which can be done using other available information, such as visual tracking of objects, position / orientation determination based on accelerometers, magnetometers, etc.
[0217] A camera 816 is provided to capture images and image streams of a real-world environment surrounding user A, e.g., a room, a cabin, a natural environment, etc. In various embodiments, multiple cameras are included in the HMD 802, including a rear-facing camera that faces away from user A, e.g., when user A is looking at a display of the HMD 802, and a front-facing camera that faces towards user A, e.g., when user A is looking at a display of the HMD 802, etc. Additionally, in some embodiments, a depth camera 818 is included in the HMD 802 to sense depth information of objects in the real-world environment.
[0218] The HMD 802 includes a speaker 820 for providing audio output and, in some embodiments, a microphone 822 for capturing audio from the real world environment, including sounds from the surrounding environment, speech made by User A, etc. The HMD 802 includes a haptic feedback module 824, such as a vibration device, for providing haptic feedback to User A. In one embodiment, the haptic feedback module 824 may generate movement and / or vibration of the HMD 802 to provide haptic feedback to User A.
[0219] An LED 826 is included as a visual indicator of the status of the HMD 802. For example, an LED may indicate battery level, power on, etc. A card reader 828 is included to allow the HMD 802 to read and write information from and to a memory card. A USB interface 830 is included as an example of an interface to allow connection of peripheral devices, or connection to other devices, such as other portable devices, computers, etc. In various embodiments of the HMD 802, any of a variety of types of interfaces may be included to allow greater connectivity of the HMD 802.
[0220] A Wi-Fi module 832 is included to enable connection to the Internet via wireless network technology. The HMD 802 also includes a Bluetooth module 834 to enable wireless connection to other devices. In some embodiments, a communication link 836 is also included for connection to other devices. In one embodiment, the communication link 836 utilizes infrared transmission for wireless communication. In other embodiments, the communication link 836 utilizes any of a variety of wireless or wired transmission protocols for communication with other devices.
[0221] Input buttons / sensors 838 are included to provide an input interface for User A (FIG. 1). Any of a variety of types of input interfaces may be included, such as buttons, touchpads, joysticks, trackballs, etc. In various embodiments, an ultrasonic communication module 840 is included in the HMD 802 to facilitate communication with other devices via ultrasonic technology.
[0222] Biosensor 842 is included to enable detection of physiological data from a user. In one embodiment, biosensor 842 includes one or more dry electrodes to detect the user's bioelectrical signals through the user's skin.
[0223] The aforementioned components of the HMD 802 have been described merely as example components that may be included within the HMD 802. In various embodiments, the HMD 802 may or may not include some of the various components described above.
[0224] 9 illustrates an embodiment of an information service provider (INSP) architecture. INSP 902 provides many information services to users A, B, and C and user D who are geographically distributed and connected via computer network 102, such as a LAN, a WAN, or a combination thereof. An example of a WAN includes the Internet, and an example of a LAN includes an intranet. User A operates a client device 920-1, user B operates another client device 920-2, user C operates yet another client device 920-3, and user D operates another client device 920-4. Client device 920-1 is an example of client device 1 (FIG. 1), client device 920-2 is an example of client device 2 (FIG. 1), and client device 920-3 is an example of client device 3 (FIG. 1).
[0225] In some embodiments, each client device 920-1, 920-2, 920-3, and 920-4 includes a central processing unit (CPU), a display, and an input / output (I / O) interface. Examples of each client device 920-1, 920-2, 920-3, and 920-4 include a personal computer (PC), a mobile phone, a netbook, a tablet, a gaming system, a personal digital assistant (PDA), a gaming console 700 and a display device, an HMD 802 (FIG. 11), a gaming console 700 and an HMD 802, a desktop computer, a laptop computer, a smart television, etc. In some embodiments, the INSP 902 recognizes the type of client device and adjusts the communication method to be used.
[0226] In some embodiments, an INSP provides one type of service, such as stock price updates, or a variety of services, such as broadcast media, news, sports, games, etc. Furthermore, the services provided by each INSP are dynamic, i.e., services can be added or removed at any time. Thus, the INSPs that provide a particular type of service to a particular individual may change over time. For example, when client device 920-1 is in user A's residence, client device 920-1 is served by an INSP near client device 920-1, and when user A moves to a different city, client device 920-1 is served by a different INSP. The residence INSP forwards the requested information and data to the new INSP, so that the information "follows" client device 920-1 to the new city, bringing the data closer to client device 920-1 and making it easier to access. In various embodiments, a master-server relationship is established between a master INSP that manages the information for client device 920-1 and a server INSP that interfaces directly with client device 920-1 under control from the master INSP. In some embodiments, as client device 920-1 moves around the world, data is routed from one ISP to another, such that the INSP that is better positioned to provide services to client device 920-1 is the INSP that delivers those services.
[0227] The INSP 902 includes an Application Service Provider (ASP) 909 that provides computer-based services to customers via the computer network 102. Software provided using the ASP model is sometimes referred to as on-demand software or software as a service (SaaS). A simple form of providing access to computer-based services, such as customer relationship management, is by using a standard protocol such as the Hypertext Transfer Protocol (HTTP). The application software resides on a vendor's server and is accessed by each client device 920-1, 920-2, 920-3, and 920-4 through a web browser, such as using Hypertext Markup Language (HTML), by special purpose client software provided by the vendor, and / or other remote interfaces, such as thin clients.
[0228] Services delivered over large geographical regions often use cloud computing, a computing paradigm in which dynamically scalable and often virtualized resources are provided as a service over a computer network 102. Users A, B, C, and D do not need to be experts in the technical infrastructure of the "cloud" that supports them. In some embodiments, cloud computing is divided into different services such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services often provide common business applications online that are accessed from a web browser, but the software and data are stored on servers. The term cloud is used as a metaphor for the computer network 102, using, for example, servers, storage, and logic, based on how the computer network 102 is depicted in a computer network diagram, and is an abstraction of the complex infrastructure that the computer network 102 hides.
[0229] Additionally, the INSP 902 includes a game processing provider (GPP) 910, also referred to herein as a game processing server, which is used by the client devices 920-1, 920-2, 920-3, and 920-4 to play single and multiplayer video games. Most video games played on the computer network 102 operate through a connection to a game server. Typically, the games use a dedicated server application that collects data from the client devices 920-1, 920-2, 920-3, and 920-4 and distributes the data to other clients operated by other users. This is more efficient and effective than a peer-to-peer configuration, but uses a separate server that hosts the server application. In some embodiments, the GPP 910 establishes communication between the client devices 920-1, 920-2, 920-3, and 920-4, which exchange information without further reliance on the centralized GPP 910.
[0230] A dedicated GPP is a server that runs independently of the clients. Such servers usually run on dedicated hardware located in a data center, providing more bandwidth and dedicated processing power. Dedicated servers are the method of hosting game servers for most PC-based multiplayer games. Massively multiplayer online games run on dedicated servers, usually hosted by the software company that owns the game title, allowing the dedicated server to control and update the content.
[0231] The broadcast processing server (BPS) 912, also referred to herein as a broadcast processing provider, distributes the audio or video signals to the audience. Broadcasting to a very small audience is sometimes referred to as narrowcasting. The final leg of broadcast distribution is how the signal reaches the client devices 920-1, 920-2, 920-3, and 920-4, which in some embodiments is distributed over the air to antennas and receivers, similar to radio or television stations, or through "wireless cable" via cable television or cable radio or stations. The computer network 102 also brings either the over the air or television signals to the client devices 920-1, 920-2, 920-3, and 920-4, in various embodiments, particularly with multicasting, which allows for sharing of signals and bandwidth. Historically, broadcasts have been bounded by geographical regions, e.g., national broadcast, local broadcast, etc., in some embodiments. However, with the proliferation of high speed Internet, content can reach almost every country in the world, so broadcasts are not defined by geography.
[0232] A storage service provider (SSP) 914 provides computer storage space and related management services. The SSP 914 also provides periodic backup and archiving. By providing storage as a service, the client devices 920-1, 920-2, 920-3, and 920-4 use more storage than if storage was not used as a service. Another major advantage is that the SSP 914 includes a backup service, and the client devices 920-1, 920-2, 920-3, and 920-4 do not lose data in the event of a hard drive failure. Further, in some embodiments, multiple SSPs have full or partial copies of data received from client devices 920-1, 920-2, 920-3, and 920-4, allowing client devices 920-1, 920-2, 920-3, and 920-4 to access the data in an efficient manner regardless of where client devices 920-1, 920-2, 920-3, and 920-4 are located or the type of client. For example, User A accesses personal files via a computer at home and via a mobile phone while User A is on the move.
[0233] The communications provider 916 provides connectivity to the client devices 920-1, 920-2, 920-3, and 920-4. One type of communications provider 916 is an Internet Service Provider (ISP) that provides access to the computer network 102. The ISP connects the client devices 920-1, 920-2, 920-3, and 920-4 using a data transmission technology suitable for delivering Internet protocol datagrams, such as dial-up, digital subscriber line (DSL), cable modem, fiber, wireless, or a dedicated high-speed interconnect. The communications provider 916 also provides messaging services, such as email, instant messaging, and short message service (SMS) text, in some embodiments. Another type of communications provider is a network service provider (NSP) that sells bandwidth or network access by providing direct backbone access to the computer network 102. Examples of network service providers include telecommunications carriers, data carriers, wireless communication providers, Internet service providers, cable television operators providing high-speed Internet access, and the like.
[0234] Data exchange 918 interconnects several modules in INSP 902 and connects these modules to client devices 920-1, 920-2, 920-3, and 920-4 via computer network 102. Data exchange 918, in various embodiments, covers a small area where all modules of INSP 902 are in close proximity, or covers a large geographic area when the various modules are geographically dispersed. For example, data exchange 902 may include a high-speed Gigabit Ethernet in a cabinet in a data center, or an intercontinental virtual LAN.
[0235] In one embodiment, one or more nodes described herein determine an amount of misprediction that a predicted user input, such as User Input 2 (FIG. 3A), will be received and use a first threshold to trigger a full rendering or generation of a frame, such as an image frame or an audio frame. The amount of misprediction is an example of a prediction level that is greater than the first threshold. The first threshold is when it is determined that the current image frame or the current audio frame was incorrectly predicted or generated based on the predicted user input and another image frame or audio frame should be fully generated or rendered.
[0236] In an embodiment, there is a second threshold below the first threshold, the second threshold being used to trigger the rendering or generation of the next frame. For example, one or more of the nodes described herein detect that they are on the cusp of a misprediction and do not preemptively generate or render the next frame, such as an image frame or an audio frame, based on the next predicted user input. The next predicted user input is contiguous with the predicted user input. The generation or rendering of the current image frame or the current audio frame is based on a prediction of a predetermined level of the predicted user input. For example, the level of prediction falls below the first threshold and one or more nodes predict that the generation or rendering of the next frame will not be based on the predetermined predicted level of the next predicted user input. Each of the first threshold and the second threshold are examples of a predetermined prediction level.
[0237] Also, in one embodiment, if one or more nodes described herein determine that a prediction of receipt of a predicted user input does not meet, e.g., exceed, a predetermined prediction level, the one or more nodes temporarily suspend or do not operate to generate or render image frames based on the predicted user input. While operation is suspended, the one or more nodes generate or render image frames and audio frames based on actual user input, such as user input 1' (FIG. 3A). While operation is suspended, the one or more nodes learn predictions to achieve a predetermined level of prediction. As an example, operation is suspended and / or learning occurs for a period of time, for the duration of a game session, or until a known event occurs in the game, or until the prediction engine makes a prediction that is higher than the predetermined level. The prediction engine is a computer program executed by one or more of the nodes. An example of a known event is the end of a game scene, which is a scene that is difficult to predict at a predetermined prediction level. An example of a difficult scene is a scene with movement or manipulation of a virtual object or virtual character that is greater than a predetermined amount, or a combination thereof. After a period of time, or after the game session ends, or after a known event occurs, or after the prediction engine predicts to greater than a predetermined level, one or more of the nodes described herein return to predicting user input and generating image and audio frames based on the predicted user input.
[0238] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.
[0239] The embodiments described in this disclosure may be implemented with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. In one implementation, the embodiments described in this disclosure are practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.
[0240] With the foregoing embodiments in mind, it should be understood that in one implementation, the embodiments described in this disclosure employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments described in this disclosure are useful machine operations. Some embodiments described in this disclosure also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, in one embodiment, various general-purpose machines are used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
[0241] In an embodiment, some embodiments described in this disclosure are embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that stores data that is subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), recordable CD (CD-R), rewritable CD (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, etc. As an example, the computer readable medium includes computer readable tangible media distributed over network coupled computer systems such that the computer readable code is stored and executed in a distributed fashion.
[0242] Additionally, although some of the above embodiments are described with respect to a gaming environment, in some embodiments other environments are used instead of a game, such as a video conferencing environment.
[0243] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals related to processing, so long as the processing of the overlay operations is performed in the desired manner.
[0244] Although the foregoing embodiments described in this disclosure have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope and equivalents of the appended claims.
Claims
1. 1. A method for reducing latency during game execution, comprising: a first node generating a current state of the game from user input; the first node predicting a next state of the game based on the user input and one or more predicted user inputs; said first node providing a task of determining a complex physical state to a second node; said first node having the task of creating simple physics states when predicting said next state of said game; Including, The method, wherein the first node includes a first central processing unit and the second node includes a second central processing unit, the second central processing unit having a processor performance greater than the processor performance of the first central processing unit for determining the complex physical state.
2. The method of claim 1 , wherein the complex physics state includes contact points for a collision of two virtual objects of the game, and the simple physics state includes the probability that the collision will occur.
3. generating one or more predicted frames from the next state; determining whether the one or more predicted user inputs are received over a computer network; in response to receiving the one or more predictive user inputs, transmitting the one or more predictive frames to reduce the latency during execution of the game; The method of claim 1 further comprising:
4. 2. The method of claim 1 , wherein the predicting the next state of the game and the generating the one or more predicted frames occur before the one or more predicted user inputs are received to reduce the latency in the execution of the game.
5. wherein the determining whether the one or more predicted user inputs have been received includes determining that a user input different from the one or more predicted user inputs has been received over a computer network, the method comprising: generating a game state from the user input that differs from the one or more predicted user inputs; generating one or more frames from the game state; transmitting the one or more frames of the game state instead of transmitting the one or more predicted frames; and The method of claim 3 further comprising:
6. 6. The method of claim 5, wherein the user input different from the one or more predicted user inputs is received during the predicting the next state of the game, and the generating of the game state from the user input and the generating of the one or more frames from the game state are given a higher priority than the predicting the next state of the game.
7. 6. The method of claim 5, wherein the user input that is different from the one or more predicted user inputs is received during the generating of the one or more predicted frames, and wherein the generating of the game state from the user input and the generating of the one or more frames from the game state are given a higher priority than the generating of the one or more predicted frames.
8. The method of claim 1 , wherein the predicting the next state of the game is based on an output of the current state of the game.
9. generating one or more predicted frames from the next state; determining that a user input different from the one or more predicted user inputs is received over a computer network; identifying one or more elements of a game state of the game that are not included within the next state; generating the one or more elements of the game state from the user input that differs from the one or more predicted user inputs; generating one or more frames from the one or more elements of the game state; 2. The method of claim 1 , further comprising: transmitting the one or more frames of the one or more elements of the game state to enable display of one or more composite images, the one or more composite images having the one or more elements of the game state superimposed on the one or more predicted frames.
10. generating one or more predicted frames from the next state, the one or more predicted frames having a simple graphics state; providing a task of determining a complex graphics state to said second node; receiving one or more image frames of the complex graphics state from the second node; combining the one or more frames of the complex graphics state with the one or more predicted frames; The method of claim 1 further comprising:
11. 1. A system for reducing latency in game execution, the system comprising: a first node including a first central processing unit; a second node coupled to the first node, the second node including a second central processing unit; The first node comprises: generating a current state of the game from user input; predicting a next state of the game based on the user input and one or more predicted user inputs; providing the second node with the task of determining a complex physical state; In predicting the next state of the game, the player retains the task of creating a simple physics state. It is configured as follows: The system, wherein the second central processing unit of the second node has a processor performance greater than the processor performance of the first central processing unit for determining the complex physical state.
12. The system of claim 11 , wherein the complex physics state includes contact points for a collision of two virtual objects of the game, and the simple physics state includes the probability that the collision will occur.
13. The second node generating one or more predicted frames from the next state; determining whether the one or more predictive user inputs have been received over a computer network; Transmitting the one or more predicted frames in response to receiving the one or more predicted user inputs to reduce the latency during execution of the game. The system of claim 11 configured to:
14. 12. The system of claim 11, wherein the second node is configured to predict the next state of the game and generate the one or more predicted frames before the one or more predicted user inputs are received to reduce the latency in the execution of the game.
15. the second node is configured to generate one or more predicted frames from the next state; one of the first node and the second node determines that a user input different from the one or more predicted user inputs is received over a computer network; generating a game state from the user input different from the one or more predicted user inputs; generating one or more frames from the game state; and Transmitting the one or more frames of the game state The system of claim 11 configured to:
16. 16. The system of claim 15, wherein the one of the first node and the second node is configured to generate the game state from the user input and to generate the one or more frames from the game state with a higher priority than the prediction of the next state by the second node.
17. 16. The system of claim 15, wherein the one of the first node and the second node is configured to generate the game state from the user input and to generate the one or more frames from the game state with a higher priority than the generation of the one or more predicted frames by the second node.
18. The system of claim 11 , wherein the second node is configured to predict the next state of the game based on an output of the current state of the game.
19. The second node generating one or more predicted frames from the next state; determining that a user input different from the one or more predicted user inputs is received over a computer network; It is configured as follows: one of the first node and the second node determines that a user input different from the one or more predicted user inputs is received over the computer network; identifying one or more elements of a game state of the game that are not included within the next state; generating the one or more elements of the game state from the user input different from the one or more predicted user inputs; generating one or more frames from the one or more elements of the game state; Transmitting the one or more frames of the one or more elements of the game state to enable display of one or more composite images.
12. The system of claim 11, configured such that the one or more composite images have the one or more elements of the game state superimposed on the one or more predicted frames.
20. the first node is configured to generate one or more predicted frames from the next state, the one or more predicted frames having a simple graphics state; The first node comprises: providing the second node with the task of determining a complex graphics state; receiving one or more image frames of the complex graphics state from the second node; Combining the one or more frames of the complex graphics state with the one or more predicted frames. The system of claim 11 configured to:
Citation Information
Patent Citations
Cloud streaming service system, method for providing cloud streaming service, and device for same
EP2937789A1
Selective caching of interactive objects
US20150119139A1
Predictive virtual reality content streaming techniques
US20180115743A1
Speculative rendering using historical player data
US9498715B2
