Introducing higher input latency in multiplayer programs

Abstract

A server computing device for introducing latency on a target input stream is provided. The server computing device includes a processor configured to receive a plurality of input streams from a corresponding plurality of client computing devices. Each input stream includes a plurality of inputs that control actions of a corresponding character in a multiplayer online software program. The processor is further configured to: determine a latency of each of the input streams; identify a higher latency input stream and a lower latency input stream in the plurality of input streams; and introduce a higher latency in the lower latency input stream to reduce a latency difference between the higher latency input stream and the lower latency input stream.

Description

[0001] background

[0002] A typical multiplayer online software program includes a local software program state executed on each player's client computing device. Each local software program state is updated based on the input of the local user, which is also sent to a server system that maintains the server software program state that serves as an arbitrator of the "ground truth" of the multiplayer session state. However, due to network latency, each player's local software program state may become slightly different from the server software program state and the local software program states of other players. Lag compensation techniques can be used to synchronize the local software program state to the server software program state. However, if the player's client computing device is a thin client computing device that does not usually execute software programs, these techniques may not be applicable. Summary of the invention

[0003] In order to solve the problem described above, a server computing device for introducing latency on a target input stream is provided. The server computing device may include a processor configured to receive a plurality of input streams from a corresponding plurality of client computing devices. Each input stream may include a plurality of inputs for controlling actions of a corresponding character in a multiplayer online software program. The processor may be further configured to: determine the latency of each of the input streams; identify a higher latency input stream and a lower latency input stream in the plurality of input streams; and introduce a higher latency in the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream.

[0004] This Summary is provided to introduce in simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of an example computing system including a client computing device and a server computing device implementing a remote execution device according to one embodiment of the present disclosure is shown.

[0007] Figure 2 Shows Figure 1 Schematic diagram of a client computing device and a server computing device.

[0008] Figure 3 Shows the use of Figure 1An example of a server computing device introducing higher latency in the input stream.

[0009] Figure 4 Shows the use of Figure 1 A server computing device is used to predict examples of user input.

[0010] Figure 5 Shown by Figure 1 A computer-implemented method for introducing higher latency into a low-latency input stream is implemented by a computing system of the invention.

[0011] Figure 6 A schematic representation of an example computing system is shown in accordance with one embodiment of the present disclosure.

[0012] Detailed Description

[0013] A typical multiplayer online software program includes a local software program state executed on each user's client computing device. Each local software program state can be updated immediately based on the input of the local user. The input can be further sent to a server system that is executing and managing the server software program state that acts as an arbitrator of the "ground truth" of the multiplayer online software program. Therefore, the local software program state of each user can be synchronized and updated to the server software program state to reflect the user input typed by each other user in the multiplayer online software program. Due to network latency, the local software program state of each player may become different from the server software program state. For example, although the input of the first user can be immediately reflected in the local software program state executed by the client computing device of the user, the same input may not be reflected in the local software program state of other users until it is sent to the server system, processed by the server system to affect the server software program state of the multiplayer online software program, and communicated to the client computing device of each other user. Typically, multiplayer online software programs utilize different latency compensation techniques on client computing devices to help minimize the potential user experience degradation effect of high and / or unstable network latency.

[0014] However, these same latency compensation techniques may no longer apply when each player's client computing device is a thin computing device that does not locally execute and render the local software program state of the multi-player online software program. Figure 1An example computing system 100 is illustrated that includes multiple client computing devices 102 configured to communicate with a server computing device 104 via a respective network 106. The server computing device 104 may include one or more discrete server computing devices working in coordination. In one example, the server computing device 104 may include multiple server computing devices operating in a cloud computing configuration that work in coordination to implement the functions and processes of the server computing device 104 described herein.

[0015] In one example, each client computing device 102 may take the form of a thin computing device that does not execute its own local software program instance in a multi-player online software program. For example, client computing devices 102 may include mobile computing devices, game streaming computing devices, tablet computing devices, streaming consoles, and other types of computing devices that may not include sufficient processing power to execute a local instance of an online multi-player software program. Figure 1 As illustrated, each client computing device 102 is configured to execute a streaming client 108 that is configured to send user input to the server computing device 104 and display output frames generated by a remote execution device 110 implemented by the server computing device 104. Alternatively, the client computing device 102 may be a more powerful computer with processing power and memory capable of running a full version of a multiplayer online software program, but still function as a thin client that operates in the form of a streaming client 108.

[0016] In one example, because each client computing device 102 may be configured to display the output frames generated by the remote execution device 110 and not render its own local frames, each player's input may not be immediately reflected in the frame displayed to the user until the input has been sent to the server computing device 104 and the remote execution device 110 of the server computing device 104 has rendered the output frame for each user input. In addition, because multiple client computing devices 102 may be located in different locations, the latency of the network over which each client computing device 102 communicates with the server computing device 104 may be different for each client computing device 102. That is, a first client computing device may communicate with the server computing device 104 via a first network 106A, a second client computing device may communicate with the server computing device 104 via a second network 106B, and a third client computing device 102 may communicate with the server computing device 104 via a third network 106C, and the network latencies of the first, second, and third networks may be different from each other. For example, if third network 106C has a higher network latency than first network 106A, user input from the first client computing device may reach server computing device 104 faster than user input from the third client computing device, thereby potentially giving the user of the first client computing device an unfair advantage over the user of the third client computing device.

[0017] In order to solve the problems discussed above, Figure 2 An example server computing device 104 configured to introduce latency on a target input stream from a client computing device participating in an online multiplayer software program is illustrated. Selectively introducing latency on the target input stream potentially allows the server computing device 104 to mitigate the unfair effects of varying network latencies to maintain fairness among all participating users, and allows the server computing device 104 to have the opportunity to reduce the bandwidth required to stream the multiplayer online software program to the client computing devices.

[0018] like Figure 2 As illustrated, the computing system 100 may include a plurality of client computing devices 102 configured to communicate with a server computing device 104 via a respective network 106. Each client computing device 102 may include computer components such as non-volatile memory 114, volatile memory 116, an input device suite 118, an output device suite 120, and a processor 122. The input device suite 118 may include any suitable type of input device, such as, for example, a keyboard and mouse, a console controller, a joystick, etc. The output device suite 120 may include any suitable type of output device, such as, for example, a display 120A, speakers, etc.

[0019] The processor 122 of each client computing device 102 is configured to execute a client-side streaming platform 124 for a multi-player online software program, which may take the form of an online multi-player game or another type of multi-user application, for example. Each client-side streaming platform 124 of each client computing device 102 is configured to communicate with the server computing device 104 via the corresponding network 106.

[0020] The server computing device 104 may include one or more server computing devices located at different locations. For example, a first server computing device communicating with a first client computing device may be located on the west coast of the United States, while a second server computing device communicating with a second client computing device may be located on the east coast of the United States. In this example, the first and second server computing devices 104 may be configured to work together to implement the processes and methods of the server computing devices 104 described herein. Figure 2 As illustrated, each of the one or more server computing devices 104 may include computer components such as non-volatile memory 126, volatile memory 128, and a processor 130. The processor 130 of the server computing device 104 may be configured to execute a server-side streaming platform 132. The client-side streaming platform 124 and the server-side streaming platform 132 are configured to send and receive data to and from each other to execute the multi-player online software program 134 and present it to the respective users of each client computing device 102.

[0021] During execution of the multiplayer online software program 134, users may enter inputs 136 via the input device suite 118 of their respective client computing devices 102. In one example, the multiplayer online software program 134 is an online multiplayer video game, and the plurality of inputs 136 entered by the users are inputs to control actions of characters in the multiplayer online software program 134, such as, for example, movement, selection, execution of actions, etc. Figure 2 As illustrated, the streaming client 108 of the client-side streaming platform 124 can send a plurality of inputs 136 in an input stream 138 to the server computing device 104 via the network 106. Each client computing device 102 participating in the multiplayer online software program 134 can be configured to send an input stream 138 to the server computing device 104 that controls the actions of a corresponding character in the multiplayer online software program 134.

[0022] The input handler 140 of the server-side streaming platform 132 executed by the processor 130 of the server computing device 104 may be configured to receive a plurality of input streams 138 from a corresponding plurality of client computing devices 102. In one example, each input 136 in the plurality of input streams 138 is configured to be sent by each client computing device 102 to the server computing device 104 at a target input frequency 142 (such as, for example, 1 / (target frame rate)) of the multiplayer online software program 134. Each input 136 may include data regarding the state of the input device suite 118, such as, for example, which buttons are currently pressed / activated, in which direction the joystick has been moved, etc. In some examples, if no input is entered by the user into the input device suite 118, the input 136 in the input stream 138 may be a "heartbeat" input indicating that no input is entered by the user at that point in time based on the target input frequency 142. Thus, in this example, the input handler 140 of the server-side streaming platform 132 may be configured to track whether input 136 is received from each client computing device 102 at a target input frequency 142 and detect any gaps in the input received from each client computing device 102 .

[0023] The input 136 of the plurality of input streams 138 is sent to both the input emulator 144 and the input predictor 146 of the input handler 140. The input emulator 144 is configured to emulate the input type expected by the remote execution device 110 of the server computing device 104 executing the multiplayer online software program 134. As a specific example, the multiplayer online software program 134 may be a console game, and the remote execution device 110 may be a remote console device. In this example, the input 136 from the client computing device 102 may be processed by the input emulator 144, which may be configured to generate emulated input 148 of a type suitable for the remote execution device 110 and the multiplayer online software program 134 (such as console controller input). Since the client computing device 102 may take different forms (such as a tablet computing device, a mobile computing device, etc.), and therefore may have different input methods, types, and modalities, the input emulator 144 is configured to map those types of inputs to corresponding inputs suitable for the remote execution device 110. For example, if remote execution device 110 is a remote console device and the client computing device is a desktop computing device using a keyboard and mouse, input emulator 144 may be configured to map keyboard and mouse inputs to corresponding console controller inputs to generate emulated inputs 148 .

[0024] The input emulator 144 of the input handler 140 is configured to perform these input emulation processes on each input stream received from the plurality of client computing devices 102. These emulated inputs may be sent to the remote execution device 110 in an emulated input stream for each respective client computing device. The remote execution device 110 is configured to manage the software program state of the multiplayer online software program 134 and update this software program state based on the stream of emulated input 148 received from the input handler 140 for each of the plurality of client computing devices 102.

[0025] Based on the stream of simulated input 148, the remote execution device 110 can determine the current perspective of the corresponding character of each client computing device in the scene of the multiplayer online software program 134, and can render the views of these current perspectives as output frames 150 for each client computing device 102. In one example, the remote execution device 110 can be configured to render output frames 150 for the software program state of the multiplayer online software program 134 for each of the multiple client computing devices 120 at a controllable frame rate 151. For example, the controllable frame rate 151 can be set to 20 frames per second, 60 frames per second, or another suitable target frame rate. The output frames 150 can be encoded by the global latency manager and encoder 152 of the server-side streaming platform 132. The global latency manager and encoder 152 can be further configured to send the rendered output frames 150 to the corresponding multiple client computing devices 102 in multiple output frame streams 154.

[0026] like Figure 2 As illustrated, each client computing device 102 may be configured to receive their respective output frame stream 154 from the server computing device 104. The streaming client 108 of each client computing device 102 may process the output frame stream 154 and present the output frames 150 as display frames 156 on the display 120A of the output device suite 120. For example, the streaming client 108 may process the output frames 150 based on a native resolution selected by a user, the type of display 120A, a brightness / darkness setting selected by a user, etc.

[0027] It should be understood that the server computing device 104 may further send other types of output, such as, for example, audio output, along with the output frame stream 154. The streaming client 108 may be configured to process these other types of output for presentation along with the display frames 156 via the output device suite 120. In the manner described above, each client computing device 102 may send input received from a user to the server computing device 104 over the corresponding network 106 in an input stream, receive corresponding output frames 150 from the server computing device 104 in an output frame stream, and display these output frames as display frames 156 on the display 120A.

[0028] However, as previously discussed, the respective network 106 used by each client computing device 102 to communicate with the server computing device 104 may have varying network latency and / or network stability. Figure 2 As illustrated, the client-side streaming platform 124 executed by the processor 122 of each client computing device 102 further includes a network predictor 158 that is configured to profile the network latency and jitter of its corresponding network 106. For example, the network predictor 158 may be configured to send packets to the server computing device 104 and measure the round-trip time of each packet (the time from sending the packet to receiving a TCP acknowledgment for the packet from the server). The network predictor 158 may analyze the changes in the round-trip time over time to determine short-term and long-term trends in network latency and stability. In this way, the network predictor 158 may generate network latency data 160, which includes long-term and short-term predictions for network latency. The network predictor 158 may be configured to send the network latency data 160 to the global latency manager and encoder 152 of the server-side streaming platform 132 implemented by the server computing device 104. In one example, the global latency manager and encoder 152 can be configured to also profile the respective networks 106 used to communicate with the plurality of client computing devices 102 to similarly measure short-term and long-term network latency trends and stability. The network latency data 160 received from each client computing device 102 is stored and managed by the global latency manager and encoder 152.

[0029] Based on the network latency data for each client computing device 162, the global latency manager and encoder 152 is configured to determine the latency of each input stream 138 received from the plurality of client computing devices 102, respectively. As discussed above, the respective networks 106 utilized by the plurality of client computing devices 102 may have different latencies and network stability. Therefore, based on the determined latency of each input stream 138, the global latency manager and encoder 152 may be configured to identify higher latency input streams and lower latency input streams in the plurality of input streams 138. Figure 3 In the illustrated example, the input streams 138 sent by the first client computing device 102A and the second client computing device 102B are identified as lower latency input streams 138A and 138B by the global latency manager and encoder 152 of the server computing device 104. On the other hand, based on the network and latency data of each client computing device 162 managed by the global latency manager and encoder 152, the input stream 138 sent by the third client computing device 102C is identified as a higher latency input stream 138C.

[0030] After determining that at least one of the input streams received from the plurality of client computing devices 102 has a latency difference greater than a threshold latency compared to the other input streams, the global latency manager and encoder 152 may be configured to introduce a higher latency into the lower latency input stream 138A / 138B to reduce the latency difference between the higher latency input stream 138C and the lower latency input stream 138A / 138B. In one configuration, the global latency manager and encoder 152 may be configured to introduce the higher latency into each of the plurality of input streams except for the highest latency input stream. For example, Figure 3 , the input streams 138 sent by both the first client computing device 102A and the second client computing device 102B are identified as lower latency input streams 138A and 138B. Therefore, the global latency manager and encoder 152 can be configured to introduce higher latency in both the lower latency streams 138A and 138B sent over the first network 106A and the second network 106B.

[0031] In one example, the introduced higher latency 164 for each lower latency stream may be scaled and controlled by the global latency manager and encoder 152. The global latency manager and encoder 152 may be configured to control the introduced higher latency 164 based on the predicted long-term latency of each of the plurality of input streams 138, which may be determined based on the network latency data 162 received from the network predictor of each client computing device 102. That is, the global latency manager and encoder 152 may be configured to: identify the highest predicted long-term latency 166 in the plurality of input streams 138; and increase the introduced higher latency 164 in one or more of the lower latency input streams 138A / 138B up to the highest predicted long-term latency 166 of the plurality of input streams 138. In this manner, the global latency manager and encoder 152 may be configured to: introduce increasing amounts of artificial latency into the identified lower latency streams until each of the plurality of input and output streams of the client computing device 102 has a latency within a threshold latency difference.

[0032] The global latency manager and encoder 152 may be configured to introduce higher latency on the selected input stream via different methods. In one example, the global latency manager and encoder 152 may be configured to introduce the higher latency by buffering the input of each of the plurality of input streams 138. Figure 3 , each input 136 in each input stream 138 from a plurality of client computing devices 102 may be placed into an input buffer 168 as it is received by the server computing device 104. The input 136 for each of the lower latency input streams 138A and 138B may remain in the input buffer 168 until the corresponding input from the higher latency input stream 138C has been received by the input handler 140 of the server computing device 104. Once the inputs 136 have been received from all client computing devices 102, they may be pulled from the input buffer 168 and sent to the remote execution device 110, which may then render a corresponding output frame 150 based on the inputs. It should be appreciated that the input buffer 168 may be configured to buffer any suitable number of inputs, such as one input, two inputs, three inputs, etc.

[0033] In one example, the server computing device 104 may be configured to buffer input from the lower latency input streams 138A and 138B independently to the output frame stream 154. That is, even if input is stored in the input buffer 168 while the input handler 140 is waiting for input from the higher latency input stream 138C, the remote execution device 110 may continue to render output frames 150 at a controllable frame rate 151 in the absence of new input and send these output frames to the client computing device 102 in the corresponding output frame stream 154. Due to input buffering, end users of the client computing device 102 with the low latency input stream 138 on the low latency network will be presented with an uninterrupted stream of display frames 156 that do not yet reflect their input 136, and will therefore perceive the multiplayer online software program 134 as less responsive and having a higher latency. However, the visual experience of these end users will be preserved. In this way, the perceived responsiveness of the multiplayer online software program 134 (e.g., the time elapsed between user input and a displayed frame reflecting that input) can be selectively reduced so that a user with a low latency network can match the perceived responsiveness experienced by other users with a high latency network, without degrading the frame rate on the client side.

[0034] In another example, the global latency manager and encoder 152 may be configured to introduce the higher latency by buffering the output frames 150 of each of the plurality of output frame streams 154. Figure 3 As illustrated, the output frames 150 rendered by the remote execution device 110 may be stored in a corresponding output frame buffer 170 for each output frame stream 154 of the plurality of client computing devices 102. The server computing device 104 may then pull the output frames 150 from the corresponding output frame buffer 170 to be sent to the plurality of client computing devices 102 according to the control of the global latency manager and encoder 152. For example, the global latency manager and encoder 152 may pull and send the output frames 150 from the output frame buffer 170 based on the highest predicted long-term latency 166, so that the output frames 150 arrive at the client computing device 102 at a similar time. That is, the output frames of the lower latency stream may be delayed in the output frame buffer 170 according to the latency difference between the lower latency stream and the higher latency stream, so that the output frames arrive at the client computing device at a similar time. In this way, the end user of each client computing device 102 can experience a similar degree of "jitter" in the displayed frame due to high network latency, so that fairness is maintained in the multi-player online software program 134. It should be understood that the input buffering and outframe buffering processes described above can be implemented together or separately by the server computing device 104.

[0035] In another example, the global latency manager and encoder 152 may be configured to introduce the higher latency by reducing the controllable frame rate 151 to less than the target frame rate 172. For example, the global latency manager and encoder 152 may modify the controllable frame rate 151 of the multiplayer online software program 134 based on the highest predicted long-term latency 166. That is, because the perceived responsiveness of the multiplayer online software program 134 (e.g., the time elapsed between a user input and a displayed frame reflecting the input) will be reduced due to high network latency, the server computing device 104 may be configured to opportunistically reduce the controllable frame rate 151 of the multiplayer online software program 134. By reducing the controllable frame rate 151, the output frames 150 will be rendered by the remote execution device 110 at a lower rate, thereby reducing the computational burden imposed on the server computing device 104. In addition, because there are fewer output frames 150, the required streaming bandwidth for sending the output frame stream 154 to the client computing device will also be reduced, thereby potentially saving costs and hardware resources of the server computing device 104. Although reducing the controllable frame rate 151 will also reduce the perceived responsiveness of the multiplayer online software program 134 , the global latency manager and encoder 152 can reduce the controllable frame rate 151 in proportion to the highest predicted long-term latency 166 .

[0036] Reducing the frame rate of the output frames 150 displayed to users may potentially reduce the perceived smoothness of the multiplayer online software program 134 for these users. Therefore, in one example, the global latency manager and encoder 152 of the server computing device 104 may be further configured to cause each client computing device 102 to interpolate intermediate frames 174 based on one or more output frames 150 in the output frame stream 154 to achieve the target frame rate 172. Figure 2 , the global latency manager and encoder 152 may be configured to send the output frame 150 in the output frame stream 154 of the corresponding client computing device 102 to the frame interpolator 176 of the client-side streaming platform 124 executed by each of these client computing devices 102. The frame interpolator 176 may be configured to interpolate the intermediate frame 174 based on the rendered output frame before the intermediate frame 174 and / or the rendered output frame after the intermediate frame 174. That is, if the controllable frame rate 151 is 60 frames per second, and the controllable frame rate 151 has been reduced to 30 frames per second, the global latency manager and encoder 152 may be configured to instruct the frame interpolator 176 to interpolate the intermediate frame 174 every other frame to achieve the target frame rate 172 of 60 frames per second.

[0037] The frame interpolator 176 may be configured to interpolate the intermediate frames 174 using interpolation techniques such as isomorphism, time warping, image-based rendering mesh warping, late reprojection techniques, neural network interpolation, and other suitable interpolation techniques. These interpolation techniques do not require a complete rendering process and may be performed on thin client computing devices that do not have sufficient hardware and processing resources to execute a full rendering processor of the multiplayer online software program 134. The interpolated intermediate frames 174 may be sequentially sent to the streaming client 108 along with the output frames 150 and processed into display frames 156 presented to the user via the display 120A at a target frame rate 172. In this way, the server computing device 104 may reduce the visual fidelity of the multiplayer online software program 134 to reduce bandwidth and hardware resource costs while maintaining the target frame rate 172 of the multiplayer online software program 134.

[0038] The global latency manager and encoder 152 may be further configured to detect short-term latency changes in the input streams 138 received from the plurality of client computing devices 102 based on an expected target input frequency 142 that each client computing device 102 is expected to follow. That is, each client computing device 102 is expected to send input 136 at a target input frequency 142. For example, even if a user does not type input 136 at a particular period, the client computing device 102 may still send a "heartbeat" input to the server computing device 104 indicating that no input was typed by the user. Thus, the global latency manager and encoder 152 may be configured to determine the latency of each of the input streams 138 based on detecting whether input was received from each of the input streams 138 at the target input frequency 142. If the global latency manager and encoder 152 detects that the server computing device 104 did not receive input in the input stream 138 from a particular client computing device 102, the global latency manager and encoder 152 may determine a short-term increase in latency or a disruption in stability of the network for the particular client computing device 102.

[0039] like Figure 2, based on detecting that one of the input streams 138 from the client computing device 102 has missing input, the input predictor 146 of the input handler 140 can be configured to determine a predicted input 178 to fill in the missing input. The predicted input 178 can be sent to the input emulator 144, which can be configured to generate a corresponding simulated input 148 that is sent to the remote execution device 110. The remote execution device 110 can render the output frame 150 of the predicted input 178 as if the user had typed the input in the input stream 138. In this way, missing input in the input stream 138 that occurs due to short-term increases in latency and / or network stability issues can be filled in by the server computing device 104 with the predicted input 178.

[0040] In one example, the predicted input 178 may be determined based on trained input data 180 including a history of inputs in the input stream 138. For example, the input predictor 146 may determine the predicted input 178 based on previous inputs in the input stream 138. The history of inputs may be further used to generate a trained model for the user that is used to predict future inputs. The input predictor 146 may be trained using feedback from players. Initially, the input predictor 146 may bootstrap the user using a trained model of players with similar characteristics to the user (e.g., favorite or most played games, the user's ranking in an online service, achievements, etc.).

[0041] Figure 4 An example input predictor 146 that generates predicted input to fill in missing input in a higher latency input stream 138C is illustrated. As illustrated, the server computing device 104 expects to receive input from the client computing device 102 according to a target input frequency 142. The input handler 140 of the server-side streaming platform 132 implemented by the server computing device 104 can be configured to detect missing input 184 in the higher latency input stream 138C based on the target input frequency 142. That is, if the server computing device 104 does not receive input from the client computing device within a threshold time period after the expected time point, the input handler 140 can determine that the client computing device has lost input due to a short-term high latency or instability of the network between the client computing device and the server computing device 104. Upon detecting the missing input 184, the input predictor 146 can be configured to determine the predicted input 178 as discussed above, and insert the predicted input 178 into the higher latency input stream 138C to fill in the missing input 184.

[0042] Figure 4The illustrated example shows a sequence where only one or two inputs are lost consecutively in the higher latency input stream 138C. However, in some examples, due to the instability of the network, the client computing device may potentially lose more consecutive inputs, such as, for example, five inputs, ten inputs, etc. In this example, the global latency manager and encoder 152 may be configured to detect whether the higher latency input stream 138C has lost more consecutive inputs than a threshold amount of lost inputs (which may be defined by the program developer), such as 5 inputs, 10 inputs, or 15 inputs, etc. If the higher latency input stream 138C has lost more inputs than the threshold amount of inputs, the global latency manager and encoder 152 may be configured to stop the execution of the multiplayer online software program 134 until the next input in the higher latency input stream 138C is received. If the higher latency input stream 138C has less than a threshold amount of missing input, the global latency manager and encoder 152 may be configured to instruct the input handler 140 to pull new inputs from the input buffer 168 for that input stream and / or instruct the input predictor 146 to generate new predicted inputs 178 to be inserted into the missing input slots in the higher latency input stream 138C. A modified higher latency input stream 182 including the actual inputs 136 and the predicted inputs 178 may be sent to the remote execution device 110 and processed according to the methods described above.

[0043] It should be appreciated that input predictor 146 may be implemented using any suitable combination of current and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. For example, Figure 2 The machine learning model of the trained data 180 shown may be trained to generate a state machine that is utilized by the input predictor 146 to generate predicted inputs 178 for a user of a particular client computing device 102. In one example, the machine learning model may be trained based on a history of input data received from this particular client computing device 102. In another example, the machine learning model may be trained based on a history of input received from multiple different users across different game sessions over time. Additionally, the machine learning model of the trained data 180 may be shared among multiple different server computing devices 104 that may be located in different geographic locations, data centers, etc. For example, Figure 2 The machine learning model of the trained data 180 shown in FIG. 1 may have been trained using input data received by multiple different server computing devices 104 located in different data centers, each of which may each receive input data from multiple different client computing devices.

[0044] Non-limiting examples of techniques that may be incorporated into an implementation include: support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, composition and / or dependency analysis, and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., hidden dynamic models)).

[0045] In some examples, one or more differentiable functions may be used to implement the methods and processes described herein, wherein the gradient of the differentiable function may be calculated and / or estimated with reference to the input and / or output of the differentiable function (e.g., with reference to training data and / or with reference to a target function). Such methods and processes may be determined, at least in part, by a set of trainable parameters. Thus, the trainable parameters for a particular method or process may be adjusted by any appropriate training procedure in order to continually improve the functionality of the method or process.

[0046] Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), bandit methods with context, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the systems described herein may be trained simultaneously with reference to a target function that measures the performance of the collective functions of multiple components (e.g., with reference to enhanced feedback and / or with reference to labeled training data). Training multiple methods, processes, and / or components simultaneously may improve such collective functions. In some examples, one or more methods, processes, and / or components may be trained independently of other components (e.g., offline training on historical data).

[0047] Figure 55 shows a flowchart of a computer-implemented method 500. The method 500 may be performed by Figure 1 At 502, the method 500 may include: receiving a plurality of input streams from a corresponding plurality of client computing devices, each input stream including a plurality of inputs for controlling actions of a corresponding character in a multiplayer online software program, such as, for example, moving, selecting, performing actions, etc. As described above with reference to Figure 2 As depicted, a streaming client 108 implemented by a client computing device may send a plurality of inputs 136 in an input stream 138 to a server computing device 104 via a network 106. Each client computing device 102 participating in a multiplayer online software program 134 may be configured to send an input stream 138 to the server computing device 104.

[0048] At 504, the method 500 may include determining a latency of each of the input streams. Figure 2 As described, the network predictor 158 of the client computing device can be configured to summarize the network latency and jitter of its corresponding network 106. For example, the network predictor 158 can be configured to send packets to the server computing device 104 and measure the round-trip time of each packet. The network predictor 158 can analyze the changes in round-trip time over time to determine short-term and long-term trends in network latency and stability. In this way, the network predictor 158 can generate network latency data 160, which includes long-term and short-term predictions for network latency.

[0049] At 506, the method 500 may include predicting a long-term latency for each of the plurality of input streams. Figure 2 As described, the predicted long-term latency of each of the plurality of input streams 138 may be determined based on the network latency data 162 received from the network predictor of each client computing device 102 .

[0050] At 508, the method 500 may include identifying a higher latency input stream and a lower latency input stream in the plurality of input streams. Figure 2 As described, the global latency manager and encoder of the server computing device 104 may be configured to compare network latency data received from the client computing device 102 and identify higher latency input streams and lower latency input streams. In one example, the global latency manager and encoder may be further configured to identify a particular input stream as a high latency input stream based on detecting the presence of lost input in the particular input stream according to a target input frequency.

[0051] At 510, the method 500 may include introducing a higher latency into the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream. In one example, the higher latency is introduced into each of the multiple input streams except for the highest latency input stream. The highest latency input stream may be determined based on network latency data received from each client computing device.

[0052] At step 510, higher latency can be introduced into the low latency input stream using different techniques. In one example, introducing the higher latency includes buffering the input of each of the plurality of input streams. Figure 3 As described, each input 136 in each input stream 138 from the plurality of client computing devices 102 may be placed into an input buffer 168 as it is received by the server computing device 104. The input 136 for each of the lower latency input streams 138A and 138B may be held in the input buffer 168 until the corresponding input from the higher latency input stream 138C has been received by the input handler 140 of the server computing device 104. Once the inputs 136 have been received from all of the client computing devices 102, the inputs may be pulled from the input buffer 168 and sent to the remote execution device 110, which may then render corresponding output frames 150 based on the inputs.

[0053] In another example, introducing the higher latency may include buffering output frames of each of the plurality of output frame streams. Figure 3 As described, the output frames 150 rendered by the remote execution device 110 may be stored in the corresponding output frame buffers 170 of each output frame stream 154 of the plurality of client computing devices 102. The server computing device 104 may then pull the output frames 150 to be sent to the plurality of client computing devices 102 from the corresponding output frame buffers 170 according to the control of the global latency manager and the encoder 152.

[0054] In another example, introducing the higher latency may include reducing the controllable frame rate to less than the target frame rate. Figure 3 As described, the global latency manager and encoder 152 may modify the controllable frame rate 151 of the multiplayer online software program 134 based on the highest predicted long-term latency 166. That is, because the perceived responsiveness of the multiplayer online software program 134 (e.g., the time elapsed between a user input and a displayed frame reflecting the input) will be reduced due to high network latency, the server computing device 104 may be configured to opportunistically reduce the controllable frame rate 151 of the multiplayer online software program 134.

[0055] At 512, the method 500 may include increasing the introduced higher latency in one or more lower latency input streams up to a highest predicted long-term latency of the plurality of input streams. Step 512 may include introducing increasing amounts of artificial latency into the identified lower latency streams until each of the plurality of input and output streams of the client computing device 102 has a latency within a threshold latency difference.

[0056] At 514, the method 500 may include rendering an output frame of a software program state for a multiplayer online software program for each of a plurality of client computing devices at a controllable frame rate. Figure 2 As described, based on the stream of simulated input 148 , the remote execution device 110 may determine the current perspective of each client computing device's corresponding character in the scene of the multiplayer online software program 134 , and may render views of these current perspectives as output frames 150 for each client computing device 102 .

[0057] At 516, the method 500 may include sending the rendered output frames to a corresponding plurality of client computing devices in a plurality of output frame streams. As discussed at step 512, the rendered output frames may be stored in an output frame buffer to introduce higher latency on a selected stream. The rendered output frames may be pulled from the output frame buffer and then sent to the associated client computing devices to achieve a target introduced latency in this stream.

[0058] At 518, the method 500 may include causing each client computing device to interpolate intermediate frames based on one or more output frames in the output frame stream to achieve the target frame rate. Figure 2 As described, a frame interpolator 176 implemented by a client computing device may be configured to interpolate intermediate frames 174 based on rendered output frames preceding intermediate frames 174 and / or rendered output frames following intermediate frames 174. Frame interpolator 176 may be configured to use interpolation techniques such as conformal, time warping, image-based rendering grid warping, post-reprojection techniques, neural network interpolation, and other suitable interpolation techniques to interpolate intermediate frames 174. The output frames and the interpolated intermediate frames may be presented to a user via a display of the client computing device.

[0059] The systems and methods described above can be used to maintain fairness in a multi-player session of an online software program by using various techniques to introduce latency into some player streams, thereby allowing each player in the multi-player session to have a more similar gaming experience despite each player having different latency in their respective network connections.

[0060] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0061] Figure 6 A non-limiting embodiment of a computing system is schematically shown. In some embodiments, the methods and processes described herein can be bound to a computing system of one or more computing devices. Specifically, such methods and processes can be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.

[0062] Figure 6 A non-limiting embodiment of a computing system 600 that can perform one or more of the above methods and processes is schematically shown. The computing system 600 is shown in simplified form. The computing system 600 can embody the above description and Figure 1 and Figure 2 1 and 10. The computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0063] The computing system 600 includes a logic processor 602, a volatile memory 604, and a non-volatile storage device 606. The computing system 600 may optionally include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or Figure 6 Other components not shown.

[0064] Logical processor 602 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0065] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. Each processor of the logical processor 602 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The individual components of the logical processor may optionally be distributed between two or more separate devices, which may be remotely located and / or configured for collaborative processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. It will be understood that in such a case, these virtualization aspects are run on different physical logical processors of various different machines.

[0066] Non-volatile storage device 606 includes one or more physical devices configured to hold instructions executable by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of non-volatile storage device 606 may be transformed—eg, to hold different data.

[0067] The non-volatile storage device 606 may include a removable and / or built-in physical device. The non-volatile storage device 606 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 606 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It will be understood that the non-volatile storage device 606 is configured to save instructions even when power to the non-volatile storage device 606 is cut off.

[0068] The volatile memory 604 may include a physical device that includes random access memory. The volatile memory 604 is typically used by the logical processor 602 to temporarily store information during the processing of software instructions. It will be understood that when the power to the volatile memory 604 is cut off, the volatile memory 604 typically does not continue to store instructions.

[0069] Aspects of the logic processor 602, volatile memory 604, and non-volatile storage device 606 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0070] The terms "module," "program," and "engine" may be used to describe aspects of the computing system 600 that are typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformation process that specifically configures the processor to perform that function. Thus, a module, program, or engine may be instantiated using portions of volatile memory 604 via the logical processor 602 executing instructions held by the non-volatile storage device 606. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" are intended to cover single or grouped executable files, data files, libraries, drivers, scripts, database records, and the like.

[0071] When a display subsystem 608 is included, the display subsystem 608 can be used to present a visual representation of the data held by the non-volatile storage device 606. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 608 can also be transformed to visually represent the changes in the underlying data. The display subsystem 608 can include one or more display devices utilizing substantially any type of technology. Such display devices can be combined with the logical processor 602, the volatile memory 604, and / or the non-volatile storage device 606 in a shared housing, or such display devices can be peripheral display devices.

[0072] When an input subsystem 610 is included, the input subsystem 610 may include or interface with one or more user input devices such as a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include selected natural user input (NUI) components or interface with the above-mentioned natural user input (NUI) components. Such components can be integrated or peripheral, and the transduction and / or processing of input actions can be handled on or off the board. Example NUI components may include a microphone for language and / or speech recognition; infrared, color, stereo display and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for evaluating brain activity; and / or any other suitable sensors.

[0073] When the communication subsystem 612 is included, the communication subsystem 612 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple to other devices. The communication subsystem 612 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network (such as HDMI over a Wi-Fi connection). In some embodiments, the communication subsystem may allow the computing system 600 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0074] The following paragraphs provide additional support for the claims of the present application. One aspect provides a server computing device including a processor, the processor being configured to: receive a plurality of input streams from a corresponding plurality of client computing devices. Each input stream includes a plurality of inputs that control the actions of a corresponding character in a multiplayer online software program. The processor is further configured to: determine the latency of each of the input streams; identify a higher latency input stream and a lower latency input stream in the plurality of input streams; and introduce a higher latency in the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream. In this aspect, additionally or alternatively, the higher latency may be introduced into each of the other input streams except the highest latency input stream in the plurality of input streams. In this aspect, additionally or alternatively, the processor may be further configured to: predict the long-term latency of each of the plurality of input streams; and increase the introduced higher latency in one or more lower latency input streams to a maximum of the highest predicted long-term latency of the plurality of input streams. In this aspect, additionally or alternatively, the processor may be configured to introduce the higher latency by buffering the input of each of the multiple input streams. In this aspect, additionally or alternatively, the processor may be configured to render output frames for the software program state of the multiplayer online software program for each of the multiple client computing devices at a controllable frame rate; and send the rendered output frames to the corresponding multiple client computing devices in multiple output frame streams. In this aspect, additionally or alternatively, the processor may be further configured to introduce the higher latency by buffering the output frames of each of the multiple output frame streams. In this aspect, additionally or alternatively, the processor may be further configured to introduce the higher latency by reducing the controllable frame rate to less than the target frame rate. In this aspect, additionally or alternatively, the processor may be configured to cause each client computing device to interpolate intermediate frames based on one or more output frames in the output frame stream to achieve the target frame rate. In this aspect, additionally or alternatively, the processor may be configured to: determine the latency of each of the input streams based on detecting whether input is received from each of the input streams at a target input frequency. In this aspect, additionally or alternatively, the processor may be configured to: detect missing inputs in the higher latency input stream based on the target input frequency; and insert predicted inputs into the higher latency input stream to fill in the missing inputs. In this aspect, additionally or alternatively, the predicted inputs may be determined based on trained input data including a history of inputs in the input streams.

[0075] Another aspect provides a method, including receiving a plurality of input streams from a corresponding plurality of client computing devices at a server computing device including a processor. Each input stream includes a plurality of inputs that control actions of a corresponding character in a multiplayer online software program. The method further includes: determining a latency of each of the input streams; identifying a higher latency input stream and a lower latency input stream in the plurality of input streams; and introducing a higher latency in the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream. In this aspect, additionally or alternatively, the higher latency may be introduced into each of the plurality of input streams except for the highest latency input stream. In this aspect, additionally or alternatively, the method may further include: predicting a long-term latency of each of the plurality of input streams; and increasing the introduced higher latency in one or more lower latency input streams up to the highest predicted long-term latency of the plurality of input streams. In this aspect, additionally or alternatively, introducing the higher latency may include buffering the input of each of the plurality of input streams. In this aspect, additionally or alternatively, the method may further include: rendering output frames for the software program state of the multi-player online software program for each of the plurality of client computing devices at a controllable frame rate; and sending the rendered output frames to the corresponding plurality of client computing devices in a plurality of output frame streams. In this aspect, additionally or alternatively, introducing the higher latency may include buffering output frames of each of the plurality of output frame streams. In this aspect, additionally or alternatively, introducing the higher latency may include reducing the controllable frame rate to less than a target frame rate. In this aspect, additionally or alternatively, the method may further include causing each client computing device to interpolate intermediate frames based on one or more output frames in the output frame stream to achieve the target frame rate.

[0076] On the other hand, a server computing device including a processor is provided, the processor being configured to: receive a plurality of input streams from a corresponding plurality of client computing devices. Each input stream includes a plurality of inputs that control actions of a corresponding character in a multiplayer online software program. The processor is further configured to: determine the latency of each of the input streams, including the highest predicted long-term latency; identify a higher latency input stream and a lower latency input stream in the plurality of input streams; introduce a higher latency up to the highest predicted long-term latency in the lower latency input stream; and detect one or more missing inputs in the higher latency input stream based on a target input frequency. The processor is configured to: stop execution of the multiplayer online software program if the one or more missing inputs are greater than a threshold input amount. The processor is configured to: insert a predicted input into the higher latency input stream to fill in the missing input if the one or more missing inputs are less than the threshold input amount.

[0077] It should be understood that the configurations and / or approaches described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have limiting significance, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in other orders, in parallel, or omitted. Likewise, the order of the processes described above may be changed.

[0078] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A server computing device comprising a processor, the processor being configured to: receiving a plurality of input streams from a corresponding plurality of client computing devices, each input stream comprising a plurality of inputs for controlling actions of a corresponding character in a multiplayer online software program; determining a latency of each of the input streams, further comprising determining the latency of each of the plurality of input streams based on detecting whether input is received from each of the plurality of input streams at a target input frequency; identifying a higher latency input stream and a lower latency input stream among the plurality of input streams, wherein an input stream is identified as a higher latency input stream based on detecting the presence of lost input in the input stream according to the target input frequency; as well as A higher latency is introduced into the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream.

2. The server computing device of claim 1, wherein: The higher latency is introduced into each of the plurality of input streams except for the highest latency input stream.

3. The server computing device of claim 1, wherein: The processor is further configured to: predicting a long term latency for each of the plurality of input streams; and The introduced higher latency in one or more lower latency input streams is increased up to a highest predicted long term latency of the plurality of input streams.

4. The server computing device of claim 1, wherein: The processor is configured to introduce the higher latency by buffering input to each of the plurality of input streams.

5. The server computing device of claim 1, wherein: The processor is configured to: rendering an output frame of a software program state for the multi-player online software program for each of the plurality of client computing devices at a controllable frame rate; as well as The rendered output frames are sent to the corresponding plurality of client computing devices in a plurality of output frame streams.

6. The server computing device of claim 5, wherein: The processor is further configured to introduce the higher latency by buffering output frames of each of the plurality of output frame streams.

7. The server computing device of claim 5, wherein: The processor is further configured to introduce the higher latency by reducing the controllable frame rate to less than a target frame rate.

8. The server computing device of claim 7, wherein: The processor is configured to cause each client computing device to interpolate intermediate frames based on one or more output frames in the output frame stream to achieve the target frame rate.

9. The server computing device of claim 8, wherein: The processor is configured to: detecting lost input in the higher latency input stream based on the target input frequency; and Predicted inputs are inserted into the higher latency input stream to fill in the missing inputs.

10. The server computing device of claim 9, wherein: The predicted input is determined based on trained input data including a history of inputs in the input stream.

11. A method for introducing latency on an input stream, comprising: At a server computing device including a processor, receiving a plurality of input streams from a corresponding plurality of client computing devices, each input stream comprising a plurality of inputs for controlling actions of a corresponding character in a multiplayer online software program; determining a latency of each of the input streams, further comprising determining the latency of each of the plurality of input streams based on detecting whether input is received from each of the plurality of input streams at a target input frequency; identifying a higher latency input stream and a lower latency input stream among the plurality of input streams, wherein an input stream is identified as a higher latency input stream based on detecting the presence of lost input in the input stream according to the target input frequency; as well as A higher latency is introduced into the lower latency input stream to reduce the latency difference between the higher latency input stream and the lower latency input stream.

12. The method according to claim 11, characterized in that The higher latency is introduced into each of the plurality of input streams except for the highest latency input stream.

13. The method of claim 11, further comprising: predicting a long term latency of each of the plurality of input streams; as well as The introduced higher latency in one or more lower latency input streams is increased up to a highest predicted long term latency of the plurality of input streams.

14. The method according to claim 11, characterized in that Introducing the higher latency includes buffering input to each of the plurality of input streams.

15. The method of claim 11, further comprising: rendering an output frame of a software program state for the multi-player online software program for each of the plurality of client computing devices at a controllable frame rate; as well as The rendered output frames are sent to the corresponding plurality of client computing devices in a plurality of output frame streams.

16. The method according to claim 15, characterized in that Introducing the higher latency includes buffering output frames of each of the plurality of output frame streams.

17. The method according to claim 15, characterized in that Introducing the higher latency includes reducing the controllable frame rate to less than a target frame rate.

18. The method according to claim 17, characterized in that Further comprising causing each client computing device to interpolate intermediate frames based on one or more output frames in the output frame stream to achieve the target frame rate.

19. A server computing device comprising a processor, the processor being configured to: receiving a plurality of input streams from a corresponding plurality of client computing devices, each input stream comprising a plurality of inputs for controlling actions of a corresponding character in a multiplayer online software program; determining a latency for each of the input streams, including a highest predicted long-term latency; identifying a higher latency input stream and a lower latency input stream among the plurality of input streams, wherein an input stream is identified as a higher latency input stream based on detecting the presence of lost input in the input stream according to a target input frequency; introducing higher latencies up to the highest predicted long-term latency into the lower latency input stream; detecting one or more missing inputs in the higher latency input stream based on the target input frequency; as well as If the one or more missing inputs are greater than a threshold input amount, stopping execution of the multi-player online software program; as well as If the one or more missing inputs are below the threshold input amount, predicted inputs are inserted into the higher latency input stream to fill in the missing inputs.

Images