Communication system, device, method and program

Abstract

To reduce a response delay in cloud gaming while solving problems of communication bands, client calculation resources and multiuser responses.SOLUTION: The present disclosure relates to a communication system in which a client and a server are connected over a communication network. In the communication system, the server predicts a command to be transmitted from the client based on a command received from the client, generates a video image corresponding to the predicted command, when receiving a command from the client, collates the received command with the predicted command, and in a case where the received command is the predicted command, transmits the generated video image to the client.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This paper relates to technology that improves the user's quality of experience in cloud gaming. [Background technology]

[0002] (Cloud gaming) Cloud gaming is a service that allows users to play games running on a server on a remote device. Game images corresponding to commands issued by the user are sequentially transmitted to the device. This method does not require the client device to process the game, so even devices with low processing power can play high-definition games. However, the increased network latency caused by video transmission can degrade the user's quality of experience.

[0003] A transaction between a server and a client in cloud gaming involves the following steps: S1: The client sends a command input to an input device. S2: The server receives the command via the communication network. S3: The server updates the game with the received commands and generates game images. S4: The server sends the updated game video to the client.

[0004] In cloud gaming, an increase in response latency (the time between a player's operation and the reception of game footage) will affect the player's gaming experience. To address this issue, a technology that implements server-client collaborative speculative processing has been proposed (see, for example, Non-Patent Document 1).

[0005] In Non-Patent Document 1, the server uses speculative processing to estimate user input, generates multiple candidate game images, and then transmits them. If the estimation is accurate, the client immediately outputs game images in response to the command entered at the client. If the estimation is inaccurate, the client corrects the game images received from the server and generates appropriate game images.

[0006] However, the following issues exist in server-client collaborative speculative processing. First challenge: communication bandwidth In order to send multiple patterns of estimated game footage to the client side, a communication bandwidth several times larger than that of normal video transmission is required. Challenge No. 2: Client Resources Since image compensation processing is required on the client side, the client also needs to have the computational resources to process the images. In particular, if the client is a device with limited computational resources, such as a smartphone, image compensation processing may not be possible. Third challenge: Supporting multiple users Because video compensation processing is performed on the client side, it is difficult to implement in multiplayer games where multiple users participate and play simultaneously. Furthermore, the game information between players becomes asynchronous, which can lead to an unfair gaming experience. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] K. Lee et al., “Outatime: Using speculation to enable low-latency continuous interaction for mobile cloud gaming”, Proceedings of the 13th Annual International Conference on Mobile Systems Applications and Services, pp. 151-165, May 2015. https: / / doi.org / 10.1145 / 2742647.2742656 Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure aims to reduce response delays in cloud gaming while solving the aforementioned problems of communication bandwidth, client computing resources, and multi-user support. [Means for solving the problem]

[0009] The communication system of the present disclosure includes: A communication system in which a client and a server are connected via a communication network, The server predicting a command to be sent from the client based on a command received from the client; Generate an image corresponding to the predicted command, Upon receiving a command from the client, matching the received command with the expected command; If the received command is the predicted command, the generated video is sent to the client.

[0010] The device of the present disclosure comprises: A device that functions as a server connected to a client through a communication network, predicting a command to be sent from the client based on a command received from the client; Generate an image corresponding to the predicted command, Upon receiving a command from the client, matching the received command with the expected command; If the received command is the predicted command, the generated video is sent to the client.

[0011] The method of the present disclosure comprises: A method executed by a server connected to a client through a communications network, comprising: predicting a command to be sent from the client based on a command received from the client; Generate an image corresponding to the predicted command, Upon receiving a command from the client, matching the received command with the expected command; If the received command is the predicted command, the generated video is sent to the client.

[0012] The program of the present disclosure is a program for causing a computer to realize each functional unit of an apparatus according to the present disclosure, and is a program for causing a computer to execute each step of a method performed by an apparatus according to the present disclosure. [Effects of the Invention]

[0013] According to the present disclosure, the game images sent from the server to the client are only those corresponding to the commands sent from the client, and no image compensation processing of the game images is performed on the client, thereby reducing response delays in cloud gaming while solving the problems of communication bandwidth, client computing resources, and multi-user support. [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates an example system configuration of the present disclosure. [Figure 2] 10 shows an example of a processing flow between a server and a client in a communication system according to the present disclosure. [Figure 3] 10 shows an example of the flow of processing between the server and the client when prediction is not performed in the server. [Figure 4] 1 shows an example of a processing flow in a client and a server. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0016] (Summary of the Disclosure) FIG. 1 shows an example of a system configuration according to the present disclosure. In the communication system according to the present disclosure, a client 11 and a server 13 are connected via a communication network 12. The number of clients 11 can be any number equal to or greater than one. The system according to the present disclosure executes server-based speculative processing. The client 11 and server 13 according to the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0017] Specifically, in the present disclosure, the server 13 predicts the next command that may occur, generates game footage for the predicted commands in descending order of probability, and renders and buffers these. When the server 13 receives a command that is actually input at the client 11, the server 13 selects a game footage paired with the actually received command from the game footage that has been rendered in advance, and transmits the game footage to the client 11.

[0018] FIG. 2 shows an example of the flow of processing between the server 13 and the client 11 in the communication system of the present disclosure. A command i0 sent by the client 11 at time t0 is received by the server 13 at time t1.

[0019] When the server 13 receives the command i0, it executes the following process. S21: The server 13 predicts the next command that may occur based on multiple states before the command i0. In this embodiment, the server 13 predicts multiple next commands for the command i0. The prediction results are stored as a predicted command set (i1 1 ,i1 2 ,i1 3 ,…i1 n ) where n is any natural number greater than or equal to 1. The predicted command set (i1 1 ,i1 2 ,i1 3 ,…i1 n ) can be predicted using any prediction algorithm that outputs the next command likely to occur in descending order of probability, such as an N-order Markov chain. In this case, the N commands before command i0 are used to create an input sequence that serves as input to the prediction algorithm.

[0020] S22: The server 13 receives the predicted command set (i1 1 ,i1 2 ,i1 3 ,…i1 n ), the video frames (p1 1 ,p1 2 ,p1 3 ,…p1 n ) and generate a set of predicted commands (i1 1 ,i1 2 ,i1 3 ,…i1 n ) and video frame group (p1 1 ,p1 2 ,p1 3 ,…p1 n) is rendered and buffered. At this time, the command and the video frame corresponding to that command are linked and stored.

[0021] S23: When the server 13 receives the actual command i1, it creates a predicted command set (i1 1 ,i1 2 ,i1 3 ,…i1 n ), a command that matches the actual command i1 is extracted, and a video frame p1 that pairs with the extracted command i1 is transmitted to the client 11. As a result, the video frame p1 is displayed on the client 11 side.

[0022] Here, in the case of a multi-user system with multiple clients 11, the server 13 executes steps S21 to S23 for each client 11. At this time, the server 13 identifies commands from the clients 11 and synchronizes the transmission of video frames p1 to the clients 11. This allows each client 11 to obtain game video based on common game information.

[0023] 3 shows an example of the flow of processing between the server 13 and the client 11 when prediction is not performed in the server 13. When prediction is not performed in the server 13, the server 13 starts generating a video frame p1 from time t2 when it receives the command i1. Therefore, the timing of transmitting the video frame p1 from the server 13 is delayed by the time ΔR1 required to generate the video frame p1.

[0024] (Advantages of this disclosure) (1) Reduces required communication bandwidth The server 13 holds the video corresponding to the predicted command group and selectively transmits only the video frame corresponding to the request of the client 11, so the communication band is not constricted. (2) No video compensation processing is required on the client 11 side The client 11 does not require a module for advanced image processing. (3) Support for multiple users The server 13 can collectively manage everything from predicting the actions of multiple clients 11 to rendering and selecting video frames, allowing each client 11 to acquire game images based on common game information.

[0025] (4) Reduction of encoding delay The server 13 may complete encoding of the game video before the command sent from the client 11 arrives at the server 13. This makes it possible to reduce the time required to encode the game video in response to a request from the client 11.

[0026] FIG. 4 shows an example of a processing flow in the client 11 and the server 13. The client 11 transmits a command i0 to the server 13 (S11). When the server 13 receives the command i0 (S31), it predicts the next command that may occur (S32). The server 13 also predicts multiple next commands for the command i0.

[0027] Here, an N-order Markov chain can be used as a prediction algorithm. An N-order Markov chain is a stochastic process in which the next state is determined depending on the past N state histories including the current state. In this disclosure, N input sequences including i0 are used as inputs, and n output sequences with high probability are obtained. If the output sequences are a set of predicted commands (i1 1 ,i1 2 ,i1 3 ,…i1 n ) is equivalent to

[0028] The server 13 receives the predicted command set (i1 1 ,i1 2 ,i1 3 ,…i1 n ) are generated and encoded using an arbitrary compression algorithm (S33). 1 ,i1 2 ,i1 3 ,…i1 nThe encoded and generated video frames corresponding to (p1 1 ,p1 2 ,p1 3 ,…p1 n ) The encoding order is, for example, i1 1 Alternatively, the video frames may be encoded in descending order of probability based on the state transition.

[0029] The server 13 determines the update timing (S34). 1 ,i1 2 ,i1 3 ,…i1 n ) continues to be encoded until the game update timing arrives. Here, the update timing is the timing at which the video is updated in the client 11, and corresponds to the timing at which the video frame corresponding to the command is transmitted to the client 11, as shown by times t1 and t2 in FIG. 2. This timing may be fixed, or the update timing may be shifted adaptively.

[0030] The server 13 determines whether to perform encoding (S35). 1 ,i1 2 ,i1 3 ,…i1 n ), and if there is a generated video frame (for example, p1) corresponding to the command i1 that actually arrived from the client 11 (Yes in S35), it is extracted and the video frame is sent to the client 11 (S37). k ) (No in S35), the server 13 performs encoding processing (S36) and transmits the encoded video frame to the client 11 (S37). As a result, the video corresponding to the command that actually arrived from the client 11 is transmitted.

[0031] The client 11 receives the video transmitted from the server 13 (S12), decodes it (S13), and executes drawing processing (S14). As a result, the video corresponding to the command i1 actually input at the client 11 is displayed at the client 11.

[0032] In the above embodiment, an example in which the number of output sequences is two or more is shown, but n may be 1. In this case, when the server 13 receives a command from the client 11, it compares the received command with a predicted command, and if the received command is the predicted command, it transmits the generated video to the client. [Industrial Applicability]

[0033] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0034] 11: Client 12: Communication Network 13: Server

Claims

1. A communication system in which a client and a server are connected via a communication network, The server predicting commands to be sent from the client and the order of probabilities of the commands using a Markov chain based on the commands received from the client; Generate an image corresponding to the predicted command, Encoding the generated images in descending order of the probability; Upon receiving a command from the client, matching the received command with the expected command; If the received command is the expected command, transmitting the encoded video to the client. Communication system.

2. The server predicting commands to be transmitted from the plurality of clients and the order of the probabilities of the commands for each of the clients; Generates an image corresponding to each predicted command, When commands are received from a plurality of the clients, a command that matches the received command is extracted from the predicted commands for each of the clients; synchronizing transmission of video corresponding to the extracted commands to the plurality of clients; The communication system of claim 1 .

3. A device that functions as a server connected to a client through a communication network, predicting commands to be sent from the client and the order of probabilities of the commands using a Markov chain based on the commands received from the client; Generate an image corresponding to the predicted command, Encoding the generated images in descending order of the probability; Upon receiving a command from the client, matching the received command with the expected command; If the received command is the expected command, transmitting the encoded video to the client. Device.

4. A method executed by a server connected to a client through a communications network, comprising: predicting commands to be sent from the client and the order of probabilities of the commands using a Markov chain based on the commands received from the client; Generate an image corresponding to the predicted command, Encoding the generated images in descending order of the probability; Upon receiving a command from the client, matching the received command with the expected command; If the received command is the expected command, transmitting the encoded video to the client. method.

5. A program for causing a computer to realize each of the functional units provided in the device according to claim 3.

Images