Video transmission system

Abstract

To prevent increase in video generation processing load on a server and transmission traffic to free from processing that depends on a specific game.SOLUTION: A server device stores user's operation information, estimates the user's operation using stored operation information, generates a video corresponding to the estimated user's operation, and transmits the generated video to the user.

Description

Technical Field

[0001] The present disclosure relates to a technique for transmitting video from a server to a client.

Background Art

[0002] In order to provide a service for remotely playing a game running on a server, a video transmission system has been proposed that sequentially transmits corresponding game videos according to a user's operation. Since such a video transmission system does not require a client to perform game processing, it is possible to play a high-definition game even on a terminal with low processing power. However, there is a problem that the user's perceived quality deteriorates due to an increase in network delay caused by video transmission.

[0003] In order to solve this problem, a system has been proposed that pre-reads and generates all game videos corresponding to all possible inputs of a user and transmits them in advance (for example, Non-Patent Document 1). In theory, zero delay can be achieved in the system of Non-Patent Document 1 if the number of pre-read frames is increased. However, it is necessary to generate and transmit pre-read videos multiplied by the number of pre-read frames of the possible input numbers. For this reason, in Non-Patent Document 1, there is a problem that the video generation processing load and transmission traffic of the server increase explosively.

[0004] On the other hand, a technique for estimating a user's input has been proposed (for example, Non-Patent Document 2). The technique of Non-Patent Document 2 has high accuracy in estimating a user's input. However, since it depends on a specific game, it is not suitable for a cloud game mechanism that enjoys various games, and there is a problem that it cannot assume "server load" and "network capacity", which are bottlenecks of cloud games.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] This disclosure aims to prevent an increase in the server's video generation processing load and transmission traffic, and to enable processing that is not dependent on specific games. [Means for solving the problem]

[0007] The video transmission system of this disclosure is a system in which a server device pre-transmits corresponding video by sequentially estimating user operations on a client device, and comprises a server device of this disclosure and a client device that transmits user operation information to the server device and displays the video transmitted from the server device on a monitor.

[0008] The server and client devices of this disclosure execute the methods of this disclosure. The programs of this disclosure are programs that cause a computer to implement each function of the server device relating to this disclosure, and programs that cause a computer to execute each procedure of the methods executed by the server device relating to this disclosure.

[0009] The server device of this disclosure stores user operation information, estimates the user's operation using the stored operation information, sequentially generates a plurality of videos corresponding to the estimated user's operation, and transmits the generated plurality of videos to the user.

[0010] By adopting the above configuration, this disclosure prevents the increase in the video generation processing load and transmission traffic of server devices due to combinatorial explosion in speculative video transmission, and enables input estimation that is mindful of cloud bottlenecks, thereby enabling a departure from game-dependent processing in input estimation.

[0011] The server device of this disclosure may calculate the probability of the user's operation occurring in the estimation and generate multiple videos with a high probability of occurrence.

[0012] Furthermore, the server device of this disclosure may, in the estimation, create a time-series branch list of user operations starting from the user operations included in the operation information, and generate video corresponding to the user operations with the set maximum number of branches in the time-series branch list.

[0013] Here, the maximum number of branches may be set based on the throughput available in the communication network used when transmitting to the user or the computing resources of the server device.

[0014] Furthermore, during transmission, the bitrate at which the video is transmitted may be controlled based on the probability of occurrence of branching patterns in the time-series branching list.

[0015] Furthermore, the above disclosures can be combined as much as possible. [Effects of the Invention]

[0016] According to this disclosure, it is possible to prevent an increase in the server's video generation processing load and transmission traffic, and to move away from processing that is dependent on specific games. [Brief explanation of the drawing]

[0017] [Figure 1] Shows a configuration example of a sequential transmission system. [Figure 2] Shows an example of transmitting game video in a speculative video transmission system. [Figure 3] Shows an example of a game screen generated by a game server. [Figure 4] Shows an example of a game screen generated by a game server. [Figure 5] Shows a configuration example of a game server according to the present disclosure. [Figure 6] Shows an example of the operation of a game server according to the present disclosure. [Figure 7] Shows an example of an operation log DB in which operation information of N users is stored. [Figure 8] Shows an example of extracting an input pattern. [Figure 9] Shows an example of a time-series branch list. [Figure 10] Shows an example of the operation of the game server of the present embodiment. [Figure 11] Shows an example of a conversion table.

Mode for Carrying Out the Invention

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These examples are merely illustrative, and the present disclosure can be implemented in various modified and improved forms based on the knowledge of those skilled in the art. In the present specification and drawings, components having the same reference numerals indicate the same components.

[0019] (Speculative Video Transmission System) Figure 1 shows an example of the configuration of a speculative video transmission system. In this speculative video transmission system, a client 10 and a game server 20 are connected by an arbitrary communication network 80, and the game server 20 transmits game video of the game state corresponding to the operation information of the client 10 each time the client 10 performs an operation.

[0020] Client 10 is A controller 11 operated by the user, An operation input unit 12 transmits operation information output from the controller 11 when the controller 11 is operated, A video receiving unit 13 that receives video data from the game server 20, A monitor 14 that displays the received video data, It is equipped with.

[0021] Game server 20, An operation input receiving unit 21 that receives operation information from client 10, The game state update unit 22 updates the game state according to the operation information received by the operation input receiving unit 21, A speculative processing unit 23 generates game images corresponding to the updated game state, A video transmission unit 24 that transmits video data of the game screen, It is equipped with.

[0022] (Summary of this disclosure) Figure 2 shows an example of game video transmission in a speculative video transmission system. When the game server 20 of this disclosure receives operation information transmitted from the client 10 at time t0, it transmits to the client 10 before time t3 image data capable of generating a game screen of a game state that may occur at any predetermined time t3 after the operation information. When the controller 11 is operated at time t3, the client 10 uses the previously received image data to display a game screen corresponding to the operation of the controller 11.

[0023] Figure 3 shows an example of a game screen generated on the game server 20. For example, the controller 11 is equipped with A button, B button, up button, ... R button, and L button, and when the A button is pressed, multiple game screens may be generated, and when the B button is pressed, multiple game screens may be generated. If all of these are processed and transmitted as video, the latency of the communication network 80 will increase.

[0024] Therefore, in speculative video transmission in cloud games, when the game server 20 generates a game screen, it estimates the input pattern from past input patterns for each game, reduces the number of input candidates, then performs speculative processing on the game video, and transmits the video obtained through this processing.

[0025] For example, as shown in Figure 4, based on past input patterns, it is estimated that the B, R, and L buttons will not be pressed. This allows for the omission of game screens generated when the B, R, and L buttons are pressed, thereby reducing the number of game screens sent to client 10. Similarly, for game screens generated after the A button is pressed, input patterns are estimated from past input patterns to reduce the number of game screens generated. Therefore, this disclosure can reduce the number of processing threads and the amount of video data transmitted.

[0026] The extent to which the number of input candidates is reduced is arbitrary, but it may be determined based on information from the communication network 80, the game, and the client 10. For example, the following can be used: • The maximum number of branches is set based on the maximum traffic volume available on the communication network 80, and the number of input candidates is reduced by performing input pattern estimation up to the maximum number of branches. Based on the processing capabilities of game server 20, including the computing resources of game server 20, the maximum number of branches is set, and input pattern estimation is performed up to the maximum number of branches to reduce the number of input candidates. When transmitting speculative video, the bitrate is controlled based on the probability of branching patterns occurring.

[0027] (First embodiment) Figure 5 shows an example of the configuration of the game server 20. The video transmission system of this embodiment includes an operation log DB 31, a time-series branch list 32, and a future input estimation unit 36, instead of the game state update unit 22 shown in Figure 1.

[0028] The game server 20 functions as the server device described in this disclosure. The game server 20 can also be implemented using a computer and a program, and the program can be recorded on a recording medium or provided over a network.

[0029] The operation input receiving unit 21 stores user operation information for the controller 11 in the operation log DB 31. The operation log DB 31 stores the controller 11 operation log for each game. Note that the operation information is not limited to controller 11 operation information and may include arbitrary operation information by the user. The time-series branch list 32 obtains the following from the operation information accumulated in the operation log DB 31. • Actual input pattern • Branching patterns starting from each input pattern • Probability of each branching pattern occurring In this way, the input patterns that the user can take are listed, including their probability of occurrence. This embodiment uses this list to reduce the amount of video data compared to a brute-force approach that tries all possible patterns.

[0030] The future input estimation unit 36 ​​estimates the operation of the controller 11 that will be input in the future based on the list of branch patterns, and passes this estimate, including the probability of occurrence, to the speculative processing unit 23. The speculative processing unit 23 performs speculative processing on the game video when it receives the selected input pattern as operation information. The video transmission unit 24 transmits video data obtained through speculative processing.

[0031] Figure 6 shows an example of the operation of the game server 20. S10. When the operation input receiving unit 21 receives operation information for the controller 11 from each user, it stores the operation information in the operation log DB 31. As a result, time-series operation information is accumulated in the operation log DB 31. Figure 7 shows an example of the operation log DB 31 with operation information for N users accumulated.

[0032] S20. Unique input patterns are extracted from time-series operation information. This reduces the number of input patterns that do not occur. Figure 8 shows an example of input pattern extraction.

[0033] S30. The time-series branches from each input pattern listed in the input list are extracted and listed. Figure 9 shows an example of the time-series branch list 32. For each extracted input pattern, the time-series branch list 32 creates a time-series branch list for the input pattern that was input later. For example, for Input[0], it is listed as Transition[0][0], Transition[0][1], Transition[0][2], Transition[0][3], and so on. This allows for the creation of a time-series branch list for each input pattern, and limits the branch patterns to be processed, such as branch patterns AA and BB.

[0034] Here, the probability of occurrence from each input pattern listed in the input list is measured for each branching pattern. The list length of each list is also measured. In this embodiment, the maximum list length is denoted as α. Figure 9 shows an example where the list length is 4.

[0035] Note that while Figure 9 shows only one branch in the branching pattern, the maximum number of branches included in the branching pattern can be any predetermined number. Also, the input content is not interpreted during the analysis in steps S10 to S30. Furthermore, log analysis is performed using only operation information and is independent of the game title. In addition, multiple branching patterns corresponding to the game situation are generated in advance for each game.

[0036] S41. When Input[0] is input to the operation input receiving unit 21, the speculative processing unit 23 performs speculative processing of Transition[0][0], and then performs speculative processing of Transition[0][1]. In this embodiment, speculative processing is performed for each extracted input pattern.

[0037] This embodiment is applicable to the parallel execution of games using thread processing. For example, to generate multiple game states that may occur at time t+1, t+2, ..., starting from the game state at time t, the parallel execution of steps S10 to S30 and S41 using thread processing may be performed, and all generated game images may be advanced to the user.

[0038] (Second embodiment) The game server 20 may include resource information 33, a maximum branch count setting unit 34, and a maximum branch count table 35 for the future input estimation unit 36 ​​to estimate future inputs. Resource information 33 describes the maximum resources of the communication network 80 and the maximum processing capacity of the game server 20. The maximum number of branches setting unit 34 sets the maximum number of branches in the time-series branch list 32 based on the resource information. Table 35, which lists the maximum number of branches in the time-series branch list 32, contains this information.

[0039] The future input estimation unit 36 ​​selects from the input patterns with the highest probability of occurrence, from the maximum number of branches listed in the maximum branch count table 35. The maximum number of branches can be determined by dividing the video bitrate by the maximum resources available in the communication network 80, or by dividing the number of clock cycles per speculative operation by the total processing capacity of the game server 20.

[0040] The video transmission unit 24 transmits video data starting with the one that has the highest probability of occurring, using a higher bitrate.

[0041] Figure 10 shows an example of the operation of the game server 20 in this embodiment. It shows an example of allocating one thread to each speculative process. In this embodiment, steps S31 to S34 are provided after step S30, step S42 is provided instead of step S41, and step S50 is provided after step S42.

[0042] S31. Measure the list length of each list and obtain the maximum list length (α) of each branching sequence and the probability of occurrence of the input pattern. S32. When the maximum list length α is less than or equal to (β-1), set the number of subthreads to perform speculative execution to α. α is compared with (β-1) because one is needed for the main thread. S33. When the maximum list length α is greater than (β-1), the branching sequence is updated by obtaining up to (β-1) branching patterns with a high probability of occurrence. This results in each branching sequence having β-1 branching patterns. S34. Set the number of subthreads to perform speculative execution to (β-1). S42. Perform speculative processing based on the generated list. A maximum of β-1 frames are rendered. S50. The video transmission unit 24 encodes and transmits each output at a bitrate corresponding to the probability of occurrence of the branching pattern. The bitrate is set based on the probability of occurrence of each branching pattern according to the allowable traffic volume.

[0043] Thus, in step S32, the number of threads to perform speculative processing is set according to the list length α obtained by the preprocessing. It is also possible to set the number of threads considering the probability of occurrence, depending on the system requirements.

[0044] Furthermore, by executing step S34, the game state of the sub-thread is advanced according to the previously acquired branch pattern, relative to the main thread that runs the game on the real time axis. Steps S42 and S50 are executed to generate game screens according to the game state of each sub-thread and send them to the user. The user displays the screen corresponding to their actual operation from among the multiple game screens received. This makes it possible for this embodiment to perform bitrate control that takes into account the probability of occurrence according to traffic requirements.

[0045] (Third embodiment) In step S10, the operation information received by the operation input receiving unit 21 differs depending on the specifications of the controller 11. Therefore, in this embodiment, the operation information is converted into a string using a conversion table.

[0046] Figure 11 shows an example of a conversion table. The conversion table is designed according to the number of buttons on the controller 11 to obtain operation logs. For example, the analog stick (including the directional pad) is binarized using a threshold value. In this embodiment, by assigning two or one digit to 18 types of operation information, the operation performed on the controller 11 is represented by a 24-digit string.

[0047] Here, all inputs from controller 11 are listed as strings. Additionally, a conversion table is prepared for each controller 11 of the same type, allowing for arbitrary arrangement and length conversions. Furthermore, by increasing the number of digits according to the number of thresholds to be set, it is possible to handle delicate operations such as half-inputs.

[0048] By employing such a conversion table, genre-independent log analysis becomes possible. The input pattern shown in Figure 7 is operation information converted into a string using the method of this embodiment. The example shown in Figure 7 shows input patterns for two frames based on N operation information entries. [Explanation of Symbols]

[0049] 10: Client 11: Controller 12: Operation Input Section 13: Video receiving unit 14: Monitor 20: Game Server 21: Operation Input Receiving Unit 22: Game Status Update Section 23: Speculative Processing Unit 24: Video transmission unit 31: Operation Log Database 32: Timeline Branch List 33: Resource Information 34: Maximum number of branches setting section 35: Maximum number of branches table 36: Future Input Estimation Unit 80: Communication Network

Claims

1. It stores user operation information for client devices, The user's actions are estimated using the accumulated operation information. Multiple videos corresponding to the estimated user's actions are generated sequentially. The generated plurality of videos are transmitted to the client device. A server device connected to the aforementioned client device via a communication network, Based on the amount of traffic available on the aforementioned communication network, the maximum number of branches in the time-series branch list of user operations, starting from the user's operation included in the operation information, is set. An input list is created by extracting the user's input patterns from the aforementioned operation information. The time-series branching list is created by listing the time-series branching patterns from the input patterns listed in the input list. The probability of occurrence from the input pattern to the branch pattern in the time-series branch list is measured for each branch pattern. To generate the multiple videos corresponding to the branching pattern with the highest probability of occurrence among the time-series branching list, Server device.

2. The operation information received from the client device is converted into a string in which each operation is represented by a predetermined number of digits, Using the aforementioned string pattern, the input pattern and the branch pattern are extracted. The server device according to claim 1.

3. The maximum value of the list length of the branching pattern and the probability of occurrence are calculated, When the maximum list length obtained by calculation is less than or equal to the set value, the plurality of videos corresponding to the branching pattern, equal to the list length, are generated. The server device according to claim 2.

4. Encoding the plurality of video images corresponding to the branch pattern at a bitrate corresponding to the probability of occurrence of the branch pattern, The server device according to claim 3.

5. A server device according to any one of claims 1 to 4, A client device that transmits the operation information by the user to the server device and displays the video operated by the user from among the multiple videos transmitted from the server device on a monitor, A video transmission system equipped with the following features.

6. A server device connected to a client device via a communication network, It stores user operation information, The user's actions are estimated using the accumulated operation information. Generate multiple videos corresponding to the estimated user's actions, The generated multiple videos are transmitted to the user. It is a method, Based on the amount of traffic available on the aforementioned communication network, the maximum number of branches in the time-series branch list of user operations, starting from the user's operation included in the operation information, is set. An input list is created by extracting the user's input patterns from the aforementioned operation information. The time-series branching list is created by listing the time-series branching patterns from the input patterns listed in the input list. The probability of occurrence from the input pattern to the branch pattern in the time-series branch list is measured for each branch pattern. To generate the multiple videos corresponding to the branching pattern with the highest probability of occurrence among the time-series branching list, method.

7. The server device is connected to the client device via a communication network. Procedure for accumulating user operation information of client devices, A procedure for estimating the user's actions using accumulated operation information, A procedure for generating multiple videos corresponding to the estimated user's actions, A procedure for transmitting the generated multiple videos to the client device, A program to execute, Based on the amount of traffic available on the aforementioned communication network, the maximum number of branches in the time-series branch list of user operations, starting from the user's operation included in the operation information, is set. An input list is created by extracting the user's input patterns from the aforementioned operation information. The time-series branching list is created by listing the time-series branching patterns from the input patterns listed in the input list. The probability of occurrence from the input pattern to the branch pattern in the time-series branch list is measured for each branch pattern. To generate the multiple videos corresponding to the branching pattern with the highest probability of occurrence among the time-series branching list, program.

Images