Game system, game terminal, display adjustment device, and computer program

Abstract

This game system comprises a game server, a game terminal, and a terminal device. The game server and the game terminal are connected via a communication network. The game server generates a game image in a first period. The display device displays a game image in a second period shorter than the first period. When the game terminal has acquired a new game image from the game server before the timing at which the display device is to display the next game image is reached, the game terminal transmits the new game image to the display device. When the game terminal has not acquired a new game image from the game server when the timing at which the display device is to display the next game image is reached, the game terminal transmits an alternative game image different from the new game image to the display device.

Description

Game system, game terminal, display adjustment device and computer program

[0001] The present invention relates to a technique for reducing the delay from image generation to display.

[0002] In recent years, a type of online game known as cloud gaming has emerged, where players can play together in real time via a communication line with players in remote locations, such as in competitive games.

[0003] The cloud gaming environment consists of a server and a client. A console and a monitor are connected to the client. In cloud gaming, the server controls the game program.

[0004] In cloud gaming, game images generated by the server are transferred to the client and then to the monitor. The server generates game images at regular intervals. The monitor displays the game images generated by the server at regular intervals.

[0005] In cloud gaming, minimizing the time (latency) between when the server generates the game image and when it is displayed on the monitor is crucial. This latency includes the time it takes for the server to send the game image to the client and the time it takes for the client to transmit the game image to the monitor.

[0006] For action-packed cloud games, reducing latency is especially important. Even the slightest difference in the time lag between when a player inputs a command and when the game image reflecting the input is displayed can have a significant impact on the experience of the game. Players want a response as quickly as possible.

[0007] Japanese Patent Application Laid-Open No. 2016-220154

[0008] It is well known that the time required to transmit and display game images from the client to the monitor can be reduced by shortening the cycle (frame rate) at which the monitor switches game images. In other words, the faster the monitor frame rate, the shorter the delay time.

[0009] On the other hand, if the monitor frame rate is set faster than the frequency at which the server generates game images, the server processing will not be able to keep up with the monitor display speed.

[0010] The present invention was completed based on the recognition of the above-mentioned problems, and its main purpose is to provide a technology for continuously displaying game images on a monitor when the monitor frame rate is increased to reduce display delay.

[0011] A game system according to one aspect of the present invention includes a game server, a game terminal, and a display device. The game server and the game terminal are connected via a communications network, and the game server includes a game control unit that controls the progress of the game in accordance with operation signals from the game terminal, an image generation unit that generates game images at a first interval, and an image transmission unit that transmits the game images to the game terminal. The game terminal includes an input unit that accepts operations from a player for the game, an operation transmission unit that transmits operation signals indicating the operations from the player to the game server, an image reception unit that receives game images from the game server, and an image transmission unit that transmits the received game images to the display device. The display device includes a display control unit that displays the game images at a second interval that is shorter than the first interval. The image transmission unit of the game terminal transmits the new game image to the display device when a new game image has been acquired from the game server before the display device is due to display the next game image, and transmits a substitute game image different from the new game image to the display device when a new game image has not been acquired from the game server when the display device is due to display the next game image.

[0012] In one aspect of the present invention, a game terminal is connected to a game server via a communications network and includes an input unit that accepts operations for a game from a player, an operation transmission unit that transmits operation signals indicating the operations from the player to the game server, an image receiving unit that receives game images generated from the game server based on the operation signals, and an image transmission unit that transmits the received game images to a display device. In a case where the game server generates game images at a first cycle and the display device displays the game images at a second cycle that is shorter than the first cycle, the image transmission unit transmits the new game image to the display device when a new game image has been obtained from the game server before the timing for the display device to display the next game image, and transmits a substitute game image different from the new game image to the display device when a new game image has not been obtained from the game server before the timing for the display device to display the next game image.

[0013] A display adjustment device according to one aspect of the present invention includes an image receiving unit that receives game images from a game executing entity, and an image transmitting unit that transmits the received game images to a display device. When the game executing entity generates game images at a first cycle and the display device displays the game images at a second cycle that is shorter than the first cycle, the image transmitting unit transmits the new game image to the display device if a new game image is acquired from the game executing entity before the timing at which the display device displays the next game image, and transmits a substitute game image different from the new game image to the display device if a new game image is not acquired from the game executing entity before the timing at which the display device displays the next game image.

[0014] According to the game system of the present invention, even when a monitor with a faster frame rate for image display is introduced, the monitor is more likely to continue to display game images stably.

[0015] FIG. 1 is an overall configuration diagram of a typical local environment game system. FIG. 2 is a time chart showing a typical game image transfer process in a local environment game system. FIG. 3 is an overall configuration diagram of a typical remote environment game system. FIG. 4 is a time chart showing a typical game image transfer process in a remote environment game system. FIG. 5 is a functional block diagram of a server in this embodiment. FIG. 6 is a functional block diagram of a game terminal in this embodiment. FIG. 7 is a functional block diagram of a monitor in this embodiment. FIG. 8 is a time chart showing a game image transfer process in this embodiment. FIG. 9 is a time chart showing a game image transfer process in a modified example.

[0016] To clarify the gist of the present invention, a typical game image transfer process will first be described. A local environment game system will be described with reference to Figures 1 and 2. A remote environment game system will be described with reference to Figures 3 and 4. Figures 5 and subsequent figures will describe the game image transfer process when the monitor frame rate is increased in a remote environment game system according to this embodiment.

[0017] First, an overview of a local environment game system will be described. FIG. 1 is a diagram showing the overall configuration of a typical local environment game system. In the local game system 100, a personal computer 101 is connected to a monitor 102 and an input device 103. The local game system 100 is not a server-client type, but a standalone type game system in which a game is run solely by the personal computer 101. Specifically, the input device 103 is assumed to be a keyboard, mouse, joystick, or the like. The game program is controlled by the personal computer 101. Although not shown in FIG. 1, the personal computer 101 is equipped with a CPU (Central Processing Unit) that executes the game program. The personal computer 101 also is equipped with a GPU (Graphics Processing Unit), which is a processor that generates game images.

[0018] When a player performs an input operation using the input device 103, this input operation is converted into an operation signal and sent to the personal computer 101. The personal computer 101 calculates the placement of objects in the game in accordance with the operation signal. The objects may be, for example, characters, buildings, or other three-dimensional structures. The GPU in the personal computer 101 generates a game image corresponding to the results obtained by this calculation process. The game image is transferred to the monitor 102 and then displayed.

[0019] 2 is a time chart showing a typical game image transfer process in a local game system. Specifically, the process is shown in which a game image generated by a GPU in a personal computer 101 is transferred to a monitor 102 and displayed on the monitor 102.

[0020] In FIG. 2 , the frame rate for image generation by the GPU (hereinafter, the frame rate for image generation will be referred to as the "generation rate") is R (times / second). That is, the GPU performs rendering (performing three-dimensional calculations, determining the color to be displayed for each pixel on the monitor 102, and generating a game image) R times per second. Hereinafter, rendering means "generation of a game image." The frame rate for image display on the monitor 102 (hereinafter, the frame rate for image display will be referred to as the "display rate") is also R (times / second). That is, the monitor 102 displays game images R times per second. In other words, in FIG. 2 , the generation rate of the GPU and the display rate of the monitor 102 are the same. For example, when the generation rate is R = 60 Hz, the display rate is also R = 60 Hz. In this case, the GPU generates a new game image approximately every 17 milliseconds (= 1 (second) ÷ 60 (Hz)), and the monitor 102 also displays the generated game images approximately every 17 milliseconds.

[0021] The time given for generating one game image will be called a processing timeslot (PTS). A processing timeslot consists of a rendering time (RT) and a wait time (WT). During the rendering time, the GPU renders (generates) the game image. The rendering time is not constant, but it is guaranteed to be completed within the processing timeslot. The remaining time from the completion of rendering to the completion of the processing timeslot is the wait time. During the wait time, the GPU waits in preparation for rendering a new game image.

[0022] 2, the processing time slot PTS0 is from time t0 to t1, the processing time slot PTS1 is from time t1 to t2, and the processing time slot PTS2 is from time t2 to t3. As described above, if the display rate R is 60 Hz, one processing time slot corresponds to approximately 17 milliseconds, and the rendering time is 17 milliseconds or less.

[0023] The GPU generates a game image for each processing time slot. To distinguish between multiple game images, the Nth game image generated since the start of the game will be referred to as "game image (N)." When processing time slot PTS0 for generating game image (N) is completed, the GPU generates the next game image (N+1) in the next processing time slot PTS1.

[0024] The rendering time for game image (N) is denoted as "rendering time RT(N)." Furthermore, the wait time after the rendering time RT(N) is completed is denoted as "wait time WT(N)." First, the GPU generates game image (N) during rendering time RT(N) from time t0 to t4 within processing time slot PTS0. During wait time WT(N) from time t4 to time t1, when the next processing time slot PTS1 starts, the GPU waits in preparation for generating a new game image (N+1). When the start time t1 of the next processing time slot PTS1 arrives, game image (N) is transferred to the monitor 102.

[0025] Here, the delay time from when a game image is transferred to the monitor 102 until the game image is actually displayed on the monitor 102 is referred to as the monitor delay time (MD). Specifically, the monitor delay time refers to the time required for display preparation, from when the personal computer 101 sends a control signal to the monitor 102 instructing the monitor 102 to display a game image, until the monitor 102 applies a drive voltage to all pixels and emits light. The control signal will be described later.

[0026] The monitor delay time required to prepare for display of the game image (N) is denoted as the "monitor delay time MD(N)." The game image (N) is transferred to the monitor 102 from time t1 to t5. The transferred game image (N) is displayed on the monitor 102 at time t6 after a monitor delay time MD(N) from time t5 to t6.

[0027] After completing processing time slot PTS0, the GPU generates a new game image (N+1) during processing time slot PTS1. The rendering time RT(N+1) starts at time t1, which is before time t6 when the previous game image (N) is displayed on the monitor 102. That is, the GPU starts generating the next game image (N+1) before the display of the previously generated game image (N) begins. When processing time slot PTS1 completes at time t2, the new game image (N+1) is transferred to the monitor 102. After a monitor delay time MD(N+1), the game image (N+1) is displayed on the monitor 102 at time t7, which is the start of processing time slot PTS2.

[0028] In Figure 2, the delay time until game image (N) is displayed on the monitor 102 is the time from time t0 when the GPU starts generating game image (N) to time t6 when the monitor 102 displays game image (N). This delay time includes the monitor delay time MD(N). The same applies to game image (N+1). In this way, the monitor 102 displays the game image after the delay time has elapsed since the GPU started generating the game image. To improve the feel of the operation, and in particular the responsiveness of the game, it is desirable to have a shorter delay time (times t0 to t6).

[0029] Next, an overview of a remote game system will be described. FIG. 3 is a diagram showing the overall configuration of a typical remote game system. A server 201 is connected to a game terminal 300 via a communication network 202. Specifically, the game terminal 300 is assumed to be a personal computer, a home game console, or an arcade game machine. The communication network 202 is assumed to be the Internet or a dedicated line. The game terminal 300 is connected to a monitor 400 and an input device 500. The game program is controlled by the server 201. Although not shown in FIG. 3 , the server 201 is equipped with a CPU that executes the game program. The server 201 also is equipped with a GPU for generating game images. The server 201 is equipped with an encoder that encodes game images before transmitting them via the communication network 202. Here, encoding refers to compressing the game images and dividing them into packets. The game terminal 300 is equipped with a decoder that decodes encoded game images (hereinafter referred to as "encoded images") transferred via the communication network 202. Here, decoding refers to restoring a game image from an encoded image.

[0030] When a player performs an input operation using the input device 500, the game terminal 300 transmits an operation signal related to this input operation to the server 201 via the communication network 202. The server 201 calculates the placement of objects in the game in accordance with the operation signal. Based on the results obtained from this calculation process, the server 201 generates a game image. The game image is encoded and then transmitted to the game terminal 300. The game terminal 300 decodes the encoded image transmitted from the server 201. The decoded game image is transmitted from the game terminal 300 to the monitor 400 and then displayed.

[0031] Fig. 4 is a time chart showing a typical game image transfer process in a game system in a remote environment. In the process shown in Fig. 4, a game image generated by the server 201 is converted into an encoded image. The encoded image is then transmitted to the game terminal 300 and decoded by a decoder. The decoded game image is transmitted to the monitor 400 and displayed. In Fig. 4, unlike the local system of Fig. 2, the game image must be transferred via the communication network 202, which adds overhead (accompanying work) such as encoding, network transmission, and decoding.

[0032] The time allotted for displaying one game image is called a Display Timeslot (DTS). Monitor 400 displays the game image that is ready to be displayed at the start of the Display Timeslot.

[0033] 4, the generation rate of the GPU and the display rate of the monitor 400 are both R (times / second). In addition, in Fig. 4, the period from time t4 to t5 is display time slot DTS0, and the period from time t5 to t6 is display time slot DTS1.

[0034] First, the GPU generates a game image (N) during a rendering time RT(N) from time t0 to t7 within the processing time slot PTS0. Here, when the encoder completes generation of the game image (N) at time t7, it begins encoding the game image (N) without waiting for the wait time WT(N) to end. The processing time required for encoding is referred to as the encoding time (EC). The encoding time for the game image (N) is denoted as the "encoding time EC(N)." The game image (N) is converted into an encoded image (N) after the encoding time EC(N) from time t7 to t8. The encoded image (N) is transmitted to the game terminal 300 after a transmission time from time t8 to t9. The encoding time varies depending on the load of the encoding process. The transmission time is also not constant. Note that encoding is performed when compressing the size of the game image (N) or converting the game image (N) into a format suitable for transmission over the communication network 202, for example. However, if the communication speed is high, the transmission time from time t8 to time t9 will be short, so in encoding, these processes may not be performed, or only the format of the game image (N) may be converted.

[0035] The coded image (N) is decoded by the game terminal 300. The time required for decoding is referred to as the decoding time (DC). The decoding time for the game image (N) is expressed as the "decoding time DC(N)." The coded image (N) is decoded after the decoding time DC(N) from time t9 to t10. At time t10 when the decoding is completed, the game terminal 300 completes acquisition of the game image (N). The decoding time also varies depending on the game image.

[0036] The game terminal 300 begins transmitting the game image (N) to the monitor 400 at time t10 when acquisition of the game image (N) is completed. In FIG. 4, the transmission time of the game image (N) is the period shown as times t10 to t11. The game image (N) is displayed on the monitor 400 from time t4, after a monitor delay time MD(N) from time t11 has elapsed. The game image (N) continues to be displayed on the monitor 400 during the display time slot DTS0.

[0037] The server 201 generates the next game image (N+1) in processing time slot PTS1. Game image (N+1) also goes through encoding time EC(N+1), transmission time, decoding time DC(N+1), and monitor delay time MD(N+1), and is then displayed on the monitor 400 at start time t5 of the next display time slot DTS1. The new game image (N+1) continues to be displayed on the monitor 400 during display time slot DTS1.

[0038] In this way, during each display time slot, monitor 400 displays the game images that were ready to be displayed before the start of that display time slot. In other words, when the next game image is ready to be displayed, the next display time slot begins and the next game image is displayed on monitor 400. Gaming terminal 300 prepares new game images at a fixed frequency of R (times / second) and displays the prepared game images on monitor 400.

[0039] As a result, the delay time for the game image (N) is the time from time t0 when generation of the game image (N) begins to time t4 when the monitor 400 displays the game image (N). In other words, there is a delay time (t0 to t4) from the completion of various calculations based on the operation results until the game image (N) is actually displayed. This delay time includes the monitor delay time MD(N), as well as the encoding time EC(N), transmission time, and decoding time DC(N).

[0040] As described above, the monitor delay time can be reduced by increasing the display rate of the monitor 400. In this embodiment, a method is proposed in which the monitor delay time is reduced by increasing the display rate, thereby reducing the delay time and improving the responsiveness of the game.

[0041] However, if the generation rate of the server 201 cannot be increased and only the display rate of the monitor 400 is increased, the generation rate of game images will not be able to keep up with the display rate of the monitor 400. Increasing the display rate of the monitor 400 shortens the interval between display time slots DTS of the monitor 400. That is, the time for the monitor 400 to display one game image will be shortened. For example, in FIG. 4 , assume that the display rate is three times the generation rate (3R). In this case, display time slot DTS0' is shortened to a period from time t4 to time t12, which is one-third the length of display time slot DTS0. Similarly, the next display time slot DTS1' is shortened to a period from time t12 to time t13. In this case, at time t12 when display time slot DTS0' is completed, the monitor 400 will not be able to prepare a new game image (N+1) by the start of the next display time slot DTS1'. As a result, the monitor 400 will not be able to display anything in the next display time slot DTS1'.

[0042] The following describes the game system of this embodiment. It is assumed that the game system of this embodiment has the same configuration as the general remote environment game system shown in FIG.

[0043] FIG. 5 is a functional block diagram of the server 201 according to this embodiment. Each component of the server 201 is implemented by hardware, including computing units such as a CPU, a GPU, and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting these devices, as well as software stored in the storage devices and supplying processing instructions to the computing units. The computer program may be configured by device drivers, an operating system, various application programs located above these, and libraries that provide common functions to these programs. The blocks described below represent functional blocks, rather than hardware configurations. The same applies to the game terminal 300 described below.

[0044] The server 201 includes a data processing unit 210 and a communication unit 220. The data processing unit 210 executes various processes based on operation signals from the game terminal 300. The communication unit 220 is responsible for communication processing with the game terminal 300 via the communication network 202.

[0045] The data processing unit 210 includes a processing command unit 211, a game control unit 212, an image generation unit 213, and an image manipulation unit 214. The processing command unit 211 commands the game control unit 212 to execute processing using a game program based on an operation signal received by the receiving unit 221 from the game terminal 300. The game control unit 212 performs various calculation processes based on the game program in response to commands from the processing command unit 211. The image generation unit 213 generates game images. The image manipulation unit 214 encodes the game images generated by the image generation unit 213. The image manipulation unit 214 functions as an "encoder." The processing command unit 211 commands the transmitting unit 222 to transmit the encoded images generated by the image manipulation unit 214 to the game terminal 300.

[0046] The communication unit 220 includes a receiving unit 221 and a transmitting unit 222. The receiving unit 221 receives an operation signal from the game terminal 300. The transmitting unit 222 transmits an encoded image to the game terminal 300. The transmitting unit 222 functions as an "image transmitting unit."

[0047] FIG. 6 is a functional block diagram of a gaming terminal 300 according to this embodiment. The gaming terminal 300 includes a user interface processing unit 310, a data processing unit 320, a communication unit 330, and a data storage unit 340. The user interface processing unit 310 is responsible for output to the monitor 400 and input from the input device 500. The data processing unit 320 is responsible for various processes related to the progress of the game. The communication unit 330 is responsible for communication processing with the server 201, the monitor 400, and the input device 500 via the communication network 202 and a cable. The communication unit 330 can also directly communicate with the monitor 400 and the input device 500 via short-range wireless communication such as Wi-Fi (registered trademark). The data storage unit 340 receives processing related to game images performed by the data processing unit 320 and temporarily stores the game images.

[0048] The user interface processing unit 310 includes an input unit 311 and an output unit 312. The input unit 311 accepts operational inputs from the player via the input device 500. The output unit 312 transmits digital signals such as control signals via the communication unit 330 as the game progresses, and outputs game images and sounds to the monitor 400.

[0049] The data processing unit 320 includes a processing command unit 321 and an image restoration unit 322. The processing command unit 321 executes commands related to the transmission of operation signals, the decoding of encoded images, the temporary storage of game images, and the output of game images. The processing command unit 321 also acquires information from the communication unit 330, such as the resolution and display rate at which the monitor 400 can display, and the audio format that can be output. Based on the information acquired from the communication unit 330, the processing command unit 321 determines the resolution, display rate, audio format, etc. of the game images to be output to the monitor 400. The image restoration unit 322 decodes the encoded images. The image restoration unit 322 functions as a "decoder."

[0050] The communication unit 330 includes a receiving unit 331 and a transmitting unit 332. The receiving unit 331 receives encoded images from the server 201. The receiving unit 331 also receives operation signals from the input device 500. The receiving unit 331 also receives information from the monitor 400, such as the resolution and display rate that the monitor 400 can display and the audio format that it can output. The receiving unit 331 functions as an "image receiving unit." The transmitting unit 332 transmits operation signals to the server 201 and transmits game images to the monitor 400. The transmitting unit 332 functions as an "operation transmitting unit" and an "image transmitting unit."

[0051] Here, a part of the processing flow in the game terminal 300 will be described. First, an operation input by the player is accepted by the input unit 311. The processing command unit 321 commands the transmission unit 332 to transmit an operation signal indicating the operation content to the server 201. The transmission unit 332 transmits the operation signal to the server 201 in accordance with the command from the processing command unit 321.

[0052] The server 201 transmits an encoded image to the game terminal 300. The receiving unit 331 of the game terminal 300 receives the encoded image. When the processing command unit 321 of the game terminal 300 receives the encoded image, it commands the image restoration unit 322 to decode the encoded image. Upon receiving the command from the processing command unit 321, the image restoration unit 322 decodes the encoded image. The processing command unit 321 commands the output unit 312 to generate a control signal from the decoded game image. In this embodiment, the control signal is a collective term for data indicating coloring specifications for each pixel of the monitor 400 and a signal indicating the timing of image display (pixel illumination). The processing command unit 321 also stores the decoded game image in the data storage unit 340. Upon receiving the command, the output unit 312 generates a control signal from the decoded game image and transfers it to the transmitting unit 332. The transmitting unit 332 transmits the control signal transferred from the output unit 312 to the monitor 400.

[0053] 7 is a functional block diagram of the monitor 400 in this embodiment. The monitor 400 includes a data processing unit 410 and a communication unit 420. The data processing unit 410 is responsible for processing to display game images on the screen. The communication unit 420 is responsible for processing communications with the game terminal 300.

[0054] The data processing unit 410 includes a display control unit 411. When the receiving unit 421 receives the control signal, the display control unit 411 completes display preparation and then displays a game image on the monitor 400.

[0055] The communication unit 420 includes a receiving unit 421. The receiving unit 421 receives a control signal obtained by converting a game image from the game terminal 300 via a cable or the like.

[0056] 8 is a time chart showing the process of transferring a game image in this embodiment. In the process shown in Fig. 8, a game image generated by the image generation unit 213 in the server 201 is converted into an encoded image by the image processing unit 214. Next, the encoded image is transmitted to the game terminal 300 and decoded by the image restoration unit 322. The decoded game image is then transmitted to the monitor 400 and displayed.

[0057] In FIG. 8 , in order to reduce delay time, the generation rate of the image generation unit 213 is set to R (times / second), while the display rate of the monitor 400 is set to 4R (four times R) (times / second). In other words, four display time slots are provided for one processing time slot. For example, when the generation rate is R = 60 Hz, the display rate is 4R = 240 Hz. In this case, the GPU of the server 201 generates a new game image approximately every 17 milliseconds (= 1 (second) ÷ 60 (Hz)). Meanwhile, the monitor 400 switches and displays game images approximately every 4 milliseconds (= 1 (second) ÷ 240 (Hz)).

[0058] First, the image generation unit 213 of the server 201 generates a game image (N) during a rendering time RT(N) from time t0 to t7 in processing time slot PTS0. Here, when generation of the game image (N) is completed at time t7, the image processing unit 214 encodes the game image (N) without waiting for the completion of the wait time WT(N). The game image (N) is converted into an encoded image (N) after an encoding time EC(N) from time t7 to t8. The encoded image (N) is transmitted to the game terminal 300 after a transmission time from time t8 to t9.

[0059] The coded image (N) is decoded by the image restoration unit 322 of the game terminal 300. The coded image (N) is decoded after a decoding time DC(N) from time t9 to time t10. At the decoding completion time t10, the processing command unit 321 acquires the game image (N).

[0060] The game image (N) is transmitted to the monitor 400 from time t10 to time t11. The processing command unit 321 also temporarily stores the game image (N) acquired at time t10 in the data storage unit 340.

[0061] The game image (N) is displayed on the monitor 400 in the display time slot DTS0 starting at time t12 after a monitor delay time MD(N) from time t11 to time t12. The game image (N) is displayed on the monitor 400 during the display time slot DTS0.

[0062] 8 is shorter than that in FIG. 4 due to the increased display rate of the monitor 400. Therefore, the delay time of the game image (N) is shorter than that in FIG.

[0063] 8, the generation rate of the image generation unit 213 is R (times / second), while the display rate of the monitor 400 is set to 4R (four times R) (times / second). From the time the image generation unit 213 starts generating a game image (N) until it generates a new game image (N+1), the monitor 400 needs to display four game images (N). In this embodiment, the monitor 400 displays the game image (N) every time in four display time slots DTS0, DTS1, DTS2, and DTS3.

[0064] Hereinafter, a game image (N) displayed in the nth display time slot after the start of display of the game image (N) will be referred to as game image (N:n). Furthermore, the expression "the processing command unit 321 commands the output unit 312 to display a game image" refers to the process of "the processing command unit 321 commands the output unit 312 to generate a control signal from the game image, the output unit 312 generates the control signal from the game image, and the transmission unit 332 transmits the control signal to the monitor 400."

[0065] 8 , the image restoration unit 322 has not completed decoding of the encoded image (N+1) by time t13, when the display time slot DTS0 is completed. In other words, the processing command unit 321 has not acquired a new game image (N+1). Therefore, the processing command unit 321 commands the output unit 312 to display the game image (N) again shortly after time t12. In other words, the processing command unit 321 commands the output unit 312 to display the game image (N:2). Specifically, the processing command unit 321 reads the game image (N) stored in the data storage unit 340 and commands the output unit 312 to re-display the game image (N) as the game image (N:2).

[0066] The receiving unit 421 of the monitor 400 receives a control signal corresponding to the game image (N:2) transmitted by the gaming terminal 300. The display control unit 411 prepares to display the game image (N:2) based on this control signal. Once the display control unit 411 has completed preparations for displaying the game image (N:2), it displays the game image (N:2) on the monitor 400 during the display time slot DTS1 starting at time t13. Note that the monitor delay time MD(N:2) is completed by time t13. In other words, the game image (N:1) is displayed in the display time slot DTS0, and the same game image (N:2) is also displayed in the next display time slot DTS1.

[0067] Even at time t14 when display time slot DTS2 starts, the image restoration unit 322 has not yet completed decoding of the coded image (N+1). Therefore, the processing command unit 321 reads out the game image (N) again from the data storage unit 340 a little after time t13. The processing command unit 321 commands the output unit 312 to display the game image (N:3).

[0068] The receiving unit 421 of the monitor 400 receives the control signal corresponding to the game image (N:3) transmitted by the gaming terminal 300. The display control unit 411 prepares to display the game image (N:3). At time t14, after the monitor delay time MD(N:3) has elapsed, preparations for displaying the game image (N:3) are completed, and the game image (N:3) is displayed on the monitor 400 during the display time slot DTS2. In other words, the game image (N) is displayed three times at this stage. To the player, the game image (N) appears to be displayed continuously.

[0069] Even at time t15 when display time slot DTS3 starts, the image restoration unit 322 has not yet completed decoding of the encoded image (N+1). The processing command unit 321 reads out the game image (N) again from the data storage unit 340 a little after time t14 in case the next game image (N+1) cannot be prepared for display by time t15. The processing command unit 321 commands the output unit 312 to display the game image (N:4).

[0070] The receiving unit 421 of the monitor 400 receives the control signal corresponding to the game image (N:4) transmitted by the gaming terminal 300. The display control unit 411 prepares to display the game image (N:4). At time t15, after the monitor delay time MD(N:4) has elapsed, preparations for displaying the game image (N:4) are completed, and the game image (N:4) is displayed on the monitor 400 during the display time slot DTS3.

[0071] 8, the game terminal 300 receives the coded image (N+1) during display time slot DTS1 (times t13 to t14). At time t16, during display time slot DTS3 (times t15 to t17), the image restoration unit 322 completes decoding of the coded image (N+1). That is, the game image (N+1) will not be ready in time to be displayed in display time slot DTS3, but it will be ready in time for display in the next display time slot DTS4. After the start of display time slot DTS3 for game image (N:4), the processing command unit 321 commands the output unit 312 to display the new game image (N+1).

[0072] The receiving unit 421 of the monitor 400 receives the control signal corresponding to the new game image (N+1) transmitted by the gaming terminal 300. The display control unit 411 prepares to display the game image (N+1:1) based on this control signal. At time t17, after the monitor delay time MD(N+1:1) has elapsed, preparations for displaying the game image (N+1:1) are completed, and the game image (N+1:1) is displayed on the monitor 400 during display time slot DTS4. That is, the game image (N) is displayed four times at this stage, and then the game image (N+1) is displayed.

[0073] As described above, the display control unit 411 of the monitor 400 sequentially displays game images that have been prepared for display on the monitor 400 at the start of each display time slot.

[0074] Basically, the processing command unit 321 of the gaming terminal 300 commands the output unit 312 to read out game images stored in the data storage unit 340 a specific number of times and display them in each display time slot, depending on the ratio between the generation rate and the display rate. In the case of FIG. 8 , the ratio between the generation rate and the display rate is 1:4. In other words, four display time slots occur during one processing time slot. That is, one game image is displayed four times. Therefore, the processing command unit 321 commands the output unit 312 to read out game images stored in the data storage unit 340 four times and display them in each display time slot.

[0075] However, as described above, the encoding time, transmission time, and decoding time are not constant. Therefore, even if four display timeslots are completed, it is conceivable that the new game image (N+1) will not be ready in time for display. In preparation for the case where the game image (N+1:1) will not be ready in time for the start of the next display timeslot DTS4, the processing command unit 321 may instruct the output unit 312 to display an additional game image (N:5). In the case of FIG. 8 , the processing command unit 321 acquires the game image (N) from the data storage unit 340 in advance, slightly after time t15. The processing command unit 321 may instruct the output unit 312 to display the game image (N:5) in the display timeslot DTS4. According to this control method, even if the game image (N+1) is not ready to be displayed in time for the next new game image (N+1), the continuous display of game images can be prevented from being interrupted by displaying the additional game image (N) in the next display timeslot.

[0076] Alternatively, it is also possible that the image restoration unit 322 completes the decode time DC(N+1) early. For example, suppose that the decode time DC(N+1) is completed during the display time slot DTS1. That is, suppose that the processing command unit 321 acquires a new game image (N+1) before the completion time t14 of the display time slot DTS1. In this case, suppose that the display control unit 411 has not yet started preparations for displaying the game image (N:4) at the time that the decode time DC(N+1) is completed. In such a case, the processing command unit 321 may instruct the monitor 400 to display the game image (N+1:1) instead of the game image (N:4). As a result, in the fourth display time slot DTS3, the game image (N+1:1) is displayed on the monitor 400 instead of the game image (N:4). According to this control method, when a situation arises in which the display of a game image (N) is scheduled but preparations for displaying a new game image (N+1) can be started earlier than planned, the game image (N) scheduled to be displayed in the next display time slot can be changed to the new game image (N+1), thereby allowing the new game image (N+1) to be displayed earlier.

[0077] The game system according to this embodiment has been described above. This embodiment assumes a situation in which, in a remote game system, the monitor's display rate is faster than the server's generation rate. In other words, the server's speed at which it generates game images cannot keep up with the monitor's display speed. In this case, the server may not be able to provide the game images or the monitor may not be ready to display them in time for the next game image to be displayed on the monitor, potentially resulting in no game image being displayed on the monitor. To resolve this issue, this embodiment transfers a single game image from the game terminal to the monitor multiple times so that a game image can be displayed in each display time slot. Therefore, the monitor is always ready to display the next game image at the start of each display time slot. This allows the monitor to continuously display game images without interruption.

[0078] Furthermore, in the game system of this embodiment, delay time can be alleviated by introducing a monitor with a high-speed display rate. In a game system in a remote environment, delay time mainly consists of encoder time, transmission time, decoder time, and monitor delay time. Of these, the encoder time, transmission time, and decoder time are not constant. Furthermore, none of these times can be directly adjusted by the player. However, monitor delay time can be shortened by increasing the display rate of the monitor 400. This can be easily achieved if the player introduces a monitor that can increase the display rate. From the above, it is expected that the game system of this embodiment will allow players to play games without feeling the stress of display delays.

[0079] The present invention is not limited to the above-described embodiments and modifications, and the components can be modified without departing from the spirit of the invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments and modifications. Furthermore, some components can be omitted from all the components shown in the above-described embodiments and modifications.

[0080] In the present embodiment, the remote game system 200 has been described as having a function for adjusting game image display within the game terminal 300. As a modification, this image display adjustment function may be implemented as hardware or software.

[0081] When implemented as hardware, the display adjustment device may be constructed as a display adjustment device. This display adjustment device may be mounted on game terminal 300 and server 201. Alternatively, the display adjustment device may be constructed as an independent device such as a relay device. The relay device may be connected between server 201 and communication network 202. The relay device may be connected between communication network 202 and game terminal 300.

[0082] When implemented as software, the display adjustment program may be constructed as a display adjustment program, which may be installed in game terminal 300, monitor 400, and the relay device described above.

[0083] In this embodiment, the game terminal 300 of the remote game system 200 performs the display adjustment. As a variation, the configuration of the game system does not need to depend on the remote environment. For example, in the local game system 100, the personal computer 101 may be equipped with a display adjustment program. The game program executes calculation processing on the personal computer 101. The GPU generates a game image corresponding to the processing result. The display adjustment program performs display adjustment of the game image according to this embodiment. The monitor 102 may display the game image adjusted by the display adjustment program. In this way, even if the display rate of the monitor 102 is higher than the generation rate of the game execution entity (game program) on the personal computer 101, the difference between the generation rate and the display rate may be adjusted by the display adjustment device or display adjustment program on the personal computer 101.

[0084] [Monitor Adjustment Method] In this embodiment, a method has been described in which the difference between the generation rate of the image generation unit 213 and the display rate of the monitor 400 is adjusted in the game terminal 300. As a variant, the monitor 400 may adjust this difference. In this case, the monitor 400 has a data storage unit for storing game images. Also, the processing command unit 321 of the game terminal 300 commands the monitor 400 to display the game image.

[0085] 8, it is assumed that the monitor 400 displays a game image (N:1) in display time slot DTS0. At this time, the monitor 400 may store the game image (N) in the data storage unit.

[0086] Assume that the processing command unit 321 is unable to command the monitor 400 to display a new game image (N+1) in time for the start of the next display time slot DTS1. In this case, the monitor 400 may display the game image (N) stored in the data storage unit as the game image (N:2) during the display time slot DTS1.

[0087] Suppose the processing command unit 321 commands the monitor 400 to display a new game image (N+1) in time for the start of the next display time slot DTS1. In this case, the monitor 400 may display the game image (N+1:1) in the display time slot DTS1. At this time, the monitor 400 overwrites and saves the new game image (N+1) in the data storage unit. Then, if the processing command unit 321 is unable to command the monitor 400 to display the new game image (N+2) in time for the start of the display time slot DTS2, the monitor 400 may display the game image (N+1) in the data storage unit 340 as the image (N+1:2). In other words, in the monitor adjustment method, the monitor 400 may repeatedly display the game image stored in the data storage unit until the monitor 400 is ready to display the new game image.

[0088] [Server Adjustment Method] In this embodiment, a method has been described in which the difference between the generation rate of the image generation unit 213 and the display rate of the monitor 400 is adjusted in the game terminal 300. As a modified example, the server 201 may adjust this difference. The image generation unit 213 of the server 201 generates game images at a generation rate of R (times / second). On the other hand, if the display rate of the monitor 400 is 4R (times / second), the server 201 may transmit the same game image four times at time intervals of 4R. In other words, the server 201 may adjust the transmission timing so that four game images are transmitted in the time it takes to generate one game image.

[0089] For example, the image generation unit 213 of the server 201 generates game image (NA), game image (NB), game image (NC), and game image (ND), which have the same drawing content as game image (N). The server 201 transmits these four images to the game terminal 300. In this case, four game images are transmitted sequentially per processing time slot. By the processing command unit 321 transmitting game image (NA), game image (NB), game image (NC), and game image (ND) sequentially to the monitor 400, the monitor 400 can display game images at a display rate of 4R (times / second).

[0090] In the server adjustment method, the image generation unit 213 may duplicate one game image (NA) to duplicate game image (NB), game image (NC), and game image (ND). Alternatively, the image generation unit 213 may independently generate four game images (NA) to (ND). For example, assume that the server 201 contains four independent GPUs, GPU (A), GPU (B), GPU (C), and GPU (D), as the image generation unit 213. In this case, each GPU may independently generate game image (N) within one processing time slot. The game images independently generated by the four GPUs may be identical, but do not need to be completely identical.

[0091] As a variation of this embodiment, the processing command unit 321 of the game terminal 300 may generate the next game image to be transmitted to the monitor 400 (hereinafter referred to as a "predicted image") from game images acquired in the past and the game image currently being acquired.

[0092] For example, the processing command unit 321 acquires a game image (N). At this time, the processing command unit 321 generates a predicted image (N+1) to be transmitted next to the monitor 400 from a past game image (N-1) stored in the data storage unit 340 and the currently acquired game image (N). A method for predicting future images from consecutive past images can be achieved by applying known technology. The processing command unit 321 commands the output unit 312 to display the predicted image (N+1). After transmitting the game image (N) to the monitor 400, the processing command unit 321 may repeatedly command the output unit 312 to display the predicted image (N+1). Assume that time has passed and the processing command unit 321 has acquired an actual new game image (N+1). In this case, the processing command unit 321 may command the output unit 312 to display the new game image (N+1). Note that this prediction function may be implemented by the monitor in a monitor adjustment system. After the new game image (N+1), the next new game image (N+2) may be displayed. If the next new game image (N+2) cannot be prepared for display in time, a predicted image (N+2) generated from the game image (N) and the new game image (N+1) may be displayed.

[0093] In the present embodiment, the monitor 400 is assumed to continue displaying the same game image throughout each display time slot DTS. As a variation, the monitor 400 may start displaying the next game image midway through the display time slot DTS.

[0094] FIG. 9 is a time chart showing an enlarged view of the transfer process of the image restoration unit 322, the display control unit 411, and the monitor 400 in FIG. 8 in a modified example. The modified example shown in FIG. 9 illustrates the process in which a new game image (N+1) is displayed on the monitor 400 midway through display time slot DTS3. For ease of explanation, illustration of monitor delay times MD(N:2), MD(N:3), and MD(N:4) is omitted. In FIG. 9 , it is assumed that the decoding time DC(N+1) is completed at time t19 during display time slot DTS2. At this time, the processing command unit 321 immediately commands the output unit 312 to display the new game image (N+1). After the monitor delay time MD(N+1:1), the monitor 400 becomes ready to display the game image (N+1:1) at time t20 midway through display time slot DTS3. The monitor 400 may display the game image (N+1:1) from time t20, prior to the start time t17 of the next display time slot DTS4, without waiting for the start time t17. In this case, the game image (N:4) is displayed on the monitor 400 from time t15 to t20, which is the first half of the display time slot DTS3. The game image (N+1:1) is displayed from time t20 to t17, which is the remaining time of the display time slot DTS3. Furthermore, in the next display time slot DTS4 (times t17 to t18), the monitor 400 continues to display the game image (N+1:2). Thus, in FIG. 9 , the game image is displayed on the monitor 400 as soon as it is ready to be displayed, without waiting for the start of the next display time slot. In other words, the delay time can be further reduced compared to when the game image display adjustment according to this embodiment is implemented.

[0095] This modified example may be applied to video distribution services, etc. For example, it may be used in the distribution of video of live sports broadcasts. In sports, the thrill lies in witnessing decisive moments that determine the outcome of a match, such as when a point is scored or a technique is executed, in real time. In order to display images capturing such moments on the monitor 400 more immediately, it is better to be able to display images regardless of the interval of the display time slot.

[0096] In this modified example, the case where the generation rate is set to R (times / second) and the display rate is set to 4R (four times R) (times / second) has been described. The ratio between the generation rate and the display rate does not have to be limited to 1:4. For example, the ratio between the generation rate and the display rate may be set to 1:8. Furthermore, the ratio between the generation rate and the display rate does not have to be limited to an integer ratio. For example, the ratio between the generation rate and the display rate may be set to 1:3.5. This modified example can reduce the delay time and display game images on the monitor 400, regardless of the ratio between the generation rate and the display rate.

[0097] Generally, for professional players who play action-packed cloud games, the acceptable range for latency is within 100 milliseconds. It is also said that even average players experience stress when latency reaches 200 milliseconds. For example, assume that a monitor 400 with a display rate of 60 Hz is installed in the remote game system 200. In this case, the monitor latency is said to be 8 milliseconds. 8 milliseconds accounts for approximately 8% of the latency tolerance of the aforementioned professional player. Therefore, for this player, even a slight reduction in the monitor latency can significantly affect the comfort of playing the game. The present invention can reduce latency by increasing the display rate of the monitor 400 and continuously transmitting game images to the monitor 400.

Claims

1. Equipped with a game server, a game terminal, and a display device, the game server and the game terminal are connected via a communication network; The game server a game control unit that controls the progress of the game in accordance with operation signals from the game terminal; an image generation unit that generates a game image in a first cycle; an image transmission unit that transmits the game image to the game terminal; The display device includes: a display control unit that displays a game image at a second cycle that is shorter than the first cycle, The game terminal includes: an input unit that accepts operations from a player with respect to the game; an operation transmitting unit that transmits an operation signal indicating the content of an operation from a player to the game server; an image receiving unit that receives game images from the game server; a cycle acquisition unit that acquires the second cycle from the display device; an image transmission unit that transmits the game image generated in the first cycle to the display device in synchronization with the acquired second cycle, The image transmission unit of the game terminal When a new game image is acquired from the game server before the timing for the display device to display the next game image arrives, the new game image is transmitted to the display device; When a new game image has not been acquired from the game server when the timing for the display device to display the next game image arrives, a substitute game image different from the new game image is transmitted to the display device.

2. The game system described in claim 1, wherein when a new game image is not acquired from the game server before the timing for the display device to display the next game image arrives, the image transmission unit of the game terminal retransmits to the display device a game image generated based on an acquired game image or an acquired game image as the substitute game image.

3. 2. The game system according to claim 1, wherein the image transmission unit of the game terminal transmits a game image predicted from one or more acquired game images to the display device as the alternative game image when a new game image is not acquired from the game server before the timing for the display device to display the next game image arrives.

4. 2. The game system according to claim 1, wherein the image transmission unit of the game terminal transmits the generated game image to the display device multiple times, and determines the number of transmissions based on the ratio between the first period and the second period.

5. 5. The game system described in claim 4, wherein the image transmission unit of the game terminal retransmits the generated game image to the display device when no new game image is obtained from the game server even after transmitting the generated game image to the display device a number of times determined based on the ratio between the first period and the second period.

6. 5. The game system according to claim 4, wherein when a new game image is obtained from the game server before the generated game image has been transmitted to the display device a number of times determined based on a ratio between the first period and the second period, the image transmission unit of the game terminal transmits the new game image to the display device without transmitting the generated game image a number of times determined.

7. 2. The game system according to claim 1, wherein when a new second game image is acquired from the game server while a first game image is being displayed and before the timing for displaying the next game image following the first game image arrives, the display control unit of the display device switches from the first game image to the second game image and displays it, even before the timing for displaying the next game image arrives.

8. the image transmission unit of the game server compresses the game image and transmits it to the game terminal; the image receiving unit of the game terminal decompresses the compressed game image received from the game server, The image transmission unit of the game terminal transmitting the restored game image to the display device and storing the restored game image; 2. The game system of claim 1, wherein when a new game image is acquired from the game server before the timing for the display device to display the next game image arrives and restoration of the new game image is not complete, the restored and stored acquired game image is transmitted to the display device.

9. It is connected to the game server via a communication network. an input unit that accepts operations from a player regarding the game; an operation transmitting unit that transmits an operation signal indicating the content of an operation from a player to the game server; an image receiving unit that receives, from the game server, a game image generated at a first cycle based on the operation signal; a cycle acquisition unit that acquires, from the display device, a second cycle that is shorter than the first cycle as a cycle at which the display device displays a game image; an image transmission unit that transmits the game image generated in the first cycle to the display device in synchronization with the acquired second cycle, The image transmission unit When a new game image is acquired from the game server before the timing for the display device to display the next game image arrives, the new game image is transmitted to the display device; When a new game image is not acquired from the game server before the timing for the display device to display the next game image arrives, the game terminal transmits a substitute game image different from the new game image to the display device.

10. an image receiving unit that receives game images generated at a first cycle from a game executing entity; a cycle acquisition unit that acquires, from the display device, a second cycle that is shorter than the first cycle as a cycle at which the display device displays a game image; an image transmission unit that transmits the game image generated in the first cycle to the display device in synchronization with the acquired second cycle, The image transmission unit when a new game image is acquired from the game executing entity before the timing for the display device to display the next game image arrives, transmitting the new game image to the display device; A display adjustment device that transmits a substitute game image different from the new game image to the display device when a new game image is not acquired from the game executing entity before the timing for the display device to display the next game image arrives.

11. In a computer connected to a game server via a communication network, a function of detecting an operation from a player and transmitting an operation signal indicating the operation content to the game server; a function of receiving, from the game server, a game image generated at a first cycle based on the operation signal; a function of acquiring, from the display device, a second period that is shorter than the first period as a period for displaying a game image by the display device; a function of transmitting the game image generated in the first cycle to the display device in synchronization with the acquired second cycle; When a new game image is acquired from the game server before the timing for the display device to display the next game image arrives, the new game image is transmitted to the display device; A computer program that transmits a substitute game image different from the new game image to the display device when a new game image is not acquired from the game server before the timing for the display device to display the next game image arrives.