Program, client terminal, game system, and processing method

Abstract

In the present invention, a client terminal has an estimation model for complementing a lost stream image. The estimation model estimates optical flow and depth information from flow images I (0) to I (t) displayed prior to a lost flow image I (t + 1), and generates a flow image I '(t + 1) based on the estimation result. The client terminal displays the generated stream image I '(t + 1) as an alternative to the lost stream image I (t + 1).

Description

Technical Field

[0001] This invention relates to a program, a client terminal, a method for processing the client terminal, a game system, and a method for processing the game system. Background Technology

[0002] Patent document 1 discloses a technology related to cloud gaming. In cloud gaming, the game program is executed by a server in the cloud. That is, the client terminal sends an input event based on user input to the server. The server executes the game program based on the input event, thereby advancing the game. After generating an image of the advanced game, the server sends the image to the client terminal in stream form. The client terminal displays the received image on a monitor or similar device. With the recent improvement in the parallel performance of GPUs (Graphics Processing Units), the advancement of GPU virtualization technology, and the emergence of broadband networks represented by 5G, cloud gaming has attracted attention.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-037626 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] In cloud gaming streaming, there is a problem of image loss of several frames due to network latency and other factors. This poses a significant obstacle to maintaining natural game visuals. Existing technologies address this issue by sending speculatively rendered images from the server to the client in advance, which places a heavy load on the network and server. Patent Document 1 does not disclose a solution to this problem.

[0008] The problem of this invention is to solve the problem of not being able to maintain a natural picture due to the loss of several frames in the image stream.

[0009] Solution for solving the problem

[0010] According to one aspect of the present invention, a program is provided that enables a computer in a client terminal to function as a unit including:

[0011] The reception department handles user input related to the progress of the game in progress.

[0012] The sending unit sends input information representing the content input by the user to the server;

[0013] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0014] The detection unit detects the phenomenon of lost streaming images;

[0015] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0016] The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

[0017] In addition, according to another aspect of the present invention, a client terminal is provided, comprising:

[0018] The reception department handles user input related to the progress of the game in progress.

[0019] The sending unit sends input information representing the content input by the user to the server;

[0020] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0021] The detection unit detects the phenomenon of lost streaming images;

[0022] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0023] The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

[0024] In addition, according to another aspect of the present invention, a processing method is provided, in which the following processing is performed by a client terminal:

[0025] Accept user input related to the progress of the game being played;

[0026] Send input information representing the content input by the user to the server;

[0027] Receive streaming images from the server, wherein the streaming images represent the progress of the game being executed in the server based on the input information;

[0028] Detect the phenomenon of missing streaming images;

[0029] Based on an estimation model and multiple stream images displayed before the missing stream image, a completed stream image representing an estimation result of the content of the missing stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently executing game, and generate a stream image representing the estimated content; and

[0030] The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image.

[0031] Furthermore, according to another aspect of the present invention, a game system is provided, comprising a client terminal and a server, wherein...

[0032] The client terminal has:

[0033] The reception department handles user input related to the progress of the game in progress.

[0034] The sending unit sends input information representing the content input by the user to the server;

[0035] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0036] The detection unit detects the phenomenon of lost streaming images;

[0037] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0038] The display control unit displays the streaming image on the display device and displays the completed streaming image as a replacement for the missing streaming image on the display device.

[0039] The server has:

[0040] An input information receiving unit receives the input information from the client terminal;

[0041] A game progression unit, based on the input information, performs processing to advance the currently running game and generates the stream image representing the result of the game's progress; and

[0042] An image transmitting unit that sends the streaming image to the client terminal.

[0043] Furthermore, according to another aspect of the present invention, a processing method is provided, in which the following processing is performed by the client terminal of a game system having a client terminal and a server:

[0044] Accept user input related to the progress of the game being played;

[0045] Send input information representing the content input by the user to the server;

[0046] Receive streaming images from the server, wherein the streaming images represent the progress of the game being executed in the server based on the input information;

[0047] Detect the phenomenon of missing streaming images;

[0048] Based on an estimation model and multiple stream images displayed before the missing stream image, a completed stream image representing an estimation result of the content of the missing stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently executing game, and generate a stream image representing the estimated content; and

[0049] The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image.

[0050] In the processing method, the server performs the following processing:

[0051] Receive the input information from the client terminal;

[0052] Based on the input information, perform processing to advance the currently running game, and generate the stream image representing the result of the progress of the currently running game; and

[0053] The streaming image is sent to the client terminal.

[0054] Invention Effects

[0055] According to one aspect of the present invention, the problem of being unable to maintain a natural picture due to the loss of several frames in the image stream can be solved. Attached Figure Description

[0056] The above-described objectives, as well as other objectives, features, and advantages, become more apparent from the preferred embodiments described below and the accompanying drawings.

[0057] Figure 1 This is an example diagram illustrating the functional block diagram of a game system.

[0058] Figure 2 This is a diagram illustrating an example of the hardware structure of a client terminal and a server.

[0059] Figure 3 This is an example of a functional block diagram showing a client terminal and a server.

[0060] Figure 4 This is a diagram used to illustrate the completion process performed by the client terminal.

[0061] Figure 5 This is a flowchart illustrating an example of the processing flow of a client terminal.

[0062] Figure 6 This is a diagram used to illustrate an embodiment of the game system. Detailed Implementation

[0063] The embodiments of the present invention will now be described using the accompanying drawings. Furthermore, throughout the drawings, the same reference numerals are used to denote the same constituent elements, and descriptions are omitted where appropriate.

[0064] "summary"

[0065] In the game system of this embodiment, the client terminal is equipped with a Neural Network (NN) model for a generation system used to complete images. Furthermore, the game system uses the NN model of the generation system on the client terminal side to complete images lost in the cloud game stream. Specifically, the client terminal caches several frames of images received from the server in a streaming manner. The client terminal uses the NN model to infer optical flow and depth information from these image frames, and uses the inference results to generate an image for completing the lost images.

[0066] Therefore, even if several frames are lost in the image stream due to network latency, the lost images can be filled in by the NN model on the client side to maintain a natural game screen.

[0067] The technical feature of the game system in this embodiment is that the client terminal does not possess information inherent in the content, such as 3D models or game logic. Instead, it uses content-independent logic, such as images obtained from the server, to complete the images lost in the stream. The advantage of this embodiment is that it does not require significant changes to the existing cloud gaming architecture or communication methods; simply by having a neural network (NN) model on the cloud gaming client terminal, it can achieve the completion of lost images.

[0068] "Overall structure"

[0069] like Figure 1 As shown, the game system 1 has a client terminal 10 and a server 20. The client terminal 10 and the server 20 are connected to each other via a communication network 30 in a manner that enables them to communicate with each other. The client terminal 10 is, for example, a smartphone, tablet terminal, personal computer, game console, mobile phone, smartwatch, etc., but is not limited to these. The server 20 is, for example, a cloud server, but is not limited to this.

[0070] Hardware structure

[0071] First, an example of the hardware structure of the client terminal 10 will be described. The various functional units of the client terminal 10 are implemented through any combination of hardware and software, centered around any computer's CPU (Central Processing Unit), memory, programs loaded into memory, storage units such as hard disks storing the programs (which can store programs pre-saved at the factory, as well as programs downloaded from recording media such as CDs (Compact Discs), servers on the Internet, etc.), and a network connection interface. Furthermore, those skilled in the art will understand that there are various modifications to its implementation methods and devices.

[0072] Figure 2 This is a block diagram illustrating the hardware structure of client terminal 10. For example... Figure 2 As shown, the client terminal 10 includes a processor 1A, a memory 2A, an input / output interface 3A, peripheral circuitry 4A, and a bus 5A. The peripheral circuitry 4A includes various modules. The processing device may also not have peripheral circuitry 4A. Furthermore, the client terminal 10 may also be composed of multiple physically and / or logically separate devices. In this case, each of the multiple devices can possess the aforementioned hardware structure.

[0073] Bus 5A is a data transmission path used for the processor 1A, memory 2A, peripheral circuits 4A, and input / output interfaces 3A to send and receive data to each other. Processor 1A is, for example, a CPU, GPU, or other computing device. Memory 2A is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). Input / output interfaces 3A include interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. Input devices include, for example, a keyboard, mouse, microphone, physical buttons, touch panel, etc. Output devices include, for example, a monitor, speaker, printer, email sender, etc. Processor 1A can issue instructions to each module and perform calculations based on the results of each module's calculations.

[0074] Next, an example of the hardware structure of server 20 will be described. Each functional unit of server 20 is implemented through any combination of hardware and software, centered around an arbitrary computer CPU, memory, a program loaded into memory, a storage unit such as a hard disk storing the program (which can store programs pre-saved at the time of device manufacture, as well as programs downloaded from recording media such as CDs, servers on the Internet, etc.), and a network connection interface. Furthermore, those skilled in the art will understand that various modifications of its implementation method and apparatus are possible.

[0075] Figure 2 This is a block diagram illustrating the hardware structure of server 20. For example... Figure 2 As shown, server 20 has a processor 1A, a memory 2A, an input / output interface 3A, peripheral circuitry 4A, and a bus 5A. The functions of these components are as described above.

[0076] Client terminal 10 and server 20 achieve their respective functions by installing programs appropriate to their respective purposes or by using components with performance appropriate to their respective purposes.

[0077] Functional Structure

[0078] Next, the functional structure of the game system 1 in this embodiment will be described in detail.

[0079] <Client Terminal 10>

[0080] Figure 3 An example of a functional block diagram of a client terminal 10 is shown. As shown, the client terminal 10 includes an acquisition unit 11, a storage unit 12, a receiving unit 13, a sending unit 14, a receiving unit 15, a detection unit 16, a generation unit 17, and a display control unit 18.

[0081] The acquisition unit 11 acquires the estimated model for completing the missing stream image and stores it in the storage unit 12.

[0082] "Streaming images" are images sent in a streaming manner from server 20 to client terminal 10 in cloud gaming. In cloud gaming, images representing the content of the game being played (so-called "game screen") are sent and received in the form of streaming images.

[0083] "Stream image loss" refers to the client terminal 10 failing to receive the stream image before a specified time. In this embodiment, if the next stream image is not received (failed to be received) within a specified time from a reference time, it is considered that the next stream image is lost. The reference time is the time when the previous stream image was received or a time based on it. The specified time is a pre-defined value.

[0084] "Completion" refers to generating missing stream images from multiple stream images in a time series based on other stream images.

[0085] The “estimation model” is used to estimate the content of a stream image that will be displayed after multiple stream images in a time series, and to generate a stream image that represents the estimated content (called a “complete stream image”).

[0086] The estimation model is based on optical flow and depth information estimated from multiple stream images. It uses neural network (NN) models from generative systems such as GAN (Generative Adversarial Network), VAE (Variational Auto-Encoder), NeRF (Neural Radiance Fields), and IFNet to generate completed stream images. There are no particular restrictions on the methods used to estimate optical flow and depth information from multiple stream images; all techniques can be employed. Well-known frame completion models such as DAIN and RIFE can be used as estimation models.

[0087] Furthermore, an estimation model is prepared for each game. Each estimation model is learned using images from each game. By preparing an estimation model for each game, the completed stream image can be generated with higher accuracy.

[0088] The acquisition unit 11 acquires any one of the multiple estimation models prepared corresponding to the multiple games from the providing server, specifically the estimation model corresponding to the game selected by the user, and stores the estimation model in the storage unit 12. That is, the storage unit 12 of each client terminal 10 stores the estimation model of the game selected by the user of each client terminal 10. The providing server is set up for each game, and can be a different server from the server 20 that manages the game progress of each game, or it can be the same server as server 20.

[0089] For example, when a desired game is selected from a specified website or application and the selected game's application is downloaded and installed on the client terminal 10, the estimation model corresponding to the selected game may also be downloaded and installed on the client terminal 10. That is, the acquisition unit 11 may also acquire the estimation model at such a time and store it in the storage unit 12.

[0090] Alternatively, the estimation model can be downloaded and installed on the client terminal 10 each time the application is downloaded to the client terminal 10 and the installed game is executed, at the start of execution (e.g., when the application is launched, when a specified operation is received after the application is launched, etc.). That is, the acquisition unit 11 can also acquire the estimation model at such times and store it in the storage unit 12.

[0091] Furthermore, the timing of obtaining the estimated model illustrated here is just one example and is not limited to these.

[0092] The receiving unit 13 accepts user input related to the progress of the game being played. There are no particular restrictions on the type of game. The receiving unit 13 accepts user input via all input devices such as touch panels, physical buttons, keyboards, mice, microphones, and gesture input devices.

[0093] The effects of user input on game progress are predefined for each game. For example, user input can cause a character to move in a specified direction, change the character's line of sight to a specified direction, attack an enemy character, or defend.

[0094] The sending unit 14 sends input information to the server 20, representing the content of user input received by the receiving unit 13.

[0095] The receiving unit 15 receives from the server 20 a stream image representing the progress of a game being executed in the server 20 based on the input information sent by the sending unit 14.

[0096] The display control unit 18 displays the streaming image received by the receiving unit 15 on the display device. The display device is a monitor, projection device, etc. After performing predetermined processing such as decoding on the data received by the receiving unit 15, the display control unit 18 displays the streaming image on the display device.

[0097] Additionally, the display control unit 18 buffers streaming images. For example, the display control unit 18 buffers (temporarily stores) up to N frames of streaming images. N is a predefined value based on the number of streaming images required to generate a complete streaming image from the estimation model.

[0098] The detection unit 16 detects the phenomenon of missing stream images. If the next stream image is not received (not received by the receiving unit 15) within a predetermined time from the reference time, the detection unit 16 determines that the next stream image is missing. The reference time is the time when the previous stream image was received or a time based on that. The predetermined time is a pre-defined value.

[0099] The generation unit 17 generates a completed stream image representing the estimation result of the content of the lost stream image based on the estimation model corresponding to the currently running game stored in the storage unit 12 and the multiple stream images displayed before the lost stream image (multiple stream images related to the currently running game cached by the display control unit 18). Then, the display control unit 18 displays the completed stream image generated by the generation unit 17 on the display device as a replacement for the lost stream image.

[0100] For example, such as Figure 4 As shown, if the stream image I(t+1) at time (t+1) is lost, the generation unit 17 generates a complete stream image I'(t+1) based on the multiple stream images I(0) to I(t) displayed before the stream image I(t+1). Then, after the display control unit 18 displays the stream images I(0) to I(t) in this order, it displays the complete stream image I'(t+1) as a replacement for the lost stream image I(t+1). Furthermore, the stream image I(t) is the stream image immediately preceding the stream image I(t+1).

[0101] Next, use Figure 5 The flowchart below illustrates an example of the processing flow of client terminal 10. Figure 5 This illustrates the flow of display control processing performed by client terminal 10 after cloud gaming begins.

[0102] When client terminal 10 obtains a streaming image from server 20 (S10), it performs predetermined processing such as decoding and then displays the streaming image on a display device (S11). Then, client terminal 10 caches the streaming image (S12).

[0103] If there is no input to end the processing ("No" in S13), the client terminal 10 enters a waiting state to wait for the next streaming image (S14).

[0104] During the period when no next stream image is acquired (S14 "No"), client terminal 10 monitors whether a predetermined time has elapsed since the reference time (S15). The reference time is the time when the previous stream image was received or a time based on that. The predetermined time is a pre-defined value.

[0105] If no predetermined time has elapsed since the reference time (S15 "No"), the client terminal 10 continues the waiting state for the next streaming image (S14). Then, when the next streaming image is acquired before the predetermined time has elapsed since the reference time (S14 "Yes"), the client terminal 10 displays the streaming image on the display device after performing predetermined processing such as decoding (S11). Then, the client terminal 10 buffers the streaming image (S12).

[0106] On the other hand, when a predetermined time has elapsed from the reference time before the next stream image is acquired (S15 "Yes"), the client terminal 10 generates a completed stream image as a replacement for the lost stream image (S16). Specifically, the client terminal 10 generates the completed stream image based on the aforementioned estimation model corresponding to the currently executing game and multiple cached stream images.

[0107] Then, the client terminal 10 displays the completed stream image generated in S16 as a replacement for the lost stream image on the display device (S17). Next, the client terminal 10 caches the generated completed stream image (S18). Alternatively, the client terminal 10 may be configured to cache only the received stream image and not the generated completed stream image.

[0108] Subsequently, client terminal 10 repeats the same process as described above.

[0109] <Server 20>

[0110] Figure 3 An example of a functional block diagram of server 20 is shown. As shown, server 20 has an input information receiving unit 21, a game progress unit 22, and an image sending unit 23.

[0111] The input information receiving unit 21 receives input information from the client terminal 10. The input information receiving unit 21 receives the input information sent by the sending unit 14 of the client terminal 10.

[0112] The game progression unit 22 performs processing to advance the currently running game based on the input information received by the input information receiving unit 21, and generates a stream image representing the result of the game's progress. The content of the game progression based on the input information is a conventional design predefined for each game. For example, the game progression unit 22, based on the input information, moves the character in a predetermined direction within the game space, changes the character's line of sight to a predetermined direction, executes an attack on an enemy character, or executes a defense. Then, the game progression unit 22 generates a new stream image representing the result of these game progressions. These processes performed by the game progression unit 22 can be implemented using all well-known technologies.

[0113] The image sending unit 23 sends the streaming image generated by the game progress unit 22 to the client terminal 10. The image sending unit 23 sends the streaming image to the client terminal 10 in a streaming manner.

[0114] Example

[0115] Next, an embodiment of this implementation will be described. This embodiment can be modeled as the following technology ( Figure 6 ): Import a neural network model on the client side of cloud gaming. When given multiple streaming images I(0), I(1), I(2), ..., I(t) on the time series, the neural network model generates the streaming image I(t+1) at time (t+1).

[0116] The most important structural feature of this embodiment is that it monitors the streaming status from the cloud side and performs frame completion when the streaming is interrupted. This system is based on... Figure 6 It consists of the following three modules installed on the client side.

[0117] The [M1] Drop Detector monitors the reception of frame information (streamed images) rendered on the cloud side. If the next frame information does not arrive within a specified time, the [M3] Neural Frame Inference Engine is activated to complete the missing frame information. Furthermore, the [M1] Drop Detector saves the frame information received from the cloud to the [M2] Frame Buffer. The specified time is a standard design, but for example, in a 30fps game receiving 30 streamed images per second, the reception wait time for one streamed image is approximately 0.033 seconds (33 milliseconds).

[0118] The [M2] frame buffer is a fixed-length circular buffer that holds a maximum of N frames of streaming images. This system uses these N streaming images to infer depth information and optical flow to fill in missing streaming images. Specifically, the images stored in the [M2] frame buffer correspond to multiple streaming images I(0), I(1), I(2), ... I(t) in the aforementioned time series.

[0119] The [M3] neural frame inference engine uses neural network models such as DAIN and RIFE to generate the stream image I(t+1) at time (t+1) by taking multiple stream images I(0), I(1), I(2), ... I(t) stored in the [M2] frame buffer as input.

[0120] "Effects"

[0121] In the game system 1 of this embodiment, a neural network model (estimation model) of a generation system that generates the next streaming image based on the currently transmitted streaming images (game screens) is imported at the edge side (client terminal 10 side) of the cloud game. In the event of a loss such as not receiving the next streaming image within a specified time, the lost streaming image is generated at the edge side. Thus, even if several frames of streaming images are lost due to network latency, the lost streaming images can be filled in by the estimation model at the edge side to maintain a natural game screen.

[0122] The technical feature of the game system 1 in this embodiment is that the client terminal 10 does not have information inherent in the content such as 3D models and game logic, but only uses content-independent logic such as image information obtained from the server 20 to fill in the missing streaming images.

[0123] The advantage of the game system 1 in this embodiment is that it does not require significant changes to the existing cloud gaming architecture and communication methods. Lost streaming images can be completed simply by downloading the estimation model corresponding to the game to be played to the cloud gaming client terminal 10. In solving the problem of lost streaming images, the game system 1 in this embodiment also eliminates the compatibility issues with existing cloud gaming systems, which are a secondary problem.

[0124] The game system 1 of this embodiment performs highly general network latency compensation without increasing the load on the server 20 side, while flexibly utilizing the processor on the client terminal 10 side. Previous network latency compensation techniques for games mostly relied on specific game types and content, making it difficult to implement universal latency countermeasures. The game system 1 of this embodiment focuses on the fact that the data structure used in cloud rendering is an extremely simple video stream, and it employs an excellent technique to compensate for communication latency by performing frame completion on this video stream.

[0125] (Modified Example)

[0126] Next, a variation of this embodiment will be described. In this variation, the same effect as in this embodiment is achieved.

[0127] <Variation Example 1>

[0128] Server 20 may also intentionally cause the loss of several frames of streaming image at specified times without sending them to client terminal 10. Even so, there is no problem because the lost streaming image will be completed on the client terminal 10 side. As a result, the load on server 20 can be reduced.

[0129] For example, server 20 may also prevent the transmission of several frames of streaming image at predetermined intervals. Alternatively, server 20 may prevent the transmission of several frames of streaming image at predetermined intervals under predetermined conditions. These conditions include situations where the load on server 20 exceeds a predetermined level, such as when "the number of client terminals 10 connected to server 20 (the number of players running the game) exceeds a threshold."

[0130] <Variation Example 2>

[0131] In this embodiment described above, the generation unit 17 generates a completed stream image to fill in the missing stream image based on the detection unit 16 detecting the loss of the stream image. As a variation, the generation unit 17 may also generate a completed stream image corresponding to all stream images regardless of whether the detection unit 16 detects the loss of the stream image. In this variation, the generation unit 17 can generate the completed stream image at an earlier time. Theoretically, the generation unit 17 can start the process of generating the completed stream image I'(t+1) to fill in the stream image I(t+1) at the stage when the stream images I(0), I(1), I(2), ... I(t) are all present.

[0132] In this modified example, if the detection unit 16 does not detect a loss, the display control unit 18 displays the streaming image received by the receiving unit 15 from the server 20 on the display device. Furthermore, if the detection unit 16 detects a loss, the display control unit 18 displays the completed streaming image generated by the generation unit 17 on the display device.

[0133] <Variation Example 3>

[0134] The technology described in this embodiment can also be used in streaming distribution such as games and animations, as an alternative to its use in cloud gaming.

[0135] The embodiments of the present invention have been described above with reference to the accompanying drawings, but these are merely examples of the invention, and various other structures besides those described above can be employed. The structures of the above-described embodiments can also be combined with each other, or some structures can be replaced with other structures. Furthermore, the structures of the above-described embodiments can be modified in various ways without departing from the spirit of the invention. Additionally, the structures and processes disclosed in the various embodiments and modifications described above can be combined with each other.

[0136] Furthermore, while the flowcharts used in the above description sequentially depict multiple processes, the execution order of these processes in each embodiment is not limited to the order in which they are depicted. In each embodiment, the order of the illustrated processes can be changed to the extent that there are no content-related obstacles. Additionally, the above embodiments can be combined to the extent that their contents do not contradict each other.

[0137] Some or all of the above-described embodiments may also be described as follows, but are not limited to the following notes.

[0138] 1. A program that enables a computer on a client terminal to function as:

[0139] The reception department handles user input related to the progress of the game in progress.

[0140] The sending unit sends input information representing the content input by the user to the server;

[0141] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0142] The detection unit detects the phenomenon of lost streaming images;

[0143] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0144] The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

[0145] 2. According to the procedure described in 1, wherein,

[0146] The program enables the computer to also function as an acquisition unit, which acquires the estimated model from the providing server.

[0147] 3. According to the procedure described in 2, wherein,

[0148] The acquisition unit obtains any one of the multiple estimation models prepared corresponding to the multiple games from the providing server.

[0149] 4. According to the procedure described in 3, wherein,

[0150] The acquisition unit obtains from the providing server the estimated model corresponding to the game selected by the user input from among multiple games.

[0151] 5. According to the procedure described in 1, wherein,

[0152] If no next stream image is received within a specified time from the reference time, the detection unit determines that the stream image is lost.

[0153] 6. The procedure according to 1, wherein,

[0154] The estimation model generates the completed flow image based on optical flow and depth information estimated from the multiple flow images.

[0155] 7. A client terminal, comprising:

[0156] The reception department handles user input related to the progress of the game in progress.

[0157] The sending unit sends input information representing the content input by the user to the server;

[0158] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0159] The detection unit detects the phenomenon of lost streaming images;

[0160] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0161] The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

[0162] 8. A processing method, wherein the following processing is performed by a client terminal:

[0163] Accept user input related to the progress of the game being played;

[0164] Send input information representing the content input by the user to the server;

[0165] Receive streaming images from the server, wherein the streaming images represent the progress of the game being executed in the server based on the input information;

[0166] Detect the phenomenon of missing streaming images;

[0167] Based on an estimation model and multiple stream images displayed before the missing stream image, a completed stream image representing an estimation result of the content of the missing stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently executing game, and generate a stream image representing the estimated content; and

[0168] The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image.

[0169] 9. A game system comprising a client terminal and a server, wherein,

[0170] The client terminal has:

[0171] The reception department handles user input related to the progress of the game in progress.

[0172] The sending unit sends input information representing the content input by the user to the server;

[0173] A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information;

[0174] The detection unit detects the phenomenon of lost streaming images;

[0175] A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and

[0176] The display control unit displays the streaming image on the display device and displays the completed streaming image as a replacement for the missing streaming image on the display device.

[0177] The server has:

[0178] An input information receiving unit receives the input information from the client terminal;

[0179] A game progression unit, based on the input information, performs processing to advance the currently running game and generates the stream image representing the result of the game's progress; and

[0180] An image transmitting unit that sends the streaming image to the client terminal.

[0181] 10. A processing method, wherein the client terminal of a game system having a client terminal and a server performs the following processing:

[0182] Accept user input related to the progress of the game being played;

[0183] Send input information representing the content input by the user to the server;

[0184] Receive streaming images from the server, wherein the streaming images represent the progress of the game being executed in the server based on the input information;

[0185] Detect the phenomenon of missing streaming images;

[0186] Based on an estimation model and multiple stream images displayed before the missing stream image, a completed stream image representing an estimation result of the content of the missing stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently executing game, and generate a stream image representing the estimated content; and

[0187] The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image.

[0188] In the processing method, the server performs the following processing:

[0189] Receive the input information from the client terminal;

[0190] Based on the input information, perform processing to advance the currently running game, and generate the stream image representing the result of the progress of the currently running game; and

[0191] The streaming image is sent to the client terminal.

[0192] This application claims priority based on Japanese Patent Application No. 2023-052653, filed on March 29, 2023, the entire disclosure of which is incorporated herein by reference.

[0193] Explanation of reference numerals in the attached figures

[0194] 1: Game system; 10: Client terminal; 11: Acquisition unit; 12: Storage unit; 13: Acceptance unit; 14: Transmission unit; 15: Receiving unit; 16: Detection unit; 17: Generation unit; 18: Display control unit; 20: Server; 21: Input information receiving unit; 22: Game progress unit; 23: Image transmission unit; 30: Communication network; 1A: Processor; 2A: Memory; 3A: Input / output (I / F); 4A: Peripheral circuit; 5A: Bus.

Claims

1. A program that enables a computer on a client terminal to function as: The reception department handles user input related to the progress of the game in progress. The sending unit sends input information representing the content input by the user to the server; The receiving unit receives the streaming image from the server, wherein, The streaming image represents the result of the progress of the game being executed on the server based on the input information; The detection unit detects the phenomenon of lost streaming images; A generation unit generates a completed stream image representing an estimation result of the content of the lost stream image, based on an estimation model and multiple stream images displayed before the lost stream image. The estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the game being executed, and to generate a stream image representing the estimated content. as well as The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

2. The procedure according to claim 1, wherein, The program enables the computer to also function as an acquisition unit, which acquires the estimated model from the providing server.

3. The procedure according to claim 2, wherein, The acquisition unit obtains any one of the multiple estimation models prepared corresponding to the multiple games from the providing server.

4. The procedure according to claim 3, wherein, The acquisition unit obtains from the providing server the estimated model corresponding to the game selected by the user input from among multiple games.

5. The procedure according to claim 1, wherein, If no next stream image is received within a specified time from the reference time, the detection unit determines that the stream image is lost.

6. The procedure according to claim 1, wherein, The estimation model generates the completed flow image based on optical flow and depth information estimated from the multiple flow images.

7. A client terminal, comprising: The reception department handles user input related to the progress of the game in progress. The sending unit sends input information representing the content input by the user to the server; The receiving unit receives the streaming image from the server, wherein, The streaming image represents the result of the progress of the game being executed on the server based on the input information; The detection unit detects the phenomenon of lost streaming images; A generation unit generates a completed stream image representing an estimation result of the content of the lost stream image, based on an estimation model and multiple stream images displayed before the lost stream image. The estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the game being executed, and to generate a stream image representing the estimated content. as well as The display control unit displays the streaming image on the display device and displays the completed streaming image on the display device as a replacement for the missing streaming image.

8. A processing method in which the following processing is performed by a client terminal: Accept user input related to the progress of the game being played; Send input information representing the content input by the user to the server; Receive streaming images from the server, wherein, The streaming image represents the result of the progress of the game being executed on the server based on the input information; Detect the phenomenon of missing streaming images; Based on an estimation model and multiple stream images displayed before the lost stream image, a completed stream image representing an estimation result of the content of the lost stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the game being executed, and to generate a stream image representing the estimated content; as well as The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image.

9. A game system comprising a client terminal and a server, wherein, The client terminal has: The reception department handles user input related to the progress of the game in progress. The sending unit sends input information representing the content input by the user to the server; A receiving unit receives a streaming image from the server, wherein the streaming image represents the result of the progress of the game being executed in the server based on the input information; The detection unit detects the phenomenon of lost streaming images; A generation unit, based on an estimation model and multiple stream images displayed before the missing stream image, generates a completed stream image representing an estimation result of the content of the missing stream image, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently running game, and generate a stream image representing the estimated content; and The display control unit displays the streaming image on the display device and displays the completed streaming image as a replacement for the missing streaming image on the display device. The server has: An input information receiving unit receives the input information from the client terminal; A game progression unit, based on the input information, performs processing to advance the currently running game and generates the stream image representing the result of the game's progress; and An image transmitting unit that sends the streaming image to the client terminal.

10. A processing method, wherein the client terminal of a game system having a client terminal and a server performs the following processing: Accept user input related to the progress of the game being played; Send input information representing the content input by the user to the server; Receive streaming images from the server, wherein the streaming images represent the progress of the game being executed in the server based on the input information; Detect the phenomenon of missing streaming images; Based on an estimation model and multiple stream images displayed before the missing stream image, a completed stream image representing an estimation result of the content of the missing stream image is generated, wherein the estimation model is used to estimate the content of a stream image displayed after the multiple stream images related to the currently executing game, and generate a stream image representing the estimated content; and The streaming image is displayed on the display device, and the completed streaming image is displayed on the display device as a replacement for the missing streaming image. In the processing method, the server performs the following processing: Receive the input information from the client terminal; Based on the input information, perform processing to advance the currently running game, and generate the stream image representing the result of the progress of the currently running game; and The streaming image is sent to the client terminal.

Images