Video transmission device, control method, and program

Abstract

To provide a moving image receiver, a moving image transmitter, a moving image transmission / reception system, a control method, and a program that can generate image data with a data size taking into account of low latency.SOLUTION: An operation data transmission unit 44 transmits operation data corresponding to user's input operation to a cloud server 10. A VSP receiving unit 46 receives a packet transmitted from the cloud server 10 in response to the start of generation of a frame image based on the operation data, the packet being associated with the operation data. A terminal-side traffic control unit 54 controls a data size of image data to be transmitted by the cloud server 10 on the basis of a packet reception time being a time from timing when the operation data is transmitted to timing when the packet associated with the operation data is received.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a moving image receiving device, a moving image transmitting device, a moving image transmission / reception system, a control method, and a program.

Background Art

[0002] In the technology of cloud gaming services, which has been attracting attention in recent years, operation data corresponding to a user's input operation on a terminal is wirelessly transmitted from the terminal to a cloud server. Then, in the cloud server, a frame image representing the play status of the game is generated based on the operation data. Then, image data obtained by encoding the frame image is wirelessly transmitted from the cloud server to the terminal, and a frame image obtained by decoding the image data is displayed on the terminal. By repeatedly executing this series of processes, a moving image representing the play status of the game is displayed on the terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In the cloud gaming service described above, it is desirable that the moving image displayed on the terminal has as high an image quality as possible. Therefore, it is desirable that the data size of the image data generated based on the frame image constituting the moving image is as large as possible.

[0004] However, in wireless communication that requires the above-mentioned immediacy, in order to reduce the user's sense of discomfort in the operation, it is important that each frame image is displayed on the terminal without delay and with low latency even if the data size of the image data is reduced. In a situation where moving image transmission is performed using a mobile communication system such as the fourth-generation mobile communication system (4G) or the fifth-generation mobile communication system (5G), which has large bandwidth fluctuations and is likely to cause delays, special attention must be paid to ensuring that the frame image is displayed on the terminal with low latency.

[0005] Furthermore, this applies not only to situations where cloud gaming services are provided, but also generally to situations where video is transmitted from a video transmission device equivalent to the aforementioned cloud server.

[0006] The present invention has been made in view of the above circumstances, and one of its objectives is to provide a video receiving device, a video transmitting device, a video transmitting and receiving system, a control method, and a program that can generate image data of an appropriate data size that takes delay into consideration. [Means for solving the problem]

[0007] To solve the above problems, the present invention provides a video receiving device that sequentially receives image data representing frame images constituting a video from a video transmitting device, and includes: an operation data transmission unit that transmits operation data corresponding to user input operations to the video transmitting device; a packet receiving unit that receives packets associated with the operation data transmitted from the video transmitting device in response to the start of generation of the frame images based on the operation data; and a data size control unit that controls the data size of the image data to be transmitted by the video transmitting device based on a packet reception time, which is the time from the time the operation data is transmitted to the time the packet associated with the operation data is received.

[0008] In one aspect of the present invention, the data size control unit controls the data size based on the time required to receive the image data transmitted from the video transmission device.

[0009] In one aspect of the present invention, the data size control unit controls the data size based on the packet reception time for the most recent packet and the packet reception time for at least one packet prior to the most recent packet.

[0010] Furthermore, in one aspect of the present invention, the data size control unit controls the data size to decrease when it is determined that the failure to receive the packet continues based on predetermined conditions.

[0011] Furthermore, the video transmission device according to the present invention is a video transmission device that sequentially transmits image data representing frame images constituting a video to a video receiving device, and includes: an operation data receiving unit that receives operation data from the video receiving device in response to user input operations; an image generation unit that generates the frame images based on the operation data; a packet transmission unit that transmits packets associated with the operation data to the video receiving device in response to the start of the generation of the frame images; an encoding processing execution unit that generates image data representing the frame images by encoding the frame images; an image data transmission unit that transmits the image data to the video receiving device; and a data size control unit that controls the data size of the image data to be transmitted by the image data transmission unit based on the time from the timing when the video receiving device transmits the operation data to the timing when the packet associated with the operation data is received.

[0012] Furthermore, the video transmission and reception system according to the present invention includes a video transmission device that sequentially transmits image data representing frame images constituting a video, and a video reception device that sequentially receives the image data, wherein the video reception device includes an operation data transmission unit that transmits operation data corresponding to user input operations to the video transmission device, a packet reception unit that receives packets associated with the operation data transmitted from the video transmission device in response to the start of generation of frame images based on the operation data, and a control that controls the data size of the image data to be transmitted by the video transmission device based on a packet reception time, which is the time from the time the operation data is transmitted to the time the packet associated with the operation data is received. The motion image transmission device includes a control data transmission unit that transmits data to the motion image transmission device, the motion image transmission device includes an operation data receiving unit that receives the operation data from the motion image receiving device, an image generation unit that generates the frame image based on the operation data, a packet transmission unit that transmits a packet associated with the operation data to the motion image receiving device in response to the start of the generation of the frame image, an encoding processing execution unit that generates image data representing the frame image by encoding the frame image, an image data transmission unit that transmits the image data to the motion image receiving device, a control data receiving unit that receives the control data, and a data size control unit that controls the data size of the image data to be transmitted by the image data transmission unit based on the control data.

[0013] Furthermore, the control method according to the present invention includes the steps of: a video receiving device that sequentially receives image data representing frame images constituting a video from a video transmitting device, and transmits operation data corresponding to a user's input operation to the video transmitting device; the video receiving device receiving a packet associated with the operation data, which is transmitted from the video transmitting device in response to the start of generation of the frame images based on the operation data; and the video receiving device controlling the data size of the image data to be transmitted by the video transmitting device based on a packet reception time, which is the time from the time the operation data was transmitted to the time the packet associated with the operation data was received.

[0014] Another control method according to the present invention includes the steps of: a video transmitting device sequentially transmitting image data representing frame images constituting a video to a video receiving device receiving operation data from the video receiving device in response to a user's input operation; the video transmitting device generating the frame images based on the operation data; the video transmitting device transmitting a packet associated with the operation data to the video receiving device in response to the start of the generation of the frame images; the video transmitting device encoding the frame images to generate image data representing the frame images; the video transmitting device transmitting the image data to the video receiving device; and the video transmitting device controlling the data size of the image data to be transmitted by the video transmitting device based on the time from when the video receiving device transmitted the operation data to when it received the packet associated with the operation data.

[0015] Furthermore, the program according to the present invention causes a computer that sequentially receives image data representing frame images constituting a moving image from a moving image transmission device to execute the following steps: a procedure to send operation data corresponding to user input operations to the moving image transmission device; a procedure to receive a packet associated with the operation data, which is sent from the moving image transmission device in response to the start of generation of the frame images based on the operation data; and a procedure to control the data size of the image data to be transmitted by the moving image transmission device based on the packet reception time, which is the time from the time the operation data was sent to the time the packet associated with the operation data was received.

[0016] Furthermore, another program according to the present invention causes a computer that sequentially transmits image data representing frame images constituting a moving image to a moving image receiving device to execute the following steps: receiving operation data corresponding to user input operations from the moving image receiving device; generating the frame images based on the operation data; transmitting packets associated with the operation data to the moving image receiving device in response to the start of the generation of the frame images; generating image data representing the frame images by encoding the frame images; transmitting the image data to the moving image receiving device; and controlling the data size of the image data to be transmitted by the computer based on the time from when the moving image receiving device transmits the operation data to when it receives the packets associated with the operation data. [Brief explanation of the drawing]

[0017] [Figure 1] This figure shows an example of the overall configuration of a cloud gaming system according to one embodiment of the present invention. [Figure 2] This diagram schematically illustrates an example of the timing of processing that occurs on a cloud server related to one embodiment of the present invention. [Figure 3] This diagram schematically shows an example of communication that occurs in a cloud gaming system according to one embodiment of the present invention. [Figure 4]It is an explanatory diagram for explaining an example of bitrate control performed in a cloud gaming system according to an embodiment of the present invention. [Figure 5] It is an explanatory diagram for explaining an example of pad sync control performed in a cloud gaming system according to an embodiment of the present invention. [Figure 6] It is a functional block diagram showing an example of functions implemented in a cloud gaming system according to an embodiment of the present invention. [Figure 7] It is a flowchart showing an example of the flow of processing performed in a cloud server according to an embodiment of the present invention. [Figure 8] It is a flowchart showing an example of the flow of processing performed in a cloud server according to an embodiment of the present invention. [Figure 9] It is a flowchart showing an example of the flow of processing performed in a cloud server according to an embodiment of the present invention. [Figure 10] It is a flowchart showing an example of the flow of processing performed in a cloud server according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] FIG. 1 is a diagram showing an example of the overall configuration of a cloud gaming system 1 according to an embodiment of the present invention. As shown in FIG. 1, the cloud gaming system 1 according to the present embodiment includes a cloud server 10 and a terminal 12, both of which are centered around a computer.

[0020] The cloud server 10 and the terminal 12 are connected to a computer network 14 including a mobile communication system such as the fourth-generation mobile communication system (4G) or the fifth-generation mobile communication system (5G), or the Internet. For example, the cloud server 10 is connected to the Internet, and the terminal 12 is connected to a mobile communication system such as 4G or 5G. Then, the cloud server 10 and the terminal 12 can communicate with each other via the computer network 14.

[0021] The cloud server 10 according to the present embodiment is, for example, a server computer that executes a program of a game related to a cloud gaming service.

[0022] As shown in FIG. 1, the cloud server 10 includes, for example, a processor 10a, a storage unit 10b, a communication unit 10c, and an encoder / decoder unit 10d.

[0023] The processor 10a is a program control device such as a CPU, for example, and executes various information processes according to a program stored in the storage unit 10b. The processor 10a according to the present embodiment also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on graphics commands and data supplied from the CPU.

[0024] The storage unit 10b is, for example, a storage element such as a ROM or a RAM, or a solid state drive (SSD). Programs executed by the processor 10a and the like are stored in the storage unit 10b. In addition, an area of a frame buffer in which an image is drawn by the GPU included in the processor 10a is secured in the storage unit 10b according to the present embodiment.

[0025] The communication unit 10c is a communication interface for exchanging data with computers such as the terminal 12 via the computer network 14, for example.

[0026] The encoder / decoder unit 10d includes, for example, an encoder and a decoder. The encoder generates image data representing an input image by encoding the input image. The decoder decodes the input image data and outputs the image represented by the image data.

[0027] The terminal 12 in this embodiment is, for example, a computer such as a smartphone or tablet used by a user of a cloud gaming service. Alternatively, the terminal 12 may be an electronic device capable of communicating with the cloud server 10 via a communication dongle, such as a television with a communication dongle.

[0028] As shown in Figure 1, the terminal 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, a display unit 12d, an operation unit 12e, a sensor unit 12f, an audio output unit 12g, and an encoder / decoder unit 12h.

[0029] The processor 12a is a program control device such as a CPU, which performs various information processing according to the program stored in the memory unit 12b.

[0030] The memory unit 12b is, for example, a memory element such as ROM or RAM, or a solid-state drive (SSD). The memory unit 12b stores programs that are executed by the processor 12a.

[0031] The communication unit 12c is a communication interface for exchanging data with a computer, such as a cloud server 10, via a computer network 14.

[0032] The display unit 12d is a display device such as a liquid crystal display or an organic EL display.

[0033] The operation unit 12e is, for example, an operating member for providing operation input to the processor 12a.

[0034] The sensor unit 12f is a sensor such as a motion sensor capable of detecting acceleration or angular velocity.

[0035] The audio output unit 12g is an audio output device, such as a speaker, that outputs sound represented by audio data.

[0036] The encoder / decoder unit 12h includes, for example, an encoder and a decoder. The encoder generates image data representing an input image by encoding the input image. The decoder decodes the input image data and outputs the image represented by the image data.

[0037] The terminal 12 may also be equipped with a touch panel. In this case, the touch panel will serve the functions of both the display unit 12d and the operation unit 12e described above.

[0038] In this embodiment, when a user performs an input operation on the control unit 12e during gameplay in a cloud gaming service, the terminal 12 generates operation data corresponding to the input operation and transmits the operation data to the cloud server 10. Hereinafter, this operation data will be referred to as pad data P.

[0039] The cloud server 10 then executes game processing according to the received pad data P. Based on the results of the game processing, the cloud server 10 generates a play image, which is a frame image representing the game's play status, and draws the play image to the cloud server 10's frame buffer. In this embodiment, the game processing and play image generation are repeatedly executed.

[0040] The cloud server 10 then acquires the play image drawn on the frame buffer and encodes the play image to generate image data representing the play image. The cloud server 10 then sends the generated image data to the terminal 12. The terminal 12 then decodes the image data received from the cloud server 10 and displays the play image generated by the decoding on the display unit 12d.

[0041] As described above, the cloud server 10 according to this embodiment streams video and images generated according to the game's play status to the terminal 12 used by the user playing the game.

[0042] Figure 2 is a schematic diagram illustrating an example of the timing of processing that occurs in the cloud server 10 according to this embodiment.

[0043] In this embodiment, for example, in the initial state, terminal 12 transmits pad data P to the cloud server 10 at a predetermined period (for example, a 4-millisecond period) that indicates the input operation received by the operation unit 12e at that timing. The pad data P in this embodiment is associated with a sequence number that represents the transmission order of the pad data P.

[0044] Figure 2 shows the reception timing of the pad data P(0) to P(10) transmitted in this manner at the cloud server 10. In Figure 2, the sequence number associated with the received pad data P is shown as the number in parentheses in P(0) to P(10). Depending on the communication quality of the computer network 14, the cloud server 10 may not necessarily receive the pad data P at 4-millisecond intervals.

[0045] The cloud server 10 generates play images at predetermined intervals (for example, every 16 milliseconds). At this time, the cloud server 10 generates a play image based on the input operation indicated by the most recently received pad data P (latest pad data P) at the start of play image generation, and draws the generated play image to the frame buffer.

[0046] The period G shown in Figure 2 represents the period for generating the gameplay image. In the example in Figure 2, during period G(m,n), a gameplay image with sequence number m is generated based on pad data P(n) with sequence number n. That is, in Figure 2, the period for generating the gameplay image based on pad data P(0) corresponds to the period shown G(1,0). The period for generating the gameplay image based on pad data P(4) corresponds to the period shown G(2,4). The period for generating the gameplay image based on pad data P(8) corresponds to the period shown G(3,8).

[0047] In this embodiment, the transmission cycle of pad data P at terminal 12 and the generation cycle of play images at cloud server 10 are different. For example, in the initial state, the generation cycle of play images is four times the transmission cycle of pad data P. Therefore, not all pad data P received by cloud server 10 is used to generate play images. In the example in Figure 2, pad data P indicated as P(1)~P(3), P(5)~P(7), P(9), and P(10) are not used to generate play images.

[0048] Furthermore, in this embodiment, the cloud server 10 sends a packet to the terminal 12 indicating that the generation of a play image based on the pad data P has started, at the time the generation of the play image based on the pad data P has started. Hereinafter, this packet will be referred to as a Video Sync Packet (VSP). In this embodiment, the VSP sent at the time the generation of a play image based on the pad data P has started includes the sequence number of the play image (m above) and the sequence number of the pad data P (n above). In this way, the VSP according to this embodiment is associated with the play image on a one-to-one basis.

[0049] Figure 2 shows that VSP(1,0) is sent when the generation of play image with sequence number 1 begins. It also shows that VSP(2,4) is sent when the generation of play image with sequence number 2 begins. Furthermore, it shows that VSP(3,8) is sent when the generation of play image with sequence number 3 begins.

[0050] Furthermore, in this embodiment, for example, when the drawing of a play image to the frame buffer is completed, the encoding of the play image and the transmission of the image data generated by the encoding are performed. The frame buffer of the cloud server 10 in this embodiment is configured as a multi-buffer, and the drawing of the next play image can be performed in parallel with the encoding of the play image that has been drawn.

[0051] The period S shown in Figure 2 corresponds to the period during which the gameplay image is encoded and the image data is transmitted. In the example in Figure 2, during period S(m,n), the encoding of the gameplay image with sequence number m, generated based on the pad data P with sequence number n, and the transmission of the image data generated by this encoding take place. That is, in Figure 2, the period during which the gameplay image with sequence number 1 is encoded and the image data generated by this encoding is transmitted corresponds to the period shown as S(1,0). Also, the period during which the gameplay image with sequence number 2 is encoded and the image data generated by this encoding is transmitted corresponds to the period shown as S(2,4).

[0052] In this embodiment, the image data generated by encoding the play image generated based on the pad data P is associated with the sequence number of the pad data P (n above) and the sequence number of the play image (m above).

[0053] Furthermore, in this embodiment, the cloud server 10 measures the time from the timing of receiving the pad data P to the timing of the start of generating the play image based on the pad data P. The cloud server 10 then generates interval data indicating the measured time. In this embodiment, the image data generated based on the play image is associated with the interval data generated in this way.

[0054] Figure 3 schematically shows an example of communication that occurs between the cloud server 10 and the terminal 12 from the transmission of pad data P(4) with sequence number 4 to the reception of image data generated based on said pad data P(4). In the example in Figure 3, the image data transmitted during the period indicated as S(2,4) in Figure 2 is indicated as D(2,4).

[0055] In this embodiment, terminal 12 identifies the packet reception time, which is the time from the time pad data P is transmitted to the time VSP associated with said pad data P is received. Hereinafter, said packet reception time will also be expressed as PadVspRTT, as shown in Figure 3. In this embodiment, terminal 12 identifies PadVspRTT based, for example, on the sequence number of pad data P included in the VSP received from cloud server 10.

[0056] Furthermore, as shown in Figure 3, the time from the transmission of pad data P at terminal 12 to the reception of the first segment of image data generated based on said pad data P will be referred to as PadFirstFragRTT.

[0057] Furthermore, the time from the transmission of pad data P in terminal 12 to the reception of the last segment of image data generated based on said pad data P will be referred to as PadFrameRTT.

[0058] Furthermore, the time from the reception of the first segment of the image data to the reception of the last segment of the image data on terminal 12 will be referred to as TransferTime. Hereafter, TransferTime will be denoted as TT.

[0059] In this embodiment, terminal 12 determines the value of TT based on image data received from cloud server 10. For example, terminal 12 may determine the value of PadFrameRTT and PadFirstFragRTT based on the sequence number of pad data P associated with the image data. Terminal 12 may then determine the value of TT by subtracting the value of PadFirstFragRTT from the value of PadFrameRTT.

[0060] In this embodiment, if a slice transfer method is employed in which a single play image is divided into multiple slices and encoding and image data transmission are performed for each slice, the TT may be determined taking into account the number of slices per frame. For example, the TT may be determined taking into account a predetermined period of inactivity between the end of transmission of image data for one slice and the start of transmission of image data for the next slice. In this case, for example, the TT value may be determined by subtracting the value obtained by subtracting the value obtained by subtracting the value obtained by subtracting the value obtained by subtracting (number of slices per frame - 1) multiplied by the aforementioned period of inactivity from the value obtained by subtracting the value obtained by subtracting the value obtained by subtracting the value obtained by PadFirstFragRTT from the value obtained by PadFrameRTT, and then subtracting that value.

[0061] Furthermore, as shown in Figure 3, the time from the timing of receiving pad data P in the cloud server 10 to the timing of the start of generating a play image based on said pad data P will be called PRecvGStartTime.

[0062] In this embodiment, terminal 12 identifies PRecvGStartTime based on interval data associated with image data.

[0063] Furthermore, as shown in Figure 2, the time from when the VSP is received in terminal 12 to when the next pad data P is sent will be referred to as DiffPadVsp.

[0064] The bitrate control performed in the cloud gaming system 1 according to this embodiment will be described below.

[0065] In the cloud gaming service provided by the cloud gaming system 1, it is desirable that the video displayed on the terminal 12 be of the highest possible quality. Therefore, it is desirable that the data size of the image data generated based on the gameplay images that make up the video be as large as possible.

[0066] However, in wireless communication where immediacy is required, as in this embodiment, it is important to reduce the data size of the image data so that each play image is displayed to the terminal 12 smoothly and with low latency, in order to reduce user discomfort during operation.

[0067] Furthermore, mobile communication systems such as 4G and 5G have several drawbacks, including variations in the quality of wireless communication channels (radio wave strength), the inability to allocate communication resources due to congestion, and handovers between base stations due to movement. As a result, mobile communication systems such as 4G and 5G tend to experience significant bandwidth fluctuations and delays.

[0068] Furthermore, in a Wi-Fi® communication environment, the communication quality does not differ significantly between the uplink (communication from terminal 12 to cloud server 10 in this embodiment) and the downlink (communication from cloud server 10 to terminal 12 in this embodiment). On the other hand, in communication environments such as 4G and 5G, it is common for the communication quality of one of the uplink or downlink to be good while the other is poor.

[0069] Therefore, in situations where video and images are transmitted using mobile communication systems such as 4G or 5G, particular attention must be paid to ensuring that frame images are displayed on the terminal 12 with low latency.

[0070] Therefore, in this embodiment, by performing the bitrate control shown in Figure 4, image data of an appropriate data size that takes delay into account is generated on the cloud server 10.

[0071] Figure 4 is an explanatory diagram illustrating an example of bitrate control performed in the cloud gaming system 1 according to this embodiment.

[0072] As shown in Figure 4, the bitrate control according to this embodiment performs two types of control: bandwidth fluctuation tracking control and packet congestion reduction control.

[0073] In bandwidth fluctuation tracking control, first, to prevent image data loss even if the communication environment deteriorates somewhat, the bitrate is controlled so that the TT value of each frame approaches half the reciprocal of the frame rate (FPS) of the transmitted video, providing a margin for jitter. For example, if the frame rate of the transmitted video is 60fps, the bitrate in the video communication is controlled so that the TT approaches approximately 8 milliseconds.

[0074] For example, the bitrate value R is determined by PID control, where 1 / (FPS × 2) is the target value, TT is the current value, and the bitrate value R is the manipulated variable.

[0075] This is shown in Figure 4 as R = PID(TT - (1 / (FPS × 2))). For example, the larger the value of TT - (1 / (FPS × 2)), the smaller the value of R will be controlled to be. Also, if the value of TT is greater than the value of 1 / (FPS × 2), the value of R will be controlled to be smaller, and if the value of TT is less than the value of 1 / (FPS × 2), the value of R will be controlled to be larger.

[0076] Furthermore, in bandwidth fluctuation tracking control, if the packet reception time (PadVspRTT) increases sharply, a further reduction in the bitrate is immediately implemented to quickly track bandwidth fluctuations.

[0077] For example, fine noise can be removed by passing the time-series data of PadVspRTT through a predetermined low-pass filter. Hereafter, PadVspRTT with fine noise removed in this way will be referred to as FilPadVspRTT. This is shown in Figure 4 as FilPadVspRTT = LPF(PadVspRTT).

[0078] Then, the bitrate reduction amount D is determined by PD control, where the value of EstBtmLatency (described later) is the target value, the value of FilPadVspRTT is the current value, and the bitrate reduction amount D is the manipulated variable. Here, PD control is used instead of PID control in order to quickly track bandwidth fluctuations.

[0079] This is shown in Figure 4 as D = PD(FilPadVspRTT - EstBtmLatency). For example, the larger the value of FilPadVspRTT - EstBtmLatency, the larger the value of D will be controlled to be. Also, if the value of FilPadVspRTT is greater than the value of EstBtmLatency, the value of D will be controlled to be larger. Also, if the value of FilPadVspRTT is smaller than the value of EstBtmLatency, the value of D will be controlled to be smaller.

[0080] In this embodiment, the value of EstBtmLatency is initially set to a predetermined value. Each time PadVspRTT is identified, the absolute value V of the difference between the latest PadVspRTT (PadVspRTT[n]) and the previous PadVspRTT (PadVspRTT[n-1]) is identified. Then, for a given value N and a predetermined threshold Th1, the value of EstBtmLatency is updated when the state in which the absolute value V is less than Th1 occurs N times in a row. For example, the value of EstBtmLatency is updated to the average value of the PadVspRTT values ​​over the N consecutive times in which the state in which the absolute value V is less than Th1 occurred. This is shown in Figure 4 as EstBtmLatency=Average(PadVspRTT[n]~PadVspRTT[n-N+1]).

[0081] In this embodiment, the PadVspRTT value in a state where the bandwidth is relatively stable is set as the EstBtmLatency value. Therefore, the greater the discrepancy between the current PadVspRTT value and the PadVspRTT value in a stable state, the larger the determined value of D becomes.

[0082] In this embodiment, the final bitrate value is determined by subtracting the bitrate reduction value D from the bitrate value R determined as described above. However, if the determined value D is negative, no adjustment is made to the value R, and the value R is used as is for value B. This is shown in Figure 4 as B=RD (where D=0 if D<0).

[0083] Bandwidth fluctuation tracking control may be performed at predetermined timings. For example, bandwidth fluctuation tracking control may be performed at a predetermined period (e.g., a 16-millisecond period). Alternatively, bandwidth fluctuation tracking control may be performed in response to the occurrence of a predetermined event (e.g., reception of the last segment of image data).

[0084] Furthermore, in this embodiment, the terminal 12 monitors the reception of VSPs, for example, at the transmission cycle of pad data P. If the number of VSPs M received in the most recent predetermined time t1 is less than a predetermined threshold Th2, the bandwidth fluctuation tracking control is interrupted. For example, the bandwidth fluctuation tracking control is interrupted if the number of received VSPs M in the most recent 100 milliseconds is less than 5, or if no VSPs have been received in the most recent 80 milliseconds.

[0085] Then, a packet congestion reduction control process is executed, which involves multiplying the bitrate value B by a predetermined ratio r (where r is less than 1). Note that the process executed in the packet congestion reduction control does not necessarily have to be a process that reduces value B by a ratio r; for example, a process that subtracts a predetermined value from value B may be executed.

[0086] In this embodiment of packet congestion reduction control, once the value B reaches a predetermined lower limit b1, the control is performed to prevent the value B from becoming any smaller.

[0087] Packet congestion reduction control may be executed at predetermined timings. For example, packet congestion reduction control may be executed at predetermined intervals (e.g., the transmission interval of pad data P). Alternatively, packet congestion reduction control may be executed in response to the occurrence of a predetermined event (e.g., the transmission of pad data P).

[0088] In this embodiment, if the number of VSPs M received in the most recent predetermined time t1 exceeds a predetermined threshold Th2, the packet congestion reduction control is terminated and the bandwidth fluctuation tracking control is restarted.

[0089] In mobile communication systems such as 4G and 5G, extensive retransmission and buffering control are implemented, making it easy for data to accumulate in the buffers of relay devices such as base stations. For example, if downlink communication stops in a 4G or 5G communication environment, data transmitted from the cloud server 10 tends to accumulate in relay devices such as base stations. This accumulated data is then transmitted all at once when downlink communication returns to normal.

[0090] Increased delays due to the discharge of accumulated data, and overflows in the receive buffer of terminal 12 due to terminal 12 receiving a large amount of data at once, are factors that cause data loss in terminal 12. In this embodiment, by performing packet congestion reduction control as described above, the amount of data accumulating in the computer network 14 when communication becomes impossible due to deterioration of the communication environment is reduced, and the occurrence of data loss is suppressed.

[0091] In this embodiment, the cloud server 10 changes the compression ratio in encoding the play image so that the bitrate value of the transmitted video image becomes a value B determined by bandwidth fluctuation tracking control or a value B updated by packet congestion reduction control. In this way, the data size of the generated image data is controlled in this embodiment.

[0092] As described above, according to this embodiment, image data of an appropriate data size that takes delay into account is generated on the cloud server 10.

[0093] Furthermore, in this embodiment, as described above, the data size of the image data to be transmitted is controlled by the cloud server 10 based on the packet reception time (PadVspRTT). By doing so, the bitrate of the video can be immediately reduced in response to a sudden drop in throughput.

[0094] The following describes the pad sync control performed in the cloud gaming system 1 according to this embodiment.

[0095] In the cloud gaming service provided by the cloud gaming system 1, it is desirable that the time from receiving operation data on the cloud server 10 to the start of generating the gameplay image be as constant as possible in order to reduce user discomfort during operation.

[0096] However, depending on the communication environment, the time it takes for operation data transmitted from terminal 12 to reach cloud server 10 may vary. This is particularly noticeable in wireless communication using mobile communication systems with large bandwidth fluctuations, such as 4G and 5G.

[0097] As a result, the time from receiving operation data on the cloud server 10 to starting the generation of gameplay images sometimes fell outside the acceptable range, causing users to feel uneasy.

[0098] Therefore, in this embodiment, by performing the pad sync control shown in Figure 5, it is possible to reduce the user's discomfort when a video generated by the cloud server 10 in response to an operation on the terminal 12 is displayed on the terminal 12.

[0099] Figure 5 is an explanatory diagram illustrating an example of pad sync control performed in the cloud gaming system 1 according to this embodiment.

[0100] As shown in Figure 5, the pad sink control according to this embodiment performs two types of control: first control and second control.

[0101] In the first control, the transmission timing of the pad data P is controlled so that the value indicating DiffPadVsp (see Figure 2) approaches a predetermined value T1 (for example, a value T1 that is 1.5 times the transmission period of the pad data P).

[0102] Here, for example, the transmission period T of the pad data P is determined by PD control, where the value T1 is the target value, the value of DiffPadVsp is the current value, and the transmission period T of the pad data P is the manipulated variable. This is shown in Figure 5 as T = PD(DiffPadVsp - T1). For example, the larger the value of (DiffPadVsp - T1), the smaller the value of T is controlled to be. Also, if the value of DiffPadVsp is greater than the value of T1, the value of T is controlled to be smaller, and if the value of DiffPadVsp is less than the value of T1, the value of T is controlled to be larger.

[0103] Furthermore, the pad data P may be transmitted at a determined transmission cycle T. For example, the pad data P may be transmitted at a time corresponding to the determined transmission cycle T that has elapsed since the transmission timing of the most recent pad data P.

[0104] Then, for example, when it is determined that the value of DiffPadVsp has stabilized, the pad sync control transitions from the first control to the second control. For example, when a predetermined condition is met, such as when a value indicating the fluctuation of the value of DiffPadVsp (for example, the absolute value of -T1 of DiffPadVsp) remains below a predetermined threshold Th3 for a predetermined time t2 or longer, the pad sync control transitions from the first control to the second control.

[0105] In the second control, the value T1, which was a fixed value in the first control, is made variable so that the value representing the time from the timing of receiving pad data P in the cloud server 10 to the timing of the start of generation of the play image based on said pad data P (PRecvGStartTime) approaches a predetermined value T2 (for example, 1 millisecond). In this case, the value T1 will be expressed as T1_adj.

[0106] Here, for example, the value T1_adj is determined by PD control, where the value T2 is the target value, the value PRecvGStartTime is the current value, and the value T1_adj is the manipulated variable. This is shown in Figure 5 as T1_adj = PD(PRecvGStartTime - T2). Then, for example, the transmission period T of the pad data P is determined by PD control, where the value T1_adj is the target value, the value DiffPadVsp is the current value, and the transmission period T of the pad data P is the manipulated variable. This is shown in Figure 5 as T = PD(DiffPadVsp - T1_adj). In summary, T = PD(DiffPadVsp - PD(PRecvGStartTime - T2)).

[0107] Furthermore, the pad data P may be transmitted at a transmission cycle T determined in the manner described above. For example, the pad data P may be transmitted at a time corresponding to the determined transmission cycle T that has elapsed since the transmission timing of the most recent pad data P.

[0108] For example, when it is determined that the value of DiffPadVsp has become unstable, the pad sync control switches from the second control to the first control. For example, when a predetermined condition is met, such as when a value indicating fluctuation in the value of DiffPadVsp (for example, the absolute value of DiffPadVsp-T1) is greater than or equal to a predetermined threshold Th3, the pad sync control switches from the second control to the first control.

[0109] Pad sync control may be performed at predetermined timings. For example, pad sync control may be performed at a predetermined period (e.g., a 16-millisecond period). Bandwidth fluctuation tracking control may also be performed in response to the occurrence of a predetermined event (e.g., reception of the last segment of image data).

[0110] As described above, in this embodiment, low latency can be achieved by synchronizing the timing of pad data P transmission with the timing of play image generation through pad sync control. In this way, according to this embodiment, the user's discomfort when a video generated by the cloud server 10 in response to operation on the terminal 12 is displayed on the terminal 12 is reduced.

[0111] Furthermore, in this embodiment, PRecvGStartTime can be stabilized by executing the first and second controls described above.

[0112] The following will further describe the functions of the cloud gaming system 1 according to this embodiment and the processes performed by the cloud gaming system 1 according to this embodiment, focusing on the points explained above.

[0113] Figure 6 is a functional block diagram showing an example of the functions implemented in the cloud gaming system 1 according to this embodiment. Note that not all of the functions shown in Figure 6 are required to be implemented in the cloud gaming system 1 according to this embodiment, and other functions may also be implemented.

[0114] As shown in Figure 6, the cloud server 10 according to this embodiment functionally includes, for example, a server-side control data storage unit 20, an operation data receiving unit 22, a frame image generation unit 24, a VSP transmission unit 26, an encoding processing execution unit 28, an image data transmission unit 30, and a server-side traffic control unit 32. In this embodiment, the cloud server 10 plays the role of a video transmission device that sequentially transmits image data representing frame images that constitute a video.

[0115] The server-side control data storage unit 20 is mainly implemented by the storage unit 10b. The operation data receiving unit 22, VSP transmission unit 26, and image data transmission unit 30 are mainly implemented by the communication unit 10c. The frame image generation unit 24 and server-side traffic control unit 32 are mainly implemented by the processor 10a. The encoding processing execution unit 28 is mainly implemented by the processor 10a and the encoder / decoder unit 10d.

[0116] The above functions may be implemented by executing a program containing the commands corresponding to the above functions, which is installed on the cloud server 10 (a computer), on the processor 10a. This program may be supplied to the cloud server 10 via a computer-readable information storage medium such as an optical disc, magnetic disc, magnetic tape, magneto-optical disc, or flash memory, or via the internet.

[0117] As shown in Figure 6, the terminal 12 according to this embodiment functionally includes, for example, a terminal-side control data storage unit 40, an operation data generation unit 42, an operation data transmission unit 44, a VSP receiving unit 46, an image data receiving unit 48, a decoding processing execution unit 50, a frame image display control unit 52, a terminal-side traffic control unit 54, and a transmission timing control unit 56. In this embodiment, the terminal 12 plays the role of a video receiving device that sequentially receives image data representing frame images that constitute a video.

[0118] The terminal-side control data storage unit 40 is mainly implemented by the storage unit 12b. The operation data generation unit 42 is mainly implemented by the processor 10a and the operation unit 12e. The operation data transmission unit 44, VSP reception unit 46, and image data reception unit 48 are mainly implemented by the communication unit 12c. The decode processing execution unit 50 is mainly implemented by the processor 10a and the encoder / decoder unit 12h. The frame image display control unit 52 is mainly implemented by the processor 12a and the display unit 12d. The terminal-side traffic control unit 54 and the transmission timing control unit 56 are mainly implemented by the processor 10a.

[0119] The above functions may be implemented by executing a program containing commands corresponding to the above functions, which is installed on the computer terminal 12, on the processor 12a. This program may be supplied to the terminal 12, for example, via a computer-readable information storage medium such as an optical disc, magnetic disc, magnetic tape, magneto-optical disc, or flash memory, or via the internet.

[0120] In this embodiment, the server-side control data storage unit 20 stores, for example, control data indicating the bitrate value B described above.

[0121] In this embodiment, the operation data receiving unit 22 receives, for example, operation data (e.g., the pad data P described above) corresponding to user input operations. The operation data receiving unit 22 also receives, for example, operation data corresponding to input operations during gameplay.

[0122] In this embodiment, the frame image generation unit 24 generates a frame image based on, for example, the operation data received by the operation data receiving unit 22. The frame image generation unit 24 generates a play image representing the gameplay status of the game being played by the user, for example, based on the pad data P.

[0123] In this embodiment, the VSP transmission unit 26 transmits a VSP, which is a packet associated with operation data, to the terminal 12 in response to the start of frame image generation based on operation data. The VSP transmission unit 26 transmits the VSP, for example, at the timing when frame image generation begins.

[0124] In this embodiment, the encoding processing execution unit 28 generates image data representing a frame image by encoding the frame image generated by the frame image generation unit 24. The encoding processing execution unit 28 may determine a compression ratio such that the value indicating the bitrate of the transmitted video image becomes the value B indicated by the control data stored in the server-side control data storage unit 20. The encoding processing execution unit 28 may then generate image data by encoding the frame image with the determined compression ratio.

[0125] Furthermore, the encoding processing execution unit 28 may generate the interval data described above. The encoding processing execution unit 28 may also associate the generated interval data with the generated image data.

[0126] In this embodiment, the image data transmission unit 30 transmits, for example, the image data generated by the encoding processing execution unit 28 to the terminal 12. The image data transmission unit 30 may also transmit the image data associated with the interval data described above to the terminal 12.

[0127] In this embodiment, the server-side traffic control unit 32 controls, for example, the data size of the image data to be transmitted by the image data transmission unit 30.

[0128] As described above, the server-side traffic control unit 32 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the TT value described above, which indicates the time required for the terminal 12 to receive the image data.

[0129] Furthermore, as described above, the server-side traffic control unit 32 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the packet reception time (PadVspRTT) described above.

[0130] In this embodiment, the terminal-side control data storage unit 40 stores, for example, control data indicating the bitrate value B described above.

[0131] In this embodiment, the operation data generation unit 42 generates the above-mentioned operation data in response to user input operations. The operation data generation unit 42 may also generate operation data associated with control data stored in the terminal-side control data storage unit 40.

[0132] In this embodiment, the operation data transmission unit 44 transmits, for example, the operation data generated by the operation data generation unit 42 to the cloud server 10.

[0133] The operation data transmission unit 44 may transmit operation data associated with control data. In this case, the server-side traffic control unit 32 may acquire the control data associated with the operation data received by the operation data reception unit 22. The server-side traffic control unit 32 may then update the control data stored in the server-side control data storage unit 20 with the acquired control data.

[0134] Furthermore, the operation data transmission unit 44 does not need to transmit control data in association with operation data; it may transmit control data to the cloud server 10 independently of operation data. The operation data receiving unit 22 may then receive the control data transmitted in this manner. In this case, the server-side traffic control unit 32 may update the control data stored in the server-side control data storage unit 20 with the control data received by the operation data receiving unit 22.

[0135] In this embodiment, the VSP receiving unit 46 receives, for example, a VSP packet associated with the operation data, which is transmitted from the cloud server 10 in response to the start of generating a frame image based on the operation data, from the cloud server 10.

[0136] In this embodiment, the image data receiving unit 48 receives image data transmitted from, for example, the cloud server 10. As described above, the image data receiving unit 48 may also receive image data associated with interval data.

[0137] In this embodiment, the decoding processing unit 50 generates a frame image (e.g., a play image) represented by the image data by decoding the image data received by the image data receiving unit 48.

[0138] In this embodiment, the frame image display control unit 52 causes the frame image (e.g., play image) generated by the decoding processing execution unit 50 to be displayed on the display unit 12d.

[0139] In this embodiment, the terminal-side traffic control unit 54 controls, for example, the data size of the image data to be transmitted by the cloud server 10.

[0140] As described above, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the TT value described above, which indicates the time required to receive the image data transmitted from the cloud server 10.

[0141] Furthermore, as described above, the terminal-side traffic control unit 54 may determine the packet reception time (PadVspRTT), which is the time from when the terminal 12 transmits operation data to when it receives the VSP associated with said operation data. Based on the packet reception time, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30.

[0142] The terminal-side traffic control unit 54 may perform the bandwidth fluctuation tracking control described above. The terminal-side traffic control unit 54 may also update the control data stored in the terminal-side control data storage unit 40 with control data that indicates the value B determined by performing the bandwidth fluctuation tracking control.

[0143] Furthermore, as explained in the bandwidth fluctuation tracking control described above, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted based on the packet reception time for the most recent VSP and the packet reception time for at least one previous VSP. Specifically, for example, control such as D=PD(FilPadVspRTT-EstBtmLatency) shown in Figure 4 may be performed.

[0144] Furthermore, the terminal-side traffic control unit 54 may perform the packet congestion reduction control described above. The terminal-side traffic control unit 54 may also update the control data stored in the terminal-side control data storage unit 40 with control data that indicates the value B which is updated by performing the packet congestion reduction control.

[0145] Furthermore, as explained in the packet congestion reduction control above, the terminal-side traffic control unit 54 may control the image data transmission unit 30 to reduce the data size of the image data to be transmitted if it determines that VSP reception failures are continuing based on predetermined conditions. For example, as described above, the packet congestion reduction control described above may be executed when the number of VSPs M received in the most recent predetermined time t1 is smaller than a predetermined threshold Th2.

[0146] Furthermore, the terminal-side traffic control unit 54 may maintain a bitrate control mode flag indicating the bitrate control mode. For example, when the value of the bitrate control mode flag is 0, bandwidth fluctuation tracking control may be performed, and when the value of the bitrate control mode flag is 1, packet congestion reduction control may be performed.

[0147] In this embodiment, the transmission timing control unit 56 controls, for example, the transmission timing of operation data.

[0148] The transmission timing control unit 56 may control the time from the timing of receiving the VSP until the next operation data is transmitted. For example, the transmission timing control unit 56 may perform a first control to control the timing at which operation data is transmitted so that the time from the timing of receiving the VSP until the next operation data is transmitted achieves a first objective. For example, in the above example, the first objective corresponds to satisfying a predetermined condition such as the state in which the value indicating the change in the value of DiffPadVsp is less than a predetermined threshold Th3 for a predetermined time t2 or more.

[0149] Furthermore, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data. For example, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data associated with the image data received by the image data receiving unit 48.

[0150] Here, for example, the transmission timing control unit 56 may perform a second control to control the timing at which operation data is transmitted so that the time indicated by the interval data meets the second objective. For example, in the above example, the second objective is that the value of PRecvGStartTime becomes a predetermined value T2 (for example, 1 millisecond).

[0151] Furthermore, as described above, the transmission timing control unit 56 may start the second control in response to the achievement of the first objective in the first control.

[0152] Furthermore, as described above, the transmission timing control unit 56 may control the transmission period T of the operation data to be transmitted based on the interval data.

[0153] Furthermore, the image data transmission unit 30 does not need to transmit image data associated with interval data. The image data transmission unit 30 may transmit interval data to the cloud server 10 independently of the image data. The image data receiving unit 48 may then receive the interval data transmitted in this manner. The transmission timing control unit 56 may then control the timing at which the next operation data is transmitted based on the interval data received from the cloud server 10 in this manner.

[0154] Furthermore, the transmission timing control unit 56 may hold a pad sync control mode flag indicating the mode of pad sync control. For example, the first control may be executed when the value of the pad sync control mode flag is 0, and the second control may be executed when the value of the pad sync control mode flag is 1.

[0155] The transmission timing control unit 56 may, for example, output a transmission command to the operation data transmission unit 44 at a transmission period T determined as described above. The operation data transmission unit 44 may then send operation data to the cloud server 10 in response to receiving the transmission command.

[0156] Below, an example of the processing flow in the bandwidth fluctuation tracking control performed in terminal 12 according to this embodiment will be explained with reference to the flowchart illustrated in Figure 7.

[0157] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to bandwidth fluctuation tracking control arrives (S101).

[0158] When a predetermined execution timing arrives, the terminal-side traffic control unit 54 identifies the TT value for the latest image data received by the image data receiving unit 48 (S102).

[0159] Then, the terminal-side traffic control unit 54 determines the above-mentioned value R based on the TT identified in the process shown in S102 (S103).

[0160] Then, the terminal-side traffic control unit 54 identifies the latest PadVspRTT value (S104).

[0161] Then, the terminal-side traffic control unit 54 identifies the absolute value V of the difference between the latest PadVspRTT value and the previous PadVspRTT value (S105).

[0162] Then, the terminal-side traffic control unit 54 checks whether the state in which the absolute value V is less than Th1 has occurred N times in a row (S106).

[0163] If it is confirmed that the absolute value V is less than Th1 for N consecutive times (S106:Y), the terminal-side traffic control unit 54 updates the value of EstBtmLatency (S107).

[0164] If it is confirmed that the state in which the absolute value V is less than Th1 has not occurred N times in a row (S106:N), or if the process shown in S107 is executed, the terminal-side traffic control unit 54 determines the value of FilPadVspRTT (S108).

[0165] Then, the terminal-side traffic control unit 54 determines the above-mentioned value D based on the value of FilPadVspRTT identified in the process shown in S108 and the latest value of EstBtmLatency (S109).

[0166] Then, the terminal-side traffic control unit 54 determines value B based on the value R identified in the process shown in S103 and the value D identified in the process shown in S109 (S110).

[0167] Then, the terminal-side traffic control unit 54 updates the control data stored in the terminal-side control data storage unit 40 so that the set value B becomes the value B identified in the process shown in S109 (S111), and returns to the process shown in S101.

[0168] Below, an example of the processing flow in packet congestion reduction control performed in terminal 12 according to this embodiment will be explained with reference to the flowchart illustrated in Figure 8.

[0169] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to packet congestion reduction control arrives (S201).

[0170] When a predetermined execution timing arrives, the terminal-side traffic control unit 54 identifies the bitrate value B indicated by the control data stored in the terminal-side control data storage unit 40 (S202).

[0171] Then, the terminal-side traffic control unit 54 checks whether the value B identified in the process shown in S202 is less than the lower limit b1 (S203).

[0172] If it is confirmed that value B is less than the lower limit b1 (S203:Y), the process returns to the procedure shown in S201.

[0173] If it is confirmed that value B is not less than the lower limit b1 (S203:N), the terminal-side traffic control unit 54 identifies a new value B obtained by multiplying the value B identified in the process shown in S202 by a predetermined ratio r (r is less than 1) (S204).

[0174] Then, the terminal-side traffic control unit 54 updates the control data stored in the terminal-side control data storage unit 40 (S205) so that the set value B becomes the value B identified in the process shown in S204, and returns to the process shown in S201.

[0175] Below, an example of the bitrate control mode switching process performed in terminal 12 according to this embodiment will be explained with reference to the flowchart illustrated in Figure 9.

[0176] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to the bitrate control mode switching process arrives (S301).

[0177] Here, for example, the execution timing may be set to occur at a predetermined period (e.g., the transmission period T of operation data). Alternatively, the execution timing may be set to occur in response to the occurrence of a predetermined event (e.g., the transmission of operation data).

[0178] When a predetermined execution timing arrives, the terminal-side traffic control unit 54 identifies the number of VSPs M received in the most recent predetermined time t1 (S302).

[0179] Then, the terminal-side traffic control unit 54 checks the value of the current bitrate control mode flag (S303).

[0180] If the value of the confirmed bitrate control mode flag is 0, the terminal-side traffic control unit 54 checks whether the number of received signals M identified in the process shown in S302 is less than a predetermined threshold Th2 (S304).

[0181] If the number of received signals M is equal to or greater than the predetermined threshold Th2 (S304:N), the process returns to the one shown in S301.

[0182] If the number of received signals M is less than a predetermined threshold Th2 (S304:Y), the terminal-side traffic control unit 54 changes the value of the held bitrate control mode flag to 1 (S305) and returns to the process shown in S301.

[0183] If the value of the current bitrate control mode flag confirmed in the process shown in S303 is 1, the terminal-side traffic control unit 54 checks whether the number of received signals M identified in the process shown in S302 is greater than or equal to a predetermined threshold Th2 (S306).

[0184] If the number of received signals M is less than a predetermined threshold Th2 (S306:N), the process returns to the one shown in S301.

[0185] If the number of received signals M is equal to or greater than a predetermined threshold Th2 (S306:Y), the terminal-side traffic control unit 54 changes the value of the held bitrate control mode flag to 0 (S307) and returns to the process shown in S301.

[0186] Below, an example of the processing flow in pad sync control performed in terminal 12 according to this embodiment will be explained with reference to the flowchart illustrated in Figure 10.

[0187] First, the transmission timing control unit 56 waits until a predetermined execution timing related to pad sync control arrives (S401).

[0188] When a predetermined execution timing arrives, the transmission timing control unit 56 identifies the latest value of DiffPadVsp (S402).

[0189] Then, the transmission timing control unit 56 checks the value of the current pad sync control mode flag (S403).

[0190] If the value of the confirmed pad sync control mode flag is 0, the transmission timing control unit 56 determines a new transmission period T such that the value of DiffPadVsp identified in the process shown in S402 becomes T1 (S404).

[0191] Then, the transmission timing control unit 56 determines the absolute value V of the difference between the latest DiffPadVsp value and the value T1 (S405).

[0192] Then, the transmission timing control unit 56 identifies the duration for which the absolute value V remains below a predetermined threshold Th3 (S406).

[0193] Then, the transmission timing control unit 56 checks whether the time specified by the process shown in S406 has reached a predetermined time (S407).

[0194] If it is not confirmed that the predetermined time has been reached (S407:N), the process returns to the procedure shown in S401.

[0195] If it is confirmed that the predetermined time has been reached (S407:Y), the transmission timing control unit 56 changes the value of the held pad sync control mode flag to 1 (S408) and returns to the process shown in S401.

[0196] If the value of the current pad sync control mode flag, confirmed in the process shown in S403, is 1, the transmission timing control unit 56 identifies the value of the interval data associated with the latest image data (S409).

[0197] Then, the transmission timing control unit 56 determines a new transmission period T based on the value of the interval data identified in the process shown in S409 (S410).

[0198] Then, the transmission timing control unit 56 identifies the absolute value V of the difference between the latest DiffPadVsp value and the value T1 (S411).

[0199] Then, the transmission timing control unit 56 checks whether the identified absolute value V is greater than or equal to a predetermined threshold Th3 (S412).

[0200] If the absolute value V is not greater than or equal to the predetermined threshold Th3 (S412:N), the process returns to the procedure shown in S401.

[0201] If the absolute value V is greater than or equal to a predetermined threshold Th3 (S412:Y), the transmission timing control unit 56 changes the value of the held pad sink control mode flag to 0 (S413) and returns to the process shown in S401.

[0202] However, the present invention is not limited to the embodiments described above.

[0203] For example, interval data may be associated with the VSP rather than with the image data.

[0204] Alternatively, the frame image generation unit 24 may generate frame images with a data size corresponding to the bitrate.

[0205] Furthermore, the scope of application of the present invention is not limited to cloud gaming system 1.

[0206] Furthermore, the scope of application of the present invention is not limited to computer networks 14 including mobile communication systems such as 4G and 5G. The present invention can also be applied to computer networks 14 where wireless communication is performed via Wi-Fi (registered trademark), rather than wireless communication via mobile communication systems such as 4G and 5G.

[0207] Furthermore, the specific strings and numbers mentioned above, as well as those in the drawings, are examples only and are not limited to these. [Explanation of symbols]

[0208] 1 Cloud gaming system, 10 Cloud server, 10a Processor, 10b Memory unit, 10c Communication unit, 10d Encoder / decoder unit, 12 Terminal, 12a Processor, 12b Memory unit, 12c Communication unit, 12d Display unit, 12e Operation unit, 12f Sensor unit, 12g Audio output unit, 12h Encoder / decoder unit, 14 Computer network, 20 Server-side control data storage unit, 22 Operation data reception unit, 24 Frame image generation unit, 26 VSP transmission unit, 28 Encoding processing execution unit, 30 Image data transmission unit, 32 Server-side traffic control unit, 40 Terminal-side control data storage unit, 42 Operation data generation unit, 44 Operation data transmission unit, 46 VSP reception unit, 48 Image data reception unit, 50 Decode processing execution unit, 52 Frame image display control unit, 54 Terminal-side traffic control unit, 56 Transmission timing control unit.

Claims

1. A video transmission device that sequentially transmits image data representing frame images constituting a video to a video receiving device, An operation data receiving unit receives operation data corresponding to user input operations from the video receiving device, An image generation unit that generates the frame image based on the operation data, A packet transmission unit that transmits packets associated with the aforementioned operation data to the video receiving device, An encoding processing execution unit that generates image data representing the frame image by encoding the frame image, An image data transmission unit that transmits the image data to the video receiving device, The system includes a data size control unit that controls the data size of image data to be transmitted by the image data transmission unit based on a packet reception time, which is the time from when the video receiving device transmits the operation data to when the packet associated with the operation data is received. The data size control unit controls the data size based on the packet reception time for the most recent packet and the packet reception time for at least one packet prior to the most recent one. A video transmission device characterized by the following features.

2. A video transmission device sequentially transmits image data representing frame images that constitute a video to a video receiving device, and an operation data reception step in which the video transmission device receives operation data corresponding to user input operations from the video receiving device, An image generation step of generating the frame image based on the operation data, A packet transmission step of transmitting a packet associated with the aforementioned operation data to the video receiving device, The encoding process execution step involves encoding the aforementioned frame image to generate image data representing the frame image, An image data transmission step of transmitting the image data to the video receiving device, The process includes a data size control step that controls the data size of the image data to be transmitted in the image data transmission step, based on a packet reception time, which is the time from when the video receiving device transmits the operation data to when it receives the packet associated with the operation data. In the data size control step, the data size is controlled based on the packet reception time for the most recent packet and the packet reception time for at least one packet prior to the most recent one. A control method characterized by the following:

3. A computer sequentially transmits image data representing the frame images that make up a video to a video receiving device. Operation data reception procedure for receiving operation data from the video receiving device in response to user input operations, Image generation procedure for generating the frame image based on the aforementioned operation data, A packet transmission procedure for transmitting a packet associated with the aforementioned operation data to the video receiving device, An encoding process execution procedure that generates image data representing the frame image by encoding the aforementioned frame image, Image data transmission procedure for transmitting the aforementioned image data to the video receiving device, Based on the packet reception time, which is the time from when the video receiving device transmits the operation data to when it receives the packet associated with the operation data, the image data transmission procedure executes a data size control procedure to control the data size of the image data to be transmitted from now on. In the data size control procedure, the data size is controlled based on the packet reception time for the most recent packet and the packet reception time for at least one packet prior to the most recent one. A program characterized by the following features.

Images