Moving image transmitter, control method, and program

Abstract

To provide a moving image receiver, a moving image transmitter, a moving image transmission / reception system, a control method, and a program that can generate image data with a data size taking into account of low latency.SOLUTION: An operation data transmission unit 44 transmits operation data corresponding to user's input operation to a cloud server 10. A VSP receiving unit 46 receives a packet transmitted from the cloud server 10 in response to the start of generation of a frame image based on the operation data, the packet being associated with the operation data. A terminal-side traffic control unit 54 controls a data size of image data to be transmitted by the cloud server 10 on the basis of a packet reception time being a time from timing when the operation data is transmitted to timing when the packet associated with the operation data is received.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a moving image receiving device, a moving image transmitting device, a moving image transmission / reception system, a control method, and a program.

Background Art

[0002] In the technology of cloud gaming services that has been attracting attention in recent years, operation data corresponding to a user's input operation on a terminal is wirelessly transmitted from the terminal to a cloud server. Then, in the cloud server, a frame image representing the play state of the game is generated based on the operation data. Then, image data obtained by encoding the frame image is wirelessly transmitted from the cloud server to the terminal, and a frame image obtained by decoding the image data is displayed on the terminal. By repeatedly executing this series of processes, a moving image representing the play state of the game is displayed on the terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In the cloud gaming service described above, it is desirable that the moving image displayed on the terminal has as high a picture quality as possible. Therefore, it is desirable that the data size of the image data generated based on the frame images constituting the moving image is as large as possible.

[0004] However, in wireless communication that requires the above-mentioned immediacy, in order to reduce the user's sense of discomfort in the operation, it is important that each frame image is displayed on the terminal without delay and with low latency even if the data size of the image data is reduced. In a situation where moving image transmission is performed using a mobile communication system such as the fourth-generation mobile communication system (4G) or the fifth-generation mobile communication system (5G) in which the bandwidth variation is large and latency is likely to occur, it is particularly necessary to pay attention to ensuring that the frame image is displayed on the terminal with low latency.

[0005] Note that this is generally applicable not only to the situation where a cloud gaming service is provided, but also to the situation where a moving image is transmitted from a moving image transmission device corresponding to the above-described cloud server.

[0006] The present invention has been made in view of the above circumstances, and one of its objects is to provide a moving image receiving device, a moving image transmitting device, a moving image transmission / reception system, a control method, and a program that can generate image data of an appropriate data size considering delay.

Means for Solving the Problems

[0007] In order to solve the above problems, a moving image receiving device according to the present invention is a moving image receiving device that sequentially receives image data representing frame images constituting a moving image from a moving image transmitting device, and includes an operation data transmitting unit that transmits operation data corresponding to a user's input operation to the moving image transmitting device, a packet receiving unit that receives a packet associated with the operation data, which is transmitted from the moving image transmitting device in response to the start of generation of the frame image based on the operation data, and a data size control unit that controls the data size of image data to be transmitted from the moving image transmitting device hereinafter based on a packet reception time that is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received.

[0008] In one aspect of the present invention, the data size control unit controls the data size based on the time required for receiving the image data transmitted from the moving image transmitting device.

[0009] Also, in one aspect of the present invention, the data size control unit controls the data size based on the packet reception time in the reception of the latest packet and the packet reception time in the reception of at least one packet before the latest one.

[0010] Also, in one aspect of the present invention, when it is determined that the reception failure of the packet continues based on a predetermined condition, the data size control unit controls the data size to become smaller.

[0011] Further, a moving image transmission apparatus according to the present invention is a moving image transmission apparatus that sequentially transmits image data representing frame images constituting a moving image to a moving image reception apparatus, and includes an operation data reception unit that receives operation data corresponding to a user's input operation from the moving image reception apparatus, an image generation unit that generates the frame image based on the operation data, a packet transmission unit that transmits a packet associated with the operation data to the moving image reception apparatus in response to the start of generation of the frame image, an encoding process execution unit that generates image data representing the frame image by encoding the frame image, an image data transmission unit that transmits the image data to the moving image reception apparatus, and a data size control unit that controls the data size of the image data to be transmitted next by the image data transmission unit based on the time from the timing when the moving image reception apparatus transmits the operation data to the timing when the packet associated with the operation data is received.

[0012] In addition, the moving image transmission / reception system according to the present invention is a moving image transmission / reception system including a moving image transmission device that sequentially transmits image data representing frame images constituting a moving image, and a moving image reception device that sequentially receives the image data. The moving image reception device includes an operation data transmission unit that transmits operation data corresponding to a user's input operation to the moving image transmission device, a packet reception unit that receives a packet associated with the operation data, which is transmitted from the moving image transmission device in response to the start of generation of a frame image based on the operation data, and a control data transmission unit that transmits control data for controlling the data size of the image data to be transmitted from the moving image transmission device hereafter, based on a packet reception time, which is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received. The moving image transmission device includes an operation data reception unit that receives the operation data from the moving image reception device, an image generation unit that generates the frame image based on the operation data, a packet transmission unit that transmits a packet associated with the operation data to the moving image reception device in response to the start of generation of the frame image, an encoding process execution unit that generates image data representing the frame image by encoding the frame image, an image data transmission unit that transmits the image data to the moving image reception device, a control data reception unit that receives the control data, and a data size control unit that controls the data size of the image data to be transmitted from the image data transmission unit hereafter, based on the control data.

[0013] In addition, the control method according to the present invention includes: a moving image receiving device that sequentially receives image data representing frame images constituting a moving image from a moving image transmitting device, and transmits operation data corresponding to a user's input operation to the moving image transmitting device; the moving image receiving device receiving a packet associated with the operation data, which is transmitted from the moving image transmitting device in response to the start of generation of the frame image based on the operation data; and the moving image receiving device controlling the data size of image data to be transmitted by the moving image transmitting device hereafter based on a packet reception time, which is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received.

[0014] In addition, another control method according to the present invention includes: a moving image transmitting device that sequentially transmits image data representing frame images constituting a moving image to a moving image receiving device, and receives operation data corresponding to a user's input operation from the moving image receiving device; the moving image transmitting device generating the frame image based on the operation data; the moving image transmitting device transmitting a packet associated with the operation data to the moving image receiving device in response to the start of generation of the frame image; the moving image transmitting device generating image data representing the frame image by encoding the frame image; the moving image transmitting device transmitting the image data to the moving image receiving device; and the moving image transmitting device controlling the data size of image data to be transmitted by the moving image transmitting device hereafter based on the time from the timing when the moving image receiving device transmits the operation data to the timing when the moving image receiving device receives the packet associated with the operation data.

[0015] Further, the program according to the present invention causes a computer that sequentially receives image data representing frame images constituting a moving image from a moving image transmission device to execute a procedure of transmitting operation data corresponding to a user's input operation to the moving image transmission device, a procedure of receiving a packet associated with the operation data, which is transmitted from the moving image transmission device in response to the start of generation of the frame image based on the operation data, and a procedure of controlling the data size of image data to be transmitted from the moving image transmission device hereafter based on a packet reception time, which is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received.

[0016] Further, another program according to the present invention causes a computer that sequentially transmits image data representing frame images constituting a moving image to a moving image reception device to execute a procedure of receiving operation data corresponding to a user's input operation from the moving image reception device, a procedure of generating the frame image based on the operation data, a procedure of transmitting a packet associated with the operation data to the moving image reception device in response to the start of generation of the frame image, a procedure of generating image data representing the frame image by encoding the frame image, a procedure of transmitting the image data to the moving image reception device, and a procedure of controlling the data size of image data to be transmitted from the computer hereafter based on the time from the timing when the moving image reception device transmits the operation data to the timing when the moving image reception device receives the packet associated with the operation data.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] FIG. 1 is a diagram showing an example of the overall configuration of a cloud gaming system 1 according to an embodiment of the present invention. As shown in FIG. 1, the cloud gaming system 1 according to the present embodiment includes a cloud server 10 and a terminal 12, both of which are centered around a computer.

[0020] The cloud server 10 and the terminal 12 are connected to a computer network 14 including a mobile communication system such as a fourth-generation mobile communication system (4G) or a fifth-generation mobile communication system (5G), or the Internet. For example, the cloud server 10 is connected to the Internet, and the terminal 12 is connected to a mobile communication system such as 4G or 5G. The cloud server 10 and the terminal 12 can communicate with each other via the computer network 14.

[0021] The cloud server 10 according to the present embodiment is, for example, a server computer that executes a game program related to a cloud gaming service.

[0022] As shown in FIG. 1, the cloud server 10 includes, for example, a processor 10a, a storage unit 10b, a communication unit 10c, and an encoder / decoder unit 10d.

[0023] The processor 10a is a program control device such as a CPU, for example, and executes various information processes according to a program stored in the storage unit 10b. The processor 10a according to the present embodiment also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on a graphics command and data supplied from the CPU.

[0024] The storage unit 10b is, for example, a storage element such as a ROM or a RAM, or a solid state drive (SSD). Programs executed by the processor 10a and the like are stored in the storage unit 10b. In addition, an area of a frame buffer in which an image is drawn by the GPU included in the processor 10a is secured in the storage unit 10b according to the present embodiment.

[0025] The communication unit 10c is a communication interface for exchanging data with a computer such as the terminal 12 via the computer network 14, for example.

[0026] The encoder / decoder unit 10d includes, for example, an encoder and a decoder. The encoder generates image data representing the input image by encoding the input image. The decoder decodes the input image data and outputs the image represented by the image data.

[0027] The terminal 12 according to this embodiment is a computer such as a smartphone or a tablet terminal used by a user who uses, for example, a cloud gaming service. Note that the terminal 12 may be an electronic device capable of communicating with the cloud server 10 via a communication dongle, such as a TV including a communication dongle.

[0028] As shown in FIG. 1, the terminal 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, a display unit 12d, an operation unit 12e, a sensor unit 12f, an audio output unit 12g, and an encoder / decoder unit 12h.

[0029] The processor 12a is a program control device such as a CPU, for example, and executes various information processes according to the program stored in the storage unit 12b.

[0030] The storage unit 12b is, for example, a storage element such as a ROM or a RAM, or a solid state drive (SSD). Programs executed by the processor 12a and the like are stored in the storage unit 12b.

[0031] The communication unit 12c is a communication interface for exchanging data with a computer such as the cloud server 10 via, for example, a computer network 14.

[0032] The display unit 12d is a display device such as a liquid crystal display or an organic EL display, for example.

[0033] The operation unit 12e is an operation member for performing an operation input to the processor 12a, for example.

[0034] The sensor unit 12f is a sensor such as a motion sensor capable of detecting, for example, acceleration and angular velocity.

[0035] The voice output unit 12g is a voice output device such as a speaker that outputs, for example, the voice represented by voice data.

[0036] The encoder / decoder unit 12h includes, for example, an encoder and a decoder. The encoder generates image data representing the input image by encoding the input image. The decoder decodes the input image data and outputs the image represented by the image data.

[0037] Note that the terminal 12 may be provided with a touch panel. In this case, the touch panel serves both as the above-described display unit 12d and operation unit 12e.

[0038] In the present embodiment, when the user performs an input operation on the operation unit 12e during the play of a game in the cloud gaming service, the terminal 12 generates operation data corresponding to the input operation and transmits the operation data to the cloud server 10. Hereinafter, the operation data will be referred to as pad data P.

[0039] Then, the cloud server 10 executes game processing corresponding to the received pad data P. Then, based on the result of the game processing, the cloud server 10 generates a play image, which is a frame image representing the play state of the game, and draws the play image on the frame buffer of the cloud server 10. In the present embodiment, the game processing and the generation of the play image are repeatedly executed.

[0040] Then, the cloud server 10 acquires the play image drawn in the frame buffer and generates image data representing the play image by encoding the play image. Then, the cloud server 10 transmits the generated image data to the terminal 12. Then, the terminal 12 decodes the image data received from the cloud server 10 and causes the display unit 12d to display the play image generated by the decoding.

[0041] As described above, the cloud server 10 according to the present embodiment streams and distributes the moving image generated according to the game play situation to the terminal 12 used by the user who is playing the game.

[0042] FIG. 2 is a diagram schematically showing an example of the timing of processing occurring in the cloud server 10 according to the present embodiment.

[0043] In the present embodiment, for example, in the initial state, the terminal 12 transmits pad data P indicating the input operation received by the operation unit 12e at the timing to the cloud server 10 at a predetermined period (for example, a period of 4 milliseconds). The pad data P according to the present embodiment is associated with a sequence number representing the transmission order of the pad data P.

[0044] FIG. 2 shows the reception timing of the pad data P(0) to P(10) transmitted in this way in the cloud server 10. In FIG. 2, the sequence number associated with the received pad data P is shown as the number in parentheses in P(0) to P(10). Note that depending on the communication quality of the computer network 14, the cloud server 10 does not necessarily receive the pad data P at intervals of 4 milliseconds.

[0045] In the cloud server 10, generation of a play image is performed at a predetermined period (for example, a period of 16 milliseconds). At this time, the cloud server 10 generates a play image based on the input operation indicated by the pad data P (the latest pad data P) received most recently at the timing of starting the generation of the play image, and draws the generated play image in the frame buffer.

[0046] The period G shown in FIG. 2 represents the generation period of the play image. In the example of FIG. 2, it is assumed that a play image with the serial number m is generated in the period G(m,n) based on the pad data P(n) with the serial number n. That is, in FIG. 2, the generation period of the play image based on the pad data P(0) corresponds to the period shown as G(1,0). Also, the generation period of the play image based on the pad data P(4) corresponds to the period shown as G(2,4). Also, the generation period of the play image based on the pad data P(8) corresponds to the period shown as G(3,8).

[0047] Note that in this embodiment, the transmission period of the pad data P in the terminal 12 is different from the generation period of the play image in the cloud server 10. For example, in the initial state, the generation period of the play image is four times the transmission period of the pad data P. Therefore, not all of the pad data P received by the cloud server 10 is used for generating the play image. In the example of FIG. 2, the pad data P shown as P(1) to P(3), P(5) to P(7), P(9), and P(10) are not used for generating the play image.

[0048] Also, in the present embodiment, when the generation of the play image based on the pad data P is started, the cloud server 10 transmits a packet indicating that the generation of the play image has started to the terminal 12. Hereinafter, the packet will be referred to as a Video Sync Packet (VSP). In the present embodiment, the VSP transmitted at the timing when the generation of the play image based on the pad data P is started includes the sequence number (m as described above) of the play image and the sequence number (n as described above) of the pad data P. Thus, the VSP according to the present embodiment is associated with the play image on a one-to-one basis.

[0049] FIG. 2 shows that the VSP(1,0) is transmitted at the timing when the generation of the play image with the sequence number 1 is started. Also, it is shown that the VSP(2,4) is transmitted at the timing when the generation of the play image with the sequence number 2 is started. Also, it is shown that the VSP(3,8) is transmitted at the timing when the generation of the play image with the sequence number 3 is started.

[0050] Also, in the present embodiment, for example, when the drawing of the play image to the frame buffer is completed, the encoding of the play image and the transmission of the image data generated by the encoding are executed. Note that the frame buffer of the cloud server 10 according to the present embodiment is configured as a multi-buffer, and the drawing of the next play image can be performed in parallel with the encoding of the play image whose drawing has been completed.

[0051] The period S shown in FIG. 2 corresponds to the period during which the play image is encoded and the image data is transmitted. In the example of FIG. 2, it is assumed that during the period S(m,n), the encoding of the play image with the sequence number m generated based on the pad data P with the sequence number n and the transmission of the image data generated by the encoding are performed. That is, in FIG. 2, the period during which the encoding of the play image with the sequence number 1 and the transmission of the image data generated by the encoding are performed corresponds to the period shown as S(1,0). Also, the period during which the encoding of the play image with the sequence number 2 and the transmission of the image data generated by the encoding are performed corresponds to the period shown as S(2,4).

[0052] In the present embodiment, the image data generated by encoding the play image generated based on the pad data P is associated with the sequence number of the pad data P (n as described above) and the sequence number of the play image (m as described above).

[0053] Also, in the present embodiment, in the cloud server 10, the time from the reception timing of the pad data P to the generation start timing of the play image based on the pad data P is measured. Then, in the cloud server 10, interval data indicating the measured time is generated. And in the present embodiment, the interval data thus generated is associated with the image data generated based on the play image.

[0054] FIG. 3 is a diagram schematically showing an example of communication that occurs between the cloud server 10 and the terminal 12 from the transmission of the pad data P(4) with the sequence number 4 to the reception of the image data generated based on the pad data P(4). In the example of FIG. 3, the image data transmitted during the period shown as S(2,4) in FIG. 2 is shown as D(2,4).

[0055] The terminal 12 according to this embodiment specifies the packet reception time, which is the time from the timing when the pad data P is transmitted to the timing when the VSP associated with the pad data P is received. Hereinafter, as shown in FIG. 3, the packet reception time will also be expressed as PadVspRTT. The terminal 12 according to this embodiment specifies PadVspRTT, for example, based on the sequence number of the pad data P included in the VSP received from the cloud server 10.

[0056] Also, as shown in FIG. 3, the time from the transmission of the pad data P in the terminal 12 to the reception of the first segment of the image data generated based on the pad data P will be referred to as PadFirstFragRTT.

[0057] Also, the time from the transmission of the pad data P in the terminal 12 to the reception of the last segment of the image data generated based on the pad data P will be referred to as PadFrameRTT.

[0058] Also, the time from the reception of the first segment of the image data in the terminal 12 to the reception of the last segment of the image data will be referred to as TransferTime. Hereinafter, TransferTime will be expressed as TT.

[0059] The terminal 12 according to this embodiment specifies the value of TT based on the image data received from the cloud server 10. Here, for example, the terminal 12 may specify the value of PadFrameRTT and the value of PadFrameRTT based on the sequence number of the pad data P associated with the image data. Then, the terminal 12 may specify the value of TT by subtracting the value of PadFirstFragRTT from the value of PadFrameRTT.

[0060] In addition, in this embodiment, when a slice transfer method is adopted in which one play image is divided into a plurality of slices and encoding and transmission of image data are performed for each slice, TT may be specified in consideration of the number of slices per frame. For example, TT may be specified in consideration of the fact that a predetermined communication-free time occurs from the end of transmission of image data related to one slice to the start of transmission of image data related to the next slice. In this case, for example, a value obtained by subtracting the value of PadFirstFragRTT from the value obtained by subtracting the value of PadFirstFragRTT from the value of PadFrameRTT and further subtracting a value obtained by multiplying (the number of slices per frame - 1) by the above communication-free time may be specified as the value of TT.

[0061] Also, as shown in FIG. 3, the time from the reception timing of the pad data P in the cloud server 10 to the start timing of generating a play image based on the pad data P shall be referred to as PRecvGStartTime.

[0062] The terminal 12 according to this embodiment specifies PRecvGStartTime based on the interval data associated with the image data.

[0063] Also, as shown in FIG. 2, the time from the timing when the VSP in the terminal 12 is received to the timing when the next pad data P is transmitted shall be referred to as DiffPadVsp.

[0064] Hereinafter, bitrate control performed in the cloud gaming system 1 according to this embodiment will be described.

[0065] In the cloud gaming service provided by the cloud gaming system 1, it is desirable that the moving image displayed on the terminal 12 has as high a picture quality as possible. Therefore, it is desirable that the data size of the image data generated based on the play image constituting the moving image is as large as possible.

[0066] However, in wireless communication that requires immediacy like this embodiment, in order to reduce the user's sense of discomfort in operations, it is important that each play image be displayed on the terminal 12 smoothly with low latency even if the data size of the image data is reduced.

[0067] In addition, mobile communication systems such as 4G and 5G have situations such as variations in the quality of the wireless communication path (radio wave intensity), communication resources may not be allocated due to congestion, and handovers between base stations may occur due to movement. Therefore, in mobile communication systems such as 4G and 5G, bandwidth fluctuations are large and delays are likely to occur.

[0068] Also, in a communication environment using Wi-Fi (registered trademark), the communication quality does not vary much between the uplink (communication from the terminal 12 to the cloud server 10 in this embodiment) and the downlink (communication from the cloud server 10 to the terminal 12 in this embodiment). On the other hand, in a communication environment such as 4G or 5G, it often happens that the communication quality of one of the uplink and the downlink is good and the communication quality of the other is bad.

[0069] Therefore, in a situation where moving image transmission using a mobile communication system such as 4G or 5G is performed, it is necessary to pay particular attention to ensure that the frame image is displayed on the terminal 12 with low latency.

[0070] Therefore, in this embodiment, by performing the bitrate control shown in FIG. 4, image data with an appropriate data size considering latency is generated in the cloud server 10.

[0071] FIG. 4 is an explanatory diagram for explaining an example of the bitrate control performed in the cloud gaming system 1 according to this embodiment.

[0072] As shown in FIG. 4, in the bitrate control according to this embodiment, two types of control, i.e., bandwidth fluctuation tracking control and packet congestion reduction control, are performed.

[0073] In bandwidth variation tracking control, first, to prevent image data loss even if the communication environment deteriorates slightly, a margin for jitter is provided, and the bit rate is controlled so that the TT value of each frame approaches half of the reciprocal of the frame rate (FPS) of the transmitted moving image. For example, when the frame rate of the transmitted moving image is 60 fps, the bit rate in the communication of the moving image is controlled so that TT approaches about 8 milliseconds.

[0074] For example, the value R of the bit rate is determined by PID control with 1 / (FPS×2) as the target value, TT as the current value, and the value R of the bit rate as the manipulated variable.

[0075] This is shown in FIG. 4 as R = PID(TT-(1 / (FPS×2))). For example, the larger the value of TT-(1 / (FPS×2)), the smaller the value of R is controlled. Also, when the value of TT is larger than the value of 1 / (FPS×2), the value of R is controlled to be smaller, and when the value of TT is smaller than the value of 1 / (FPS×2), the value of R is controlled to be larger.

[0076] Also, in bandwidth variation tracking control, in order to quickly follow bandwidth variations, if the packet reception time (PadVspRTT) rises sharply, further reduction of the bit rate is immediately executed.

[0077] For example, by passing the time-series data of PadVspRTT through a predetermined low-pass filter, fine noise is removed. Hereinafter, PadVspRTT from which such fine noise has been removed will be expressed as FilPadVspRTT. This is shown in FIG. 4 as FilPadVspRTT = LPF(PadVspRTT).

[0078] Then, with the value of EstBtmLatency described later as the target value, the value of FilPadVspRTT as the current value, and the value D of the bit rate reduction amount as the manipulated variable, the value D of the bit rate reduction amount is determined by PD control. Here, in order to quickly follow bandwidth variations, PD control is performed instead of PID control.

[0079] This is shown in FIG. 4 as D = PD(FilPadVspRTT-EstBtmLatency). For example, the larger the value of FilPadVspRTT-EstBtmLatency, the larger the value of D is controlled to be. Also, when the value of FilPadVspRTT is larger than the value of EstBtmLatency, the value of D is controlled to be larger. Further, when the value of FilPadVspRTT is smaller than the value of EstBtmLatency, the value of D is controlled to be smaller.

[0080] In this embodiment, the value of EstBtmLatency is set to a predetermined value in the initial state. And each time PadVspRTT is specified, the absolute value V of the difference between the latest PadVspRTT (PadVspRTT[n]) and the previous PadVspRTT (PadVspRTT[n-1]) is specified. Then, when the state where the absolute value V is less than Th1 occurs continuously N times for a given value N and a predetermined threshold Th1, the value of EstBtmLatency is updated. For example, the value of EstBtmLatency is updated to the average value of the values of PadVspRTT in the N times when the state where the absolute value V is less than Th1 occurs continuously. This is shown in FIG. 4 as EstBtmLatency = Average(PadVspRTT[n]~PadVspRTT[n-N+1]).

[0081] In this embodiment, in this way, the value of PadVspRTT in a state where the bandwidth is stable to a certain extent is set as the value of EstBtmLatency. Therefore, the more the current value of PadVspRTT deviates from the value of PadVspRTT in the stable state, the larger the value of D determined is.

[0082] In this embodiment, a value B obtained by subtracting a bit rate reduction amount value D from a bit rate value R determined as described above is specified as the final bit rate value. However, when the determined value D is negative, the value R is not adjusted, and the value R is directly specified as the value B. This is shown in FIG. 4 as B = R - D (when D < 0, D = 0).

[0083] The bandwidth variation tracking control may be executed at a predetermined timing. For example, the bandwidth variation tracking control may be executed at a predetermined period (e.g., a 16-millisecond period). Also, the bandwidth variation tracking control may be executed in response to the occurrence of a predetermined event (e.g., the reception of the last segment of image data).

[0084] In this embodiment, at the terminal 12, for example, the reception of the VSP is monitored at the transmission period of the pad data P. When the number of received VSPs M within a recent predetermined time t1 is smaller than a predetermined threshold Th2, the bandwidth variation tracking control is interrupted. For example, when conditions such as the number of received M within the last 100 milliseconds being less than 5 or no VSP being received within the last 80 milliseconds are satisfied, the bandwidth variation tracking control is interrupted.

[0085] Then, packet congestion reduction control, which is a process of multiplying the above-described bit rate value B by a predetermined ratio r (r < 1), is executed. Note that the process executed in the packet congestion reduction control does not necessarily have to be a process of multiplying the value B by the ratio r as long as it is a process of reducing the value B. For example, a process of subtracting a predetermined value from the value B may be executed.

[0086] In the packet congestion reduction control according to this embodiment, when the value B reaches a predetermined lower limit b1, the value B is controlled not to become smaller than that.

[0087] The packet congestion reduction control may be executed at a predetermined timing. For example, the packet congestion reduction control may be executed at a predetermined period (e.g., the transmission period of the pad data P). Further, the packet congestion reduction control may be executed in response to the occurrence of a predetermined event (e.g., the transmission of the pad data P).

[0088] In the present embodiment, when the number of received VSPs M at the most recent predetermined time t1 is equal to or greater than a predetermined threshold Th2, the packet congestion reduction control is terminated and the bandwidth fluctuation tracking control is restarted.

[0089] In mobile communication systems such as 4G and 5G, retransmission and buffering control are performed thoroughly, so data is likely to be accumulated in the buffer of a relay device such as a base station. For example, when the downlink communication stops in a communication environment such as 4G or 5G, the data transmitted from the cloud server 10 is likely to be stored in the relay device such as the base station. The data stored in this way is transmitted all at once when the downlink communication returns to normal.

[0090] The increase in delay due to the discharge of the stored data and the occurrence of an overflow in the reception buffer of the terminal 12 due to the terminal 12 receiving a large amount of data at once are factors causing the loss of received data at the terminal 12. In the present embodiment, by performing the packet congestion reduction control as described above, the amount of data staying (packet congestion) in the computer network 14 when the communication becomes unavailable due to the deterioration of the communication environment is reduced, and the occurrence of loss of received data can be suppressed.

[0091] The cloud server 10 according to the present embodiment changes the compression rate in the encoding of the play image so that the value of the bit rate of the transmitted moving image becomes the value B determined by the bandwidth fluctuation tracking control or the value B updated by the packet congestion reduction control. In this way, in the present embodiment, the data size of the generated image data is controlled.

[0092] As described above, according to the present embodiment, image data with an appropriate data size considering delay is generated in the cloud server 10.

[0093] Also, in the present embodiment, as described above, based on the packet reception time (PadVspRTT), the data size of the image data to be transmitted from the cloud server 10 is controlled. By doing so, it is possible to immediately reduce the bit rate of the moving image in response to a sudden decrease in throughput.

[0094] Hereinafter, the pad sync control performed in the cloud gaming system 1 according to the present embodiment will be described.

[0095] In the cloud gaming service provided by the cloud gaming system 1, in order to reduce the discomfort of the user in the operation, it is desirable that the time from the reception of the operation data in the cloud server 10 to the start of generating the play image is as constant as possible.

[0096] However, depending on the communication environment, the time until the operation data transmitted from the terminal 12 reaches the cloud server 10 may vary. This is particularly prominent in wireless communication using a mobile communication system with large bandwidth fluctuations such as 4G or 5G.

[0097] Therefore, the time from the reception of the operation data in the cloud server 10 to the start of generating the play image may not fall within the allowable range, and the user may feel discomfort.

[0098] Therefore, in the present embodiment, by performing the pad sync control shown in FIG. 5, it is possible to reduce the discomfort of the user in the situation where the moving image generated in the cloud server 10 according to the operation at the terminal 12 is displayed on the terminal 12.

[0099] FIG. 5 is an explanatory diagram for explaining an example of the pad sync control performed in the cloud gaming system 1 according to the present embodiment.

[0100] As shown in FIG. 5, in the pad sink control according to the present embodiment, two types of control, i.e., first control and second control, are performed.

[0101] In the first control, the transmission timing of the pad data P is controlled so that the value indicated by DiffPadVsp (see FIG. 2) approaches a predetermined value T1 (for example, a value T1 that is 1.5 times the transmission period of the pad data P).

[0102] Here, for example, with the value T1 as the target value, the value of DiffPadVsp as the current value, and the transmission period T of the pad data P as the manipulated variable, the transmission period T of the pad data P is determined by PD control. This is shown in FIG. 5 as T = PD(DiffPadVsp - T1). For example, the larger the value of (DiffPadVsp - T1), the smaller the value of T is controlled. Also, when the value of DiffPadVsp is larger than the value of T1, the value of T is controlled to be smaller, and when the value of DiffPadVsp is smaller than the value of T1, the value of T is controlled to be larger.

[0103] And the pad data P may be transmitted at the determined transmission period T. For example, the pad data P may be transmitted at a timing when a time corresponding to the determined transmission period T has elapsed from the transmission timing of the most recent pad data P.

[0104] Then, for example, when it is determined that the value of DiffPadVsp has stabilized, the pad sink control shifts from the first control to the second control. For example, when a predetermined condition is satisfied, such as a state where the value indicating the fluctuation of the value of DiffPadVsp (for example, the absolute value of DiffPadVsp - T1) is less than a predetermined threshold Th3 and this state has continued for a predetermined time t2 or more, the pad sink control shifts from the first control to the second control.

[0105] In the second control, the value T1, which was a fixed value in the first control, is made variable so that the value indicating the time (PRecvGStartTime) from the reception timing of the pad data P in the cloud server 10 to the generation start timing of the play image based on the pad data P approaches a predetermined value T2 (for example, 1 millisecond). The value T1 in this case will be expressed as T1_adj.

[0106] Here, for example, the value T1_adj is determined by PD control with the value T2 as the target value, the value of PRecvGStartTime as the current value, and the value T1_adj as the manipulated variable. This is shown in FIG. 5 as T1_adj = PD(PRecvGStartTime - T2). And for example, the transmission period T of the pad data P is determined by PD control with the value T1_adj as the target value, the value of DiffPadVsp as the current value, and the transmission period T of the pad data P as the manipulated variable. This is shown in FIG. 5 as T = PD(DiffPadVsp - T1_adj). Summing up, T = PD(DiffPadVsp - PD(PRecvGStartTime - T2)).

[0107] And the pad data P may be transmitted at the transmission period T determined as described above. For example, the pad data P may be transmitted at the timing when a time corresponding to the determined transmission period T has elapsed from the transmission timing of the most recent pad data P.

[0108] And for example, when it is determined that the value of DiffPadVsp has become unstable, the pad sync control shifts from the second control to the first control. For example, when a predetermined condition is satisfied, such as when a state occurs in which a value indicating the fluctuation of the value of DiffPadVsp (for example, the absolute value of DiffPadVsp - T1) is equal to or greater than a predetermined threshold Th3, the pad sync control shifts from the second control to the first control.

[0109] Pad sink control may be executed at a predetermined timing. For example, the pad sink control may be executed at a predetermined period (e.g., a 16-millisecond period). Also, the bandwidth variation tracking control may be executed in response to the occurrence of a predetermined event (e.g., the reception of the last segment of image data).

[0110] As described above, in this embodiment, by synchronizing the transmission timing of the pad data P and the generation timing of the play image through pad sink control, low latency can be achieved. In this way, according to this embodiment, the discomfort of the user in the situation where the moving image generated by the cloud server 10 is displayed on the terminal 12 in response to the operation on the terminal 12 is reduced.

[0111] Also, in this embodiment, by executing the above-described first control and second control, PRecvGStartTime can be stabilized.

[0112] Hereinafter, centering on the points described above, the functions of the cloud gaming system 1 according to this embodiment and the processes executed in the cloud gaming system 1 according to this embodiment will be further described.

[0113] FIG. 6 is a functional block diagram showing an example of the functions implemented in the cloud gaming system 1 according to this embodiment. Note that in the cloud gaming system 1 according to this embodiment, it is not necessary to implement all of the functions shown in FIG. 6, and functions other than those shown in FIG. 6 may be implemented.

[0114] As shown in FIG. 6, the cloud server 10 according to this embodiment functionally includes, for example, a server-side control data storage unit 20, an operation data reception unit 22, a frame image generation unit 24, a VSP transmission unit 26, an encoding process execution unit 28, an image data transmission unit 30, and a server-side traffic control unit 32. The cloud server 10 in this embodiment serves as a moving image transmission device that sequentially transmits image data representing frame images constituting a moving image.

[0115] The server - side control data storage unit 20 is mainly implemented by the storage unit 10b. The operation data receiving unit 22, the VSP transmitting unit 26, and the image data transmitting unit 30 are mainly implemented by the communication unit 10c. The frame image generation unit 24 and the server - side traffic control unit 32 are mainly implemented by the processor 10a. The encoding process execution unit 28 is mainly implemented by the processor 10a and the encoder - decoder unit 10d.

[0116] The above functions may also be implemented by executing, with the processor 10a, a program including commands corresponding to the above functions, which is installed in the cloud server 10, which is a computer. This program may be supplied to the cloud server 10 via a computer - readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto - optical disk, a flash memory, etc., or via the Internet or the like.

[0117] Also, as shown in FIG. 6, the terminal 12 according to the present embodiment functionally includes, for example, a terminal - side control data storage unit 40, an operation data generation unit 42, an operation data transmitting unit 44, a VSP receiving unit 46, an image data receiving unit 48, a decoding process execution unit 50, a frame image display control unit 52, a terminal - side traffic control unit 54, and a transmission timing control unit 56. In the present embodiment, the terminal 12 serves as a moving - image receiving device that sequentially receives image data representing frame images constituting a moving image.

[0118] The terminal - side control data storage unit 40 is mainly implemented by the storage unit 12b. The operation data generation unit 42 is mainly implemented by the processor 10a and the operation unit 12e. The operation data transmitting unit 44, the VSP receiving unit 46, and the image data receiving unit 48 are mainly implemented by the communication unit 12c. The decoding process execution unit 50 is mainly implemented by the processor 10a and the encoder - decoder unit 12h. The frame image display control unit 52 is mainly implemented by the processor 12a and the display unit 12d. The terminal - side traffic control unit 54 and the transmission timing control unit 56 are mainly implemented by the processor 10a.

[0119] The above functions may be implemented by the processor 12a executing a program including commands corresponding to the above functions, which is installed in the terminal 12 that is a computer. This program may be supplied to the terminal 12 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, a flash memory, etc., or via the Internet or the like.

[0120] In this embodiment, the server-side control data storage unit 20 stores, for example, control data indicating the value B of the above-described bit rate.

[0121] In this embodiment, the operation data reception unit 22 receives operation data (for example, the above-described pad data P) corresponding to the user's input operation. The operation data reception unit 22 receives, for example, operation data corresponding to an input operation in the play of a game.

[0122] In this embodiment, the frame image generation unit 24 generates a frame image based on the operation data received by the operation data reception unit 22. The frame image generation unit 24 generates, for example, a play image representing the play status of the game being played by the user based on the pad data P.

[0123] In this embodiment, the VSP transmission unit 26 transmits a VSP, which is a packet associated with the operation data, to the terminal 12 in response to the start of generation of a frame image based on the operation data. The VSP transmission unit 26 transmits the VSP, for example, at the timing when the generation of the frame image is started.

[0124] In this embodiment, for example, the encoding process execution unit 28 generates image data representing the frame image by encoding the frame image generated by the frame image generation unit 24. The encoding process execution unit 28 may determine a compression rate such that the value indicating the bit rate of the moving image to be transmitted becomes the value B indicated by the control data stored in the server-side control data storage unit 20. Then, the encoding process execution unit 28 may generate image data by encoding the frame image at the determined compression rate.

[0125] Further, the encoding process execution unit 28 may generate the above-described interval data. Then, the encoding process execution unit 28 may associate the generated interval data with the generated image data.

[0126] In this embodiment, for example, the image data transmission unit 30 transmits the image data generated by the encoding process execution unit 28 to the terminal 12. The image data transmission unit 30 may transmit the image data associated with the above-described interval data to the terminal 12.

[0127] In this embodiment, for example, the server-side traffic control unit 32 controls the data size of the image data to be transmitted by the image data transmission unit 30.

[0128] As described above, the server-side traffic control unit 32 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the value of TT described above indicating the time required for the terminal 12 to receive the image data.

[0129] Also, as described above, the server-side traffic control unit 32 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the above-described packet reception time (PadVspRTT).

[0130] In this embodiment, for example, the terminal-side control data storage unit 40 stores control data indicating the value B of the above-described bit rate.

[0131] The operation data generation unit 42 generates, for example, the above-described operation data according to a user's input operation in the present embodiment. The operation data generation unit 42 may generate operation data associated with the control data stored in the terminal-side control data storage unit 40.

[0132] The operation data transmission unit 44 transmits, for example, the operation data generated by the operation data generation unit 42 to the cloud server 10 in the present embodiment.

[0133] The operation data transmission unit 44 may transmit operation data associated with the control data. In this case, the server-side traffic control unit 32 may acquire the control data associated with the operation data received by the operation data reception unit 22. Then, the server-side traffic control unit 32 may update the control data acquired from the control data stored in the server-side control data storage unit 20 with the acquired control data.

[0134] Note that the operation data transmission unit 44 does not necessarily need to transmit the control data in association with the operation data, and may transmit the control data to the cloud server 10 independently of the operation data. And the operation data reception unit 22 may receive the control data transmitted in this way. In this case, the server-side traffic control unit 32 may update the control data stored in the server-side control data storage unit 20 with the control data received by the operation data reception unit 22.

[0135] The VSP reception unit 46 receives, for example, a VSP which is a packet associated with the operation data and is transmitted from the cloud server 10 in response to the start of generation of a frame image based on the operation data in the present embodiment.

[0136] The image data reception unit 48 receives, for example, image data transmitted from the cloud server 10. As described above, the image data reception unit 48 may receive image data associated with the interval data.

[0137] In this embodiment, for example, the decoding process execution unit 50 decodes the image data received by the image data receiving unit 48 to generate a frame image (for example, a play image) represented by the image data.

[0138] In this embodiment, for example, the frame image display control unit 52 causes the display unit 12d to display the frame image (for example, a play image) generated by the decoding process execution unit 50.

[0139] In this embodiment, for example, the terminal-side traffic control unit 54 controls the data size of the image data to be transmitted from the cloud server 10.

[0140] As described above, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the value of TT described above, which indicates the time required for receiving the image data transmitted from the cloud server 10.

[0141] Also, as described above, the terminal-side traffic control unit 54 may specify the above-described packet reception time (PadVspRTT), which is the time from the timing when the terminal 12 transmits the operation data to the timing when the VSP associated with the operation data is received. Then, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the packet reception time.

[0142] The terminal-side traffic control unit 54 may execute the above-described bandwidth fluctuation tracking control. Then, the terminal-side traffic control unit 54 may update the control data stored in the terminal-side control data storage unit 40 to control data indicating the value B determined by executing the bandwidth fluctuation tracking control.

[0143] Also, as described in the above bandwidth fluctuation tracking control, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted hereafter based on the packet reception time in the reception of the latest VSP and the packet reception time in the reception of at least one VSP before the latest one. Specifically, for example, control such as D = PD(FilPadVspRTT - EstBtmLatency) shown in FIG. 4 may be executed.

[0144] Also, the terminal-side traffic control unit 54 may execute the above packet congestion reduction control. Then, the terminal-side traffic control unit 54 may update the control data stored in the terminal-side control data storage unit 40 to control data indicating a value B updated by executing the packet congestion reduction control.

[0145] Also, as described in the above packet congestion reduction control, when it is determined that the failure of VSP reception continues based on a predetermined condition, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted hereafter by the image data transmission unit 30 to be smaller. For example, as described above, when the number of VSP receptions M in the most recent predetermined time t1 is smaller than a predetermined threshold Th2, the above packet congestion reduction control may be executed.

[0146] Also, the terminal-side traffic control unit 54 may hold a bitrate control mode flag indicating the mode of bitrate control. Then, for example, the bandwidth fluctuation tracking control may be executed when the value of the bitrate control mode flag is 0, and the packet congestion reduction control may be executed when the value of the bitrate control mode flag is 1.

[0147] In this embodiment, the transmission timing control unit 56 controls the transmission timing of operation data, for example.

[0148] The transmission timing control unit 56 may control the time from the timing of receiving the VSP until the next operation data is transmitted. For example, the transmission timing control unit 56 may execute a first control for controlling the timing at which the operation data is transmitted so that the time from the timing of receiving the VSP until the next operation data is transmitted achieves a first target. For example, in the above example, a state where a value indicating the variation of the DiffPadVsp value is less than a predetermined threshold Th3 and a predetermined time t2 or more has elapsed corresponds to the first target.

[0149] Also, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data. For example, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data associated with the image data received by the image data receiving unit 48.

[0150] Here, for example, the transmission timing control unit 56 may execute a second control for controlling the timing at which the operation data is transmitted so that the time indicated by the interval data achieves a second target. For example, in the above example, the value of PRecvGStartTime becoming a predetermined value T2 (for example, 1 millisecond) corresponds to the second target.

[0151] Also, as described above, the transmission timing control unit 56 may start the second control in response to the first target being achieved in the first control.

[0152] Also, as described above, the transmission timing control unit 56 may control the transmission cycle T of the operation data to be transmitted hereafter based on the interval data.

[0153] In addition, the image data transmission unit 30 does not need to transmit image data associated with interval data. The image data transmission unit 30 may transmit the interval data to the cloud server 10 independently of the image data. Then, the image data reception unit 48 may receive the interval data transmitted in this manner. And the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data received from the cloud server 10 in this manner.

[0154] In addition, the transmission timing control unit 56 may hold a pad sink control mode flag indicating the mode of pad sink control. And, for example, the first control may be executed when the value of the pad sink control mode flag is 0, and the second control may be executed when the value of the pad sink control mode flag is 1.

[0155] The transmission timing control unit 56 may output a transmission command to the operation data transmission unit 44 at the transmission period T determined as described above, for example. And the operation data transmission unit 44 may transmit the operation data to the cloud server 10 in response to the reception of the transmission command.

[0156] Hereinafter, an example of the processing flow in the bandwidth fluctuation tracking control performed by the terminal 12 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 7.

[0157] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to the bandwidth fluctuation tracking control arrives (S101).

[0158] When the predetermined execution timing arrives, the terminal-side traffic control unit 54 specifies the value of TT for the latest image data received by the image data reception unit 48 (S102).

[0159] Then, the terminal-side traffic control unit 54 specifies the above-described value R based on the TT specified in the process shown in S102 (S103).

[0160] Then, the terminal-side traffic control unit 54 identifies the value of the latest PadVspRTT (S104).

[0161] Then, the terminal-side traffic control unit 54 identifies the absolute value V of the difference between the value of the latest PadVspRTT and the immediately preceding PadVspRTT (S105).

[0162] Then, the terminal-side traffic control unit 54 checks whether the state where the absolute value V is less than Th1 has occurred continuously for N times (S106).

[0163] If it is confirmed that the state where the absolute value V is less than Th1 has occurred continuously for N times (S106: Y), the terminal-side traffic control unit 54 updates the value of EstBtmLatency (S107).

[0164] If it is confirmed that the state where the absolute value V is less than Th1 has not occurred continuously for N times (S106: N), or if the process shown in S107 has been executed, the terminal-side traffic control unit 54 identifies the value of FilPadVspRTT (S108).

[0165] Then, based on the value of FilPadVspRTT identified in the process shown in S108 and the latest value of EstBtmLatency, the terminal-side traffic control unit 54 identifies the above-mentioned value D (S109).

[0166] Then, based on the value R identified in the process shown in S103 and the value D identified in the process shown in S109, the terminal-side traffic control unit 54 identifies the value B (S110).

[0167] Then, the terminal-side traffic control unit 54 updates the control data stored in the terminal-side control data storage unit 40 so that the set value B becomes the value B identified in the process shown in S109 (S111), and returns to the process shown in S101.

[0168] Next, an example of the processing flow in the packet congestion reduction control performed by the terminal 12 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 8.

[0169] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to the packet congestion reduction control arrives (S201).

[0170] When the predetermined execution timing arrives, the terminal-side traffic control unit 54 specifies the value B of the bit rate indicated by the control data stored in the terminal-side control data storage unit 40 (S202).

[0171] Then, the terminal-side traffic control unit 54 checks whether the value B specified in the process shown in S202 is smaller than the lower limit b1 (S203).

[0172] If it is confirmed that the value B is smaller than the lower limit b1 (S203: Y), the process returns to the process shown in S201.

[0173] If it is confirmed that the value B is not smaller than the lower limit b1 (S203: N), the terminal-side traffic control unit 54 specifies a value obtained by multiplying the value B specified in the process shown in S202 by a predetermined ratio r (r is less than 1) as a new value B (S204).

[0174] Then, the terminal-side traffic control unit 54 updates the control data stored in the terminal-side control data storage unit 40 so that the set value B becomes the value B specified in the process shown in S204 (S205), and returns to the process shown in S201.

[0175] Next, an example of the processing flow of the bit rate control mode switching process performed by the terminal 12 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 9.

[0176] First, the terminal-side traffic control unit 54 waits until a predetermined execution timing related to the bit rate control mode switching process arrives (S301).

[0177] Here, for example, it may be arranged such that the execution timing arrives at a predetermined period (for example, the transmission period T of operation data). Further, the execution timing may be arranged to arrive in response to the occurrence of a predetermined event (for example, the transmission of operation data).

[0178] When the predetermined execution timing arrives, the terminal-side traffic control unit 54 specifies the number of received VSPs M in the most recent predetermined time t1 (S302).

[0179] Then, the terminal-side traffic control unit 54 checks the value of the current bit rate control mode flag (S303).

[0180] If the value of the confirmed bit rate control mode flag is 0, the terminal-side traffic control unit 54 checks whether the number of received packets M specified in the process shown in S302 is less than a predetermined threshold Th2 (S304).

[0181] If the number of received packets M is equal to or greater than the predetermined threshold Th2 (S304: N), the process returns to the process shown in S301.

[0182] If the number of received packets M is less than the predetermined threshold Th2 (S304: Y), the terminal-side traffic control unit 54 changes the value of the held bit rate control mode flag to 1 (S305), and returns to the process shown in S301.

[0183] If the value of the current bit rate control mode flag confirmed in the process shown in S303 is 1, the terminal-side traffic control unit 54 checks whether the number of received packets M specified in the process shown in S302 is equal to or greater than a predetermined threshold Th2 (S306).

[0184] If the number of received packets M is less than the predetermined threshold Th2 (S306: N), the process returns to the process shown in S301.

[0185] When the number of received packets M is greater than or equal to a predetermined threshold Th2 (S306: Y), the terminal-side traffic control unit 54 changes the value of the held bitrate control mode flag to 0 (S307) and returns to the process shown in S301.

[0186] Hereinafter, an example of the flow of the process in the pad sink control performed by the terminal 12 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 10.

[0187] First, the transmission timing control unit 56 waits until a predetermined execution timing related to the pad sink control arrives (S401).

[0188] When the predetermined execution timing arrives, the transmission timing control unit 56 specifies the value of the latest DiffPadVsp (S402).

[0189] Then, the transmission timing control unit 56 checks the value of the current pad sink control mode flag (S403).

[0190] When the confirmed value of the pad sink control mode flag is 0, the transmission timing control unit 56 determines a new transmission period T such that the value of DiffPadVsp specified in the process shown in S402 becomes T1 (S404).

[0191] Then, the transmission timing control unit 56 specifies the absolute value V of the difference between the value of the latest DiffPadVsp and the value T1 (S405).

[0192] Then, the transmission timing control unit 56 specifies the time during which the state where the absolute value V is less than the predetermined threshold Th3 continues (S406).

[0193] Then, the transmission timing control unit 56 checks whether or not the time specified in the process shown in S406 has reached the predetermined time (S407).

[0194] If it is confirmed that the predetermined time has not been reached (S407: N), the process returns to the process shown in S401.

[0195] When it is confirmed that the specified time has been reached (S407: Y), the transmission timing control unit 56 changes the value of the held pad sink control mode flag to 1 (S408), and returns to the process shown in S401.

[0196] When the value of the current pad sink control mode flag confirmed in the process shown in S403 is 1, the transmission timing control unit 56 specifies the value of the interval data associated with the latest image data (S409).

[0197] Then, the transmission timing control unit 56 determines a new transmission period T based on the value of the interval data specified in the process shown in S409 (S410).

[0198] Then, the transmission timing control unit 56 specifies the absolute value V of the difference between the value of the latest DiffPadVsp and the value T1 (S411).

[0199] Then, the transmission timing control unit 56 checks whether the specified absolute value V is equal to or greater than a predetermined threshold Th3 (S412).

[0200] If the absolute value V is not equal to or greater than the predetermined threshold Th3 (S412: N), the process returns to the process shown in S401.

[0201] If the absolute value V is equal to or greater than the predetermined threshold Th3 (S412: Y), the transmission timing control unit 56 changes the value of the held pad sink control mode flag to 0 (S413), and returns to the process shown in S401.

[0202] Note that the present invention is not limited to the above-described embodiments.

[0203] For example, the interval data may be associated with the VSP instead of being associated with the image data.

[0204] Further, the frame image generation unit 24 may generate a frame image having a data size corresponding to the bit rate.

[0205] Also, the application range of the present invention is not limited to the cloud gaming system 1.

[0206] Also, the application range of the present invention is not limited to the computer network 14 including mobile communication systems such as 4G and 5G. The present invention is applicable to a computer network 14 in which wireless communication is performed by Wi-Fi (registered trademark), rather than wireless communication via a mobile communication system such as 4G and 5G.

[0207] Also, the above specific character strings, numerical values, and the specific character strings and numerical values in the drawings are examples, and are not limited to these character strings and numerical values.

Explanation of Signs

[0208] 1 Cloud gaming system, 10 Cloud server, 10a Processor, 10b Storage unit, 10c Communication unit, 10d Encoder / Decoder unit, 12 Terminal, 12a Processor, 12b Storage unit, 12c Communication unit, 12d Display unit, 12e Operation unit, 12f Sensor unit, 12g Audio output unit, 12h Encoder / Decoder unit, 14 Computer network, 20 Server-side control data storage unit, 22 Operation data reception unit, 24 Frame image generation unit, 26 VSP transmission unit, 28 Encoding process execution unit, 30 Image data transmission unit, 32 Server-side traffic control unit, 40 Terminal-side control data storage unit, 42 Operation data generation unit, 44 Operation data transmission unit, 46 VSP reception unit, 48 Image data reception unit, 50 Decoding process execution unit, 52 Frame image display control unit, 54 Terminal-side traffic control unit, 56 Transmission timing control unit.

Claims

1. A moving image receiving device that sequentially receives image data representing frame images constituting a moving image from a moving image transmitting device, an operation data transmitting unit that transmits operation data corresponding to a user's input operation to the moving image transmitting device, a packet receiving unit that receives a packet associated with the operation data, which is transmitted from the moving image transmitting device in response to the start of generation of the frame image based on the operation data, a data size control unit that controls the data size of image data to be transmitted from the moving image transmitting device hereafter based on a packet reception time, which is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received, A moving image receiving device, characterized by including the above.

2. The data size control unit controls the data size based on the time required for receiving the image data transmitted from the moving image transmitting device. The moving image receiving device according to claim 1, characterized by the above.

3. The data size control unit controls the data size based on the packet reception time in the reception of the latest packet and the packet reception time in the reception of at least one packet before the latest one. The moving image receiving device according to claim 1 or 2, characterized by the above.

4. When it is determined that the failure of receiving the packet continues based on a predetermined condition, the data size control unit controls so that the data size becomes smaller. The moving image receiving device according to any one of claims 1 to 3, characterized by the above.

5. A moving image transmitting device that sequentially transmits image data representing frame images constituting a moving image to a moving image receiving device, an operation data receiving unit that receives operation data corresponding to a user's input operation from the moving image receiving device, an image generation unit that generates the frame image based on the operation data, a packet transmitting unit that transmits a packet associated with the operation data to the moving image receiving device in response to the start of generation of the frame image, an encoding process execution unit that generates image data representing the frame image by encoding the frame image, an image data transmitting unit that transmits the image data to the moving image receiving device, Based on the time from the timing when the moving image receiving device transmits the operation data to the timing when it receives the packet associated with the operation data, a data size control unit that controls the data size of the image data to be transmitted hereafter by the image data transmission unit; A moving image transmission device, characterized by including the above. **Claim 6** A moving image transmission and reception system including a moving image transmission device that sequentially transmits image data representing frame images constituting a moving image, and a moving image receiving device that sequentially receives the image data, wherein the moving image receiving device includes an operation data transmission unit that transmits operation data corresponding to a user's input operation to the moving image transmission device; a packet receiving unit that receives a packet associated with the operation data, which is transmitted from the moving image transmission device in response to the start of generation of a frame image based on the operation data; and a control data transmission unit that transmits control data for controlling the data size of the image data to be transmitted hereafter by the moving image transmission device, based on the packet reception time, which is the time from the timing when the operation data is transmitted to the timing when the packet associated with the operation data is received; wherein the moving image transmission device includes an operation data receiving unit that receives the operation data from the moving image receiving device; an image generation unit that generates the frame image based on the operation data; a packet transmission unit that transmits a packet associated with the operation data to the moving image receiving device in response to the start of generation of the frame image; an encoding process execution unit that generates image data representing the frame image by encoding the frame image; an image data transmission unit that transmits the image data to the moving image receiving device; a control data receiving unit that receives the control data; and a data size control unit that controls the data size of the image data to be transmitted hereafter by the image data transmission unit, based on the control data; A moving image transmission and reception system, characterized by the above. **Claim 7** A moving image receiving device that sequentially receives image data representing frame images constituting a moving image from a moving image transmission device, the steps including transmitting operation data corresponding to a user's input operation to the moving image transmission device; The step of receiving, by the moving image receiving device, a packet associated with the operation data, which is transmitted from the moving image transmitting device in response to the start of generation of the frame image based on the operation data; The step of controlling, by the moving image receiving device, the data size of image data to be transmitted hereafter by the moving image transmitting device based on a packet reception time, which is the time from the timing when the operation data was transmitted to the timing when the packet associated with the operation data was received; A control method characterized by including the above.

8. A moving image transmitting device that sequentially transmits image data representing frame images constituting a moving image to a moving image receiving device includes: a step of receiving, from the moving image receiving device, operation data corresponding to a user's input operation; The step of generating, by the moving image transmitting device, the frame image based on the operation data; The step of transmitting, by the moving image transmitting device, a packet associated with the operation data to the moving image receiving device in response to the start of generation of the frame image; The step of generating, by the moving image transmitting device, image data representing the frame image by encoding the frame image; The step of transmitting, by the moving image transmitting device, the image data to the moving image receiving device; The step of controlling, by the moving image transmitting device, the data size of image data to be transmitted hereafter by the moving image transmitting device based on the time from the timing when the moving image receiving device transmitted the operation data to the timing when the moving image receiving device received the packet associated with the operation data; A control method characterized by including the above.

9. A computer that sequentially receives image data representing frame images constituting a moving image from a moving image transmitting device is caused to execute: A procedure of transmitting operation data corresponding to a user's input operation to the moving image transmitting device; A procedure of receiving, from the moving image transmitting device, a packet associated with the operation data, which is transmitted from the moving image transmitting device in response to the start of generation of the frame image based on the operation data; A procedure of controlling the data size of image data to be transmitted hereafter by the moving image transmitting device based on a packet reception time, which is the time from the timing when the operation data was transmitted to the timing when the packet associated with the operation data was received; A program characterized by causing the above to be executed.

10. A computer that sequentially transmits image data representing frame images constituting a moving image to a moving image receiving device, A procedure for receiving operation data corresponding to a user's input operation from the moving image receiving device, A procedure for generating the frame image based on the operation data, A procedure for transmitting a packet associated with the operation data to the moving image receiving device in response to the start of generation of the frame image, A procedure for generating image data representing the frame image by encoding the frame image, A procedure for transmitting the image data to the moving image receiving device, A procedure for controlling the data size of image data to be transmitted by the computer based on the time from the timing when the moving image receiving device transmits the operation data to the timing when the packet associated with the operation data is received, A program characterized by causing the above to be executed.

Images