Video receiving device, control method and program
The moving image receiving device stabilizes the timing of operation data transmission and frame image generation in cloud gaming, addressing user discomfort caused by variable delays in 4G and 5G networks through synchronized control and adaptive bit rate management.
Patent Information
- Application Number
- JP2023517463
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-30
- Filing Date
- 2022-04-19
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-04-19
AI Technical Summary
In cloud gaming services, the asynchronous transmission of operation data and frame image generation cycles can lead to variable delays, causing user discomfort due to inconsistent timing from operation data reception to frame image generation, particularly in mobile communication systems with large band fluctuations like 4G and 5G.
A moving image receiving device and method that controls the timing of operation data transmission and packet reception to stabilize the time from operation data reception to frame image generation, using interval data and transmission timing control to synchronize with the generation cycle, and adjusts bit rates to maintain consistent display.
Reduces user discomfort by stabilizing the time from operation data reception to frame image generation, ensuring smooth and timely display of game play situations, even in fluctuating wireless communication environments.
Smart Images

Figure 0007595154000001 
Figure 0007595154000002 
Figure 0007595154000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a video receiving device, a control method, and a program. [Background technology]
[0002] In a technology of cloud gaming services that has been attracting attention in recent years, operation data corresponding to input operations by a user to a terminal is wirelessly transmitted from the terminal to a cloud server. Then, in the cloud server, a frame image representing the game play status is generated based on the operation data. Then, image data obtained by encoding the frame image is wirelessly transmitted from the cloud server to the terminal, and the frame image obtained by decoding the image data is displayed on the terminal. By repeatedly executing this series of processes, a moving image representing the game play status is displayed on the terminal.
[0003] Furthermore, in some cloud gaming services, the transmission of operation data from a terminal to a cloud server and the generation of frame images in the cloud server are performed asynchronously, and the transmission cycle of the operation data differs from the generation cycle of frame images. Summary of the Invention [Problem to be solved by the invention]
[0004] In order to reduce the sense of discomfort felt by the user during operation, it is desirable to keep the time from receipt of operation data in the cloud server to the start of generation of a frame image as constant as possible.
[0005] However, depending on the communication environment, the time it takes for operation data sent from a terminal to reach a cloud server may vary. This is particularly noticeable in wireless communications using mobile communication systems with large bandwidth fluctuations, such as the fourth generation mobile communication system (4G) and the fifth generation mobile communication system (5G).
[0006] As a result, the time from when the cloud server receives the operation data to when it starts generating a frame image may not fall within an acceptable range, causing a sense of discomfort to the user.
[0007] This not only applies to situations where cloud gaming services are provided, but also generally to situations where operation data transmitted from a video receiving device corresponding to a terminal is received by a video transmitting device corresponding to a cloud server, and the video transmitting device generates frame images and transmits them to the video receiving device.
[0008] The present invention has been made in consideration of the above-mentioned situation, and one of its objectives is to provide a video receiving device, a control method, and a program that can reduce the sense of discomfort felt by a user when a video generated by a video transmitting device in response to an operation on the video receiving device is displayed on the video receiving device. [Means for solving the problem]
[0009] In order to solve the above problems, the video receiving device of the present invention is a video receiving device that sequentially receives image data representing frame images that constitute a video from a video transmitting device, and includes: an operation data transmitting unit that transmits operation data corresponding to a user's input operation to the video transmitting device; an interval data receiving unit that receives interval data from the video transmitting device that indicates the time from receipt of the operation data at the video transmitting device to the start of generation of the frame image based on the operation data; and a transmission timing control unit that controls the timing at which the next operation data is transmitted based on the interval data.
[0010] In one aspect of the present invention, the device further includes a packet receiving unit that receives a packet corresponding to the operation data and is transmitted from the video transmitting device in response to the start of generation of the frame image based on the operation data, and the transmission timing control unit controls the time from the time the packet is received to the time the next operation data is transmitted.
[0011] In this aspect, the transmission timing control unit executes a first control for controlling the timing at which the operation data is transmitted so that the time from the time at which the packet is received to the time at which the next operation data is transmitted achieves a first target, and a second control for controlling the timing at which the operation data is transmitted so that the time indicated by the interval data achieves a second target, and the transmission timing control unit may start the second control in response to the first target being achieved in the first control.
[0012] In one aspect of the present invention, the transmission timing control section controls a transmission cycle of the operation data to be transmitted based on the interval data.
[0013] In one aspect of the present invention, the moving image is a moving image generated according to a game play situation, and the input operation is an input operation during play of the game.
[0014] In addition, the control method of the present invention includes a step in which a video receiving device sequentially receives image data representing frame images that constitute a video from a video transmitting device, and transmits operation data corresponding to a user's input operation to the video transmitting device; a step in which the video transmitting device receives interval data indicating the time from reception of the operation data at the video transmitting device to the start of generation of the frame images based on the operation data; and a step in which the timing at which the next operation data is transmitted is controlled based on the interval data.
[0015] In addition, the program of the present invention causes a computer that sequentially receives frame images that constitute a moving image from a moving image transmitting device to execute the following steps: transmitting operation data corresponding to a user's input operation to the moving image transmitting device; receiving interval data from the moving image transmitting device that indicates the time from receipt of the operation data at the moving image transmitting device to the start of generation of the frame image based on the operation data; and controlling the timing at which the next operation data is transmitted based on the interval data. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a cloud gaming system according to one embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram illustrating an example of timing of processing that occurs in a cloud server according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a schematic diagram illustrating an example of communication that occurs in a cloud gaming system according to an embodiment of the present invention. [Figure 4] FIG. 1 is an explanatory diagram illustrating an example of bitrate control performed in a cloud gaming system according to one embodiment of the present invention. [Diagram 5] FIG. 11 is an explanatory diagram illustrating an example of pad sync control performed in a cloud gaming system according to one embodiment of the present invention. [Figure 6] A functional block diagram showing an example of functions implemented in a cloud gaming system according to one embodiment of the present invention. [Figure 7] FIG. 4 is a flow diagram showing an example of a flow of a process performed in a cloud server according to an embodiment of the present invention. [Figure 8] FIG. 4 is a flow diagram showing an example of a flow of a process performed in a cloud server according to an embodiment of the present invention. [Figure 9] FIG. 4 is a flow diagram showing an example of a flow of a process performed in a cloud server according to an embodiment of the present invention. [Figure 10] FIG. 4 is a flow diagram showing an example of a flow of a process performed in a cloud server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0018] Fig. 1 is a diagram showing an example of the overall configuration of a cloud gaming system 1 according to an embodiment of the present invention. As shown in Fig. 1, the cloud gaming system 1 according to this embodiment includes a cloud server 10 and a terminal 12, both of which are configured mainly with a computer.
[0019] The cloud server 10 and the terminal 12 are connected to a computer network 14 including a mobile communication system such as a fourth generation mobile communication system (4G) or a fifth generation mobile communication system (5G) and the Internet. For example, the cloud server 10 is connected to the Internet, and the terminal 12 is connected to a mobile communication system such as 4G or 5G. The cloud server 10 and the terminal 12 are capable of communicating with each other via the computer network 14.
[0020] The cloud server 10 according to the present embodiment is, for example, a server computer that executes a game program related to the cloud gaming service.
[0021] As shown in FIG. 1, the cloud server 10 includes, for example, a processor 10a, a storage unit 10b, a communication unit 10c, and an encoder / decoder unit 10d.
[0022] The processor 10a is a program-controlled device such as a CPU, and executes various information processes according to programs stored in the storage unit 10b. The processor 10a according to the present embodiment also includes a GPU (Graphics Processing Unit) that draws images in a frame buffer based on graphics commands and data supplied from the CPU.
[0023] The storage unit 10b is, for example, a storage element such as a ROM or a RAM, a solid-state drive (SSD), etc. The storage unit 10b stores programs executed by the processor 10a, etc. In addition, the storage unit 10b according to this embodiment has a frame buffer area reserved for rendering images by the GPU included in the processor 10a.
[0024] The communication unit 10c is a communication interface for transmitting and receiving data to and from a computer such as the terminal 12 via the computer network 14, for example.
[0025] The encoder-decoder unit 10d includes, for example, an encoder and a decoder. The encoder generates image data representing an input image by encoding the input image. The decoder decodes the input image data and outputs the image represented by the image data.
[0026] The terminal 12 according to the present embodiment is a computer such as a smartphone or a tablet terminal used by a user who uses the cloud gaming service. The terminal 12 may also be an electronic device capable of communicating with the cloud server 10 via a communication dongle, such as a television including a communication dongle.
[0027] As shown in FIG. 1, the terminal 12 includes, for example, a processor 12a, a memory unit 12b, a communication unit 12c, a display unit 12d, an operation unit 12e, a sensor unit 12f, an audio output unit 12g, and an encoder / decoder unit 12h.
[0028] The processor 12a is a program-controlled device such as a CPU, and executes various types of information processing according to programs stored in the storage unit 12b.
[0029] The storage unit 12b is, for example, a storage element such as a ROM or a RAM, a solid-state drive (SSD), etc. The storage unit 12b stores programs executed by the processor 12a, etc.
[0030] The communication unit 12c is a communication interface for transmitting and receiving data to and from a computer such as the cloud server 10 via the computer network 14, for example.
[0031] The display unit 12d is a display device such as a liquid crystal display or an organic EL display.
[0032] The operation unit 12e is, for example, an operation member for performing an operation input to the processor 12a.
[0033] The sensor unit 12f is a sensor such as a motion sensor capable of detecting, for example, acceleration and angular velocity.
[0034] The audio output unit 12g is an audio output device such as a speaker that outputs the sound represented by the audio data.
[0035] The encoder-decoder unit 12h includes, for example, an encoder and a decoder. The encoder generates image data representing an input image by encoding the input image. The decoder decodes the input image data and outputs an image represented by the image data.
[0036] The terminal 12 may include a touch panel. In this case, the touch panel serves as both the display unit 12d and the operation unit 12e.
[0037] In this embodiment, when a user performs an input operation on the operation unit 12e in playing a game in the cloud gaming service, the terminal 12 generates operation data corresponding to the input operation and transmits the operation data to the cloud server 10. Hereinafter, the operation data will be referred to as pad data P.
[0038] Then, the cloud server 10 executes game processing according to the received pad data P. Then, the cloud server 10 generates a play image, which is a frame image representing the play situation of the game, based on the result of the game processing, and draws the play image in a frame buffer of the cloud server 10. In this embodiment, the game processing and the generation of the play image are repeatedly executed.
[0039] Then, the cloud server 10 acquires the play image drawn in the frame buffer and encodes the play image to generate image data representing the play image. The cloud server 10 then transmits the generated image data to the terminal 12. The terminal 12 then decodes the image data received from the cloud server 10 and displays the play image generated by the decoding on the display unit 12d.
[0040] In this manner, the cloud server 10 according to the present embodiment delivers, by streaming, moving images generated in accordance with the game play status to the terminal 12 used by the user who is playing the game.
[0041] FIG. 2 is a diagram showing an example of the timing of processes occurring in the cloud server 10 according to the present embodiment.
[0042] In this embodiment, for example, in an initial state, the terminal 12 transmits, at a predetermined period (for example, every 4 milliseconds), pad data P indicating an input operation received by the operation unit 12e at that timing to the cloud server 10. A sequence number indicating the transmission sequence of the pad data P is associated with the pad data P according to this embodiment.
[0043] Fig. 2 shows the timing of reception of the pad data P(0) to P(10) transmitted in this manner at the cloud server 10. Note that in Fig. 2, the sequence numbers associated with the received pad data P are shown as numbers in parentheses in P(0) to P(10). Note that, depending on the communication quality of the computer network 14, the cloud server 10 does not necessarily receive the pad data P at 4 millisecond intervals.
[0044] In the cloud server 10, a play image is generated at a predetermined cycle (for example, a cycle of 16 milliseconds). At this time, the cloud server 10 generates a play image based on an input operation indicated by the pad data P (latest pad data P) received most recently at the timing of starting generation of the play image, and draws the generated play image in a frame buffer.
[0045] The period G shown in FIG. 2 represents the generation period of a play image. In the example of FIG. 2, during the period G(m,n), a play image with sequence number m is generated based on pad data P(n) with sequence number n. That is, in FIG. 2, the generation period of a play image based on pad data P(0) corresponds to the period shown as G(1,0). Moreover, the generation period of a play image based on pad data P(4) corresponds to the period shown as G(2,4). Moreover, the generation period of a play image based on pad data P(8) corresponds to the period shown as G(3,8).
[0046] In this embodiment, the transmission cycle of the pad data P in the terminal 12 is different from the generation cycle of the play image in the cloud server 10. For example, in the initial state, the generation cycle of the play image is four times the transmission cycle of the pad data P. Therefore, not all of the pad data P received by the cloud server 10 is used to generate the play image. In the example of FIG. 2, the pad data P shown as P(1) to P(3), P(5) to P(7), P(9), and P(10) are not used to generate the play image.
[0047] Furthermore, in this embodiment, the cloud server 10 transmits a packet indicating that generation of a play image has started to the terminal 12 at the timing when generation of a play image based on the pad data P has started. Hereinafter, this packet will be referred to as a video sync packet (VSP). In this embodiment, the VSP transmitted at the timing when generation of a play image based on the pad data P has started includes the sequence number of the play image (the above-mentioned m) and the sequence number of the pad data P (the above-mentioned n). In this way, the VSP according to this embodiment is associated one-to-one with the play image.
[0048] 2 shows that a VSP (1,0) is transmitted at the timing when generation of a play image with a sequence number of 1 starts. It also shows that a VSP (2,4) is transmitted at the timing when generation of a play image with a sequence number of 2 starts. It also shows that a VSP (3,8) is transmitted at the timing when generation of a play image with a sequence number of 3 starts.
[0049] In this embodiment, for example, when the drawing of a play image in the frame buffer is completed, the play image is encoded and the image data generated by the encoding is transmitted. Note that the frame buffer of the cloud server 10 according to this embodiment is configured as a multi-buffer, and the next play image can be drawn in parallel with the encoding of the play image that has been drawn.
[0050] The period S shown in FIG. 2 corresponds to the period during which the play image is encoded and the image data is transmitted. In the example of FIG. 2, during the period S(m,n), a play image with sequence number m, which is generated based on pad data P with sequence number n, is encoded and the image data generated by the encoding is transmitted. That is, in FIG. 2, the period during which the play image with sequence number 1 is encoded and the image data generated by the encoding is transmitted corresponds to the period shown as S(1,0). Also, the period during which the play image with sequence number 2 is encoded and the image data generated by the encoding is transmitted corresponds to the period shown as S(2,4).
[0051] In this embodiment, the image data generated by encoding a play image based on pad data P is associated with the sequential number of the pad data P (the above-mentioned n) and the sequential number of the play image (the above-mentioned m).
[0052] In this embodiment, the cloud server 10 measures the time from when the pad data P is received to when the generation of a play image based on the pad data P starts. Then, the cloud server 10 generates interval data indicating the measured time. In this embodiment, the image data generated based on the play image is associated with the interval data generated in this manner.
[0053] Fig. 3 is a diagram showing a schematic example of communication occurring between the cloud server 10 and the terminal 12 from the transmission of pad data P(4) having a sequence number of 4 to the reception of image data generated based on the pad data P(4). In the example of Fig. 3, the image data transmitted during the period shown as S(2,4) in Fig. 2 is shown as D(2,4).
[0054] The terminal 12 according to the present embodiment identifies a packet reception time, which is the time from the timing of transmitting pad data P to the timing of receiving a VSP associated with the pad data P. Hereinafter, the packet reception time will also be expressed as PadVspRTT, as shown in Fig. 3. The terminal 12 according to the present embodiment identifies PadVspRTT based on, for example, the sequence number of the pad data P included in the VSP received from the cloud server 10.
[0055] As shown in FIG. 3, the time from when the terminal 12 transmits pad data P to when it receives the first segment of image data generated based on the pad data P is referred to as PadFirstFragRTT.
[0056] The time from when the terminal 12 transmits pad data P to when it receives the last segment of image data generated based on the pad data P is referred to as PadFrameRTT.
[0057] Also, the time from when the terminal 12 receives the first segment of image data to when it receives the last segment of the image data will be referred to as Transfer Time. Hereinafter, Transfer Time will be represented as TT.
[0058] The terminal 12 according to the present embodiment specifies the value of TT based on image data received from the cloud server 10. Here, for example, the terminal 12 may specify the value of PadFrameRTT and the value of PadFrameRTT based on the sequence number of the pad data P associated with the image data. Then, the terminal 12 may specify the value of TT by subtracting the value of PadFirstFragRTT from the value of PadFrameRTT.
[0059] In this embodiment, when a slice transfer method is adopted in which one play image is divided into a plurality of slices and encoding and image data transmission are performed for each slice, the TT may be specified in consideration of the number of slices per frame. For example, the TT may be specified in consideration of the occurrence of a predetermined non-communication time from the end of the transmission of image data related to one slice to the start of the transmission of image data related to the next slice. In this case, for example, the value obtained by subtracting the value of PadFirstFragRTT from the value of PadFrameRTT, and then further subtracting the value obtained by multiplying the above-mentioned non-communication time by (the number of slices per frame - 1) may be specified as the value of TT.
[0060] As shown in FIG. 3, the time from when the cloud server 10 receives the pad data P to when it starts generating a play image based on the pad data P is referred to as PRecvGStartTime.
[0061] The terminal 12 according to the present embodiment identifies PRecvGStartTime based on the interval data associated with the image data.
[0062] As shown in FIG. 2, the time from when the terminal 12 receives the VSP to when it transmits the next pad data P is called DiffPadVsp.
[0063] The bitrate control performed in the cloud gaming system 1 according to this embodiment will be described below.
[0064] In the cloud gaming service provided by the cloud gaming system 1, it is desirable that the moving images displayed on the terminal 12 have the highest possible image quality. Therefore, it is desirable that the data size of the image data generated based on the play images constituting the moving images is as large as possible.
[0065] However, in wireless communication such as that of this embodiment, which requires immediacy, in order to reduce the discomfort felt by the user during operation, it is important to ensure that each play image is displayed smoothly and with low latency on the terminal 12, even if this means reducing the data size of the image data.
[0066] In addition, mobile communication systems such as 4G and 5G have various issues, such as variations in the quality of wireless communication paths (radio wave strength), communication resources not being allocated due to congestion, and handovers between base stations occurring due to movement. Therefore, in mobile communication systems such as 4G and 5G, there is a tendency for large bandwidth fluctuations to occur, resulting in delays.
[0067] In addition, in a communication environment using Wi-Fi (registered trademark), the communication quality does not vary much between the uplink (communication from the terminal 12 to the cloud server 10 in this embodiment) and the downlink (communication from the cloud server 10 to the terminal 12 in this embodiment). On the other hand, in a communication environment such as 4G or 5G, it is likely that the communication quality of one of the uplink and the downlink is good and the communication quality of the other is poor.
[0068] Therefore, in situations where moving images are transmitted using mobile communication systems such as 4G and 5G, special care must be taken to ensure that frame images are displayed on terminal 12 with low latency.
[0069] Therefore, in this embodiment, the bit rate control shown in FIG. 4 is performed so that image data of an appropriate data size taking delay into consideration is generated in the cloud server 10.
[0070] FIG. 4 is an explanatory diagram illustrating an example of bit rate control performed in the cloud gaming system 1 according to the present embodiment.
[0071] As shown in FIG. 4, the bit rate control according to this embodiment performs two types of control: bandwidth fluctuation tracking control and packet congestion reduction control.
[0072] In bandwidth fluctuation tracking control, first, in order to prevent image data loss even if the communication environment becomes somewhat worse, the bit rate is controlled so that the TT value of each frame approaches half the reciprocal of the frame rate (FPS) of the video to be transmitted, with a margin for jitter. For example, if the frame rate of the video to be transmitted is 60 fps, the bit rate for video communication is controlled so that the TT approaches approximately 8 milliseconds.
[0073] For example, the bit rate value R is determined by PID control with 1 / (FPS×2) as the target value, TT as the current value, and the bit rate value R as the manipulated variable.
[0074] This is shown in Figure 4 as R = PID(TT - (1 / (FPS x 2))). For example, the larger the value of TT - (1 / (FPS x 2)), the smaller the value of R is controlled. Also, when the value of TT is larger than the value of 1 / (FPS x 2), the value of R is controlled to be smaller, and when the value of TT is smaller than the value of 1 / (FPS x 2), the value of R is controlled to be larger.
[0075] In addition, in the bandwidth fluctuation tracking control, in order to quickly track the bandwidth fluctuation, if the packet reception time (PadVspRTT) increases suddenly, the bit rate is immediately further reduced.
[0076] For example, fine noise can be removed by passing the time series data of PadVspRTT through a specified low-pass filter. Hereinafter, PadVspRTT from which fine noise has been removed in this manner will be expressed as FilPadVspRTT. This is shown in Figure 4 as FilPadVspRTT = LPF (PadVspRTT).
[0077] Then, the bit rate reduction amount value D is determined by PD control using the EstBtmLatency value (described later) as a target value, the FilPadVspRTT value as a current value, and the bit rate reduction amount value D as a manipulated variable. Here, PD control is performed instead of PID control in order to quickly follow bandwidth fluctuations.
[0078] This is shown in Figure 4 as D = PD (FilPadVspRTT - EstBtmLatency). For example, the larger the value of FilPadVspRTT - EstBtmLatency, the larger the value of D is controlled to be. Also, when the value of FilPadVspRTT is larger than the value of EstBtmLatency, the value of D is controlled to be larger. Also, when the value of FilPadVspRTT is smaller than the value of EstBtmLatency, the value of D is controlled to be smaller.
[0079] In this embodiment, the value of EstBtmLatency is set to a predetermined value in the initial state. Then, every time PadVspRTT is specified, the absolute value V of the difference between the latest PadVspRTT (PadVspRTT[n]) and the immediately preceding PadVspRTT (PadVspRTT[n-1]) is specified. Then, when a state in which the absolute value V is less than Th1 occurs N times consecutively for a given value N and a predetermined threshold value Th1, the value of EstBtmLatency is updated. For example, the value of EstBtmLatency is updated to the average value of the values of PadVspRTT in the N times consecutively in which a state in which the absolute value V is less than Th1 occurs. This is shown in FIG. 4 as EstBtmLatency=Average(PadVspRTT[n] to PadVspRTT[n-N+1]).
[0080] In this manner, in the present embodiment, the value of PadVspRTT in a state where the bandwidth is relatively stable is set as the value of EstBtmLatency. Therefore, the greater the difference between the current value of PadVspRTT and the value of PadVspRTT in a stable state, the larger the value of D that is determined.
[0081] In this embodiment, the value B obtained by subtracting the bit rate reduction amount value D from the bit rate value R determined as described above is specified as the final bit rate value. However, if the determined value D is negative, the value R is not adjusted, and the value R is specified as the value B as it is. This is shown in Fig. 4 as B = RD (however, if D < 0, then D = 0).
[0082] The bandwidth fluctuation tracking control may be performed at a predetermined timing. For example, the bandwidth fluctuation tracking control may be performed at a predetermined cycle (e.g., every 16 milliseconds). The bandwidth fluctuation tracking control may also be performed in response to the occurrence of a predetermined event (e.g., the reception of the last segment of image data).
[0083] Furthermore, in this embodiment, the terminal 12 monitors reception of VSPs, for example, at the transmission cycle of pad data P. If the number M of VSPs received in the most recent specified time t1 is smaller than a specified threshold Th2, the bandwidth fluctuation tracking control is interrupted. For example, if the number M of VSPs received in the most recent 100 milliseconds is less than 5, or if no VSPs have been received in the most recent 80 milliseconds, the bandwidth fluctuation tracking control is interrupted.
[0084] Then, packet congestion reduction control is executed, which is a process of multiplying the above-mentioned bit rate value B by a predetermined ratio r (r is less than 1). Note that the process executed in the packet congestion reduction control does not have to be a process of multiplying the value B by the ratio r as long as it is a process of reducing the value B, and for example, a process of subtracting a predetermined value from the value B may be executed.
[0085] In the packet congestion reduction control according to this embodiment, once the value B reaches a predetermined lower limit b1, the value B is controlled so as not to become smaller than that limit.
[0086] The packet congestion reduction control may be executed at a predetermined timing. For example, the packet congestion reduction control may be executed at a predetermined cycle (for example, a transmission cycle of pad data P). Also, the packet congestion reduction control may be executed in response to the occurrence of a predetermined event (for example, transmission of pad data P).
[0087] In this embodiment, when the number M of VSP receptions in the most recent predetermined time t1 becomes equal to or greater than a predetermined threshold Th2, the packet congestion reduction control is ended and the bandwidth fluctuation tracking control is resumed.
[0088] In mobile communication systems such as 4G and 5G, retransmission and buffering control is performed thoroughly, so data tends to accumulate in the buffer of a relay device such as a base station. For example, when downlink communication stops in a communication environment such as 4G or 5G, data transmitted from the cloud server 10 tends to accumulate in a relay device such as a base station. The data thus accumulated is transmitted all at once when downlink communication returns to normal.
[0089] Increased delays caused by the discharge of accumulated data and overflow of the receiving buffer of terminal 12 caused by terminal 12 receiving a large amount of data at once are factors that cause loss of received data in terminal 12. In this embodiment, by executing packet congestion reduction control as described above, the amount of data backed up (packet congestion) in computer network 14 when communication becomes impossible due to a deterioration of the communication environment is reduced, making it possible to suppress loss of received data.
[0090] The cloud server 10 according to the present embodiment changes the compression ratio in encoding the play image so that the bit rate of the transmitted video image becomes the value B determined by the bandwidth fluctuation tracking control or the value B updated by the packet congestion reduction control. In this way, in the present embodiment, the data size of the generated image data is controlled.
[0091] As described above, according to this embodiment, image data of an appropriate data size taking delay into consideration is generated in the cloud server 10.
[0092] Furthermore, in this embodiment, as described above, the data size of image data to be transmitted by the cloud server 10 is controlled based on the packet reception time (PadVspRTT). In this way, the bit rate of the video can be immediately reduced in response to a sudden drop in throughput.
[0093] Hereinafter, the pad sync control performed in the cloud gaming system 1 according to this embodiment will be described.
[0094] In the cloud gaming service provided by the cloud gaming system 1, it is desirable that the time from receipt of operation data in the cloud server 10 to the start of generation of a play image be as constant as possible in order to reduce the sense of discomfort felt by the user in the operation.
[0095] However, depending on the communication environment, the time it takes for the operation data transmitted from the terminal 12 to reach the cloud server 10 may vary. This is particularly noticeable in wireless communication using a mobile communication system such as 4G or 5G, which has large bandwidth fluctuations.
[0096] Therefore, the time from when the cloud server 10 receives operation data to when the generation of a gameplay image starts is not within an acceptable range, which may cause a user to feel uncomfortable.
[0097] Therefore, in this embodiment, by performing the pad sync control shown in Figure 5, it is possible to reduce the sense of discomfort felt by the user when a video image generated by the cloud server 10 in response to an operation on the terminal 12 is displayed on the terminal 12.
[0098] FIG. 5 is an explanatory diagram illustrating an example of pad sync control performed in the cloud gaming system 1 according to the present embodiment.
[0099] As shown in FIG. 5, in the pad sync control according to this embodiment, two types of control, a first control and a second control, are performed.
[0100] In the first control, the transmission timing of the pad data P is controlled so that the value indicating DiffPadVsp (see FIG. 2) approaches a predetermined value T1 (for example, a value T1 that is 1.5 times the transmission period of the pad data P).
[0101] Here, for example, the transmission period T of the pad data P is determined by PD control with the value T1 as the target value, the value of DiffPadVsp as the current value, and the transmission period T of the pad data P as the manipulated variable. This is shown in Figure 5 as T = PD (DiffPadVsp - T1). For example, the larger the value of (DiffPadVsp - T1), the smaller the value of T is controlled. Furthermore, when the value of DiffPadVsp is larger than the value of T1, the value of T is controlled to be smaller, and when the value of DiffPadVsp is smaller than the value of T1, the value of T is controlled to be larger.
[0102] Then, the pad data P may be transmitted at the determined transmission cycle T. For example, the pad data P may be transmitted at a timing when a time equivalent to the determined transmission cycle T has elapsed since the timing of the most recent pad data P transmission.
[0103] Then, for example, when it is determined that the value of DiffPadVsp has stabilized, the pad sink control is shifted from the first control to the second control. For example, when a predetermined condition is satisfied, such as a state in which a value indicating a fluctuation in the value of DiffPadVsp (for example, the absolute value of -T1 of DiffPadVsp) is less than a predetermined threshold value Th3 for a predetermined time t2 or more, the pad sink control is shifted from the first control to the second control.
[0104] In the second control, the value T1, which was a fixed value in the first control, is made variable so that the value indicating the time (PRecvGStartTime) from the reception timing of the pad data P in the cloud server 10 to the start timing of generation of a play image based on the pad data P approaches a predetermined value T2 (for example, 1 millisecond). The value T1 in this case is expressed as T1_adj.
[0105] Here, for example, the value T1_adj is determined by PD control with the value T2 as the target value, the value of PRecvGStartTime as the current value, and the value T1_adj as the manipulated variable. This is shown in Figure 5 as T1_adj = PD(PRecvGStartTime-T2). And, for example, the transmission period T of the pad data P is determined by PD control with the value T1_adj as the target value, the value of DiffPadVsp as the current value, and the transmission period T of the pad data P as the manipulated variable. This is shown in Figure 5 as T = PD(DiffPadVsp-T1_adj). In summary, T = PD(DiffPadVsp-PD(PRecvGStartTime-T2)).
[0106] The pad data P may be transmitted at the transmission cycle T determined as described above. For example, the pad data P may be transmitted at a timing when a time equivalent to the determined transmission cycle T has elapsed from the timing of the most recent pad data P transmission.
[0107] Then, for example, when it is determined that the value of DiffPadVsp has become unstable, the pad sink control is shifted from the second control to the first control. For example, when a predetermined condition is satisfied, such as a state in which a value indicating a fluctuation in the value of DiffPadVsp (for example, the absolute value of DiffPadVsp-T1) is equal to or greater than a predetermined threshold value Th3, the pad sink control is shifted from the second control to the first control.
[0108] The pad sync control may be performed at a predetermined timing. For example, the pad sync control may be performed at a predetermined cycle (e.g., every 16 milliseconds). The bandwidth fluctuation tracking control may be performed in response to the occurrence of a predetermined event (e.g., the reception of the last segment of image data).
[0109] As described above, in this embodiment, the pad sync control is used to synchronize the timing of sending pad data P with the timing of generating play images, thereby achieving low latency. In this way, this embodiment reduces the sense of discomfort felt by the user when video images generated by the cloud server 10 in response to operations on the terminal 12 are displayed on the terminal 12.
[0110] Furthermore, in this embodiment, the above-described first control and second control are executed, thereby making it possible to stabilize PRecvGStartTime.
[0111] The functions of the cloud gaming system 1 according to the present embodiment and the processes executed by the cloud gaming system 1 according to the present embodiment will be further described below, focusing on the points described above.
[0112] Fig. 6 is a functional block diagram showing an example of functions implemented in the cloud gaming system 1 according to this embodiment. Note that it is not necessary for all of the functions shown in Fig. 6 to be implemented in the cloud gaming system 1 according to this embodiment, and functions other than the functions shown in Fig. 6 may also be implemented.
[0113] 6, the cloud server 10 according to this embodiment functionally includes, for example, a server-side control data storage unit 20, an operation data receiving unit 22, a frame image generating unit 24, a VSP transmitting unit 26, an encoding process executing unit 28, an image data transmitting unit 30, and a server-side traffic control unit 32. In this embodiment, the cloud server 10 serves as a video transmitting device that sequentially transmits image data representing frame images that make up a video.
[0114] The server-side control data storage unit 20 is implemented mainly using the storage unit 10b. The operation data receiving unit 22, VSP transmitting unit 26, and image data transmitting unit 30 are implemented mainly using the communication unit 10c. The frame image generating unit 24 and server-side traffic control unit 32 are implemented mainly using the processor 10a. The encoding process execution unit 28 is implemented mainly using the processor 10a and the encoder-decoder unit 10d.
[0115] The above functions may be implemented by executing a program including instructions corresponding to the above functions, which is installed in the cloud server 10, which is a computer, on the processor 10a. This program may be supplied to the cloud server 10 via a computer-readable information storage medium, such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet, for example.
[0116] 6, the terminal 12 according to this embodiment functionally includes, for example, a terminal-side control data storage unit 40, an operation data generation unit 42, an operation data transmission unit 44, a VSP reception unit 46, an image data reception unit 48, a decode process execution unit 50, a frame image display control unit 52, a terminal-side traffic control unit 54, and a transmission timing control unit 56. In this embodiment, the terminal 12 serves as a video receiving device that sequentially receives image data representing frame images that make up a video.
[0117] The terminal side control data storage unit 40 is mainly implemented by the storage unit 12b. The operation data generation unit 42 is mainly implemented by the processor 10a and the operation unit 12e. The operation data transmission unit 44, the VSP reception unit 46, and the image data reception unit 48 are mainly implemented by the communication unit 12c. The decode process execution unit 50 is mainly implemented by the processor 10a and the encoder / decoder unit 12h. The frame image display control unit 52 is mainly implemented by the processor 12a and the display unit 12d. The terminal side traffic control unit 54 and the transmission timing control unit 56 are mainly implemented by the processor 10a.
[0118] The above functions may be implemented by executing a program including instructions corresponding to the above functions on the processor 12a, which is installed in the terminal 12, which is a computer. This program may be supplied to the terminal 12 via a computer-readable information storage medium, such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet, for example.
[0119] In this embodiment, the server-side control data storage unit 20 stores, for example, control data indicating the above-mentioned bit rate value B.
[0120] In this embodiment, the operation data receiving unit 22 receives, for example, operation data corresponding to an input operation by a user (for example, the above-mentioned pad data P). The operation data receiving unit 22 receives, for example, operation data corresponding to an input operation during game play.
[0121] In this embodiment, for example, the frame image generating unit 24 generates a frame image based on the operation data received by the operation data receiving unit 22. For example, the frame image generating unit 24 generates a play image representing the play situation of the game being played by the user based on the pad data P.
[0122] In this embodiment, for example, in response to the start of generation of a frame image based on operation data, the VSP transmission unit 26 transmits a VSP, which is a packet associated with the operation data, to the terminal 12. The VSP transmission unit 26 transmits the VSP, for example, at the timing when generation of a frame image starts.
[0123] In this embodiment, for example, the encoding process execution unit 28 generates image data representing a frame image by encoding the frame image generated by the frame image generation unit 24. The encoding process execution unit 28 may determine a compression ratio such that a value indicating a bit rate of the moving image to be transmitted becomes value B indicated by the control data stored in the server-side control data storage unit 20. The encoding process execution unit 28 may then generate image data by encoding the frame image at the determined compression ratio.
[0124] The encoding process execution section 28 may also generate the above-mentioned interval data, and may associate the generated interval data with the generated image data.
[0125] In this embodiment, for example, the image data transmission unit 30 transmits image data generated by the encoding process execution unit 28 to the terminal 12. The image data transmission unit 30 may transmit image data associated with the above-mentioned interval data to the terminal 12.
[0126] In this embodiment, the server-side traffic control unit 32 controls, for example, the data size of image data that is to be transmitted by the image data transmission unit 30 .
[0127] As described above, the server-side traffic control unit 32 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the above-mentioned TT value, which indicates the time required for the terminal 12 to receive the image data.
[0128] As described above, the server-side traffic control unit 32 may also control the data size of the image data to be transmitted by the image data transmission unit 30 based on the packet reception time (PadVspRTT) described above.
[0129] In this embodiment, the terminal side control data storage unit 40 stores, for example, control data indicating the value B of the bit rate described above.
[0130] In the present embodiment, the operation data generating unit 42 generates the above-mentioned operation data in response to an input operation by the user, for example. The operation data generating unit 42 may generate operation data associated with the control data stored in the terminal-side control data storage unit 40.
[0131] In this embodiment, the operation data transmission unit 44 transmits, for example, the operation data generated by the operation data generation unit 42 to the cloud server 10.
[0132] The operation data transmission unit 44 may transmit the operation data associated with the control data. In this case, the server-side traffic control unit 32 may obtain the control data associated with the operation data received by the operation data reception unit 22. Then, the server-side traffic control unit 32 may update the control data stored in the server-side control data storage unit 20 to the obtained control data.
[0133] Note that the operation data transmission unit 44 does not need to transmit the control data in association with the operation data, and may transmit the control data to the cloud server 10 independently of the operation data. The operation data receiving unit 22 may receive the control data transmitted in this manner. In this case, the server-side traffic control unit 32 may update the control data stored in the server-side control data storage unit 20 to the control data received by the operation data receiving unit 22.
[0134] In this embodiment, for example, the VSP receiving unit 46 receives from the cloud server 10 a VSP, which is a packet associated with the operation data, transmitted from the cloud server 10 in response to the start of generation of a frame image based on the operation data.
[0135] In this embodiment, the image data receiving unit 48 receives, for example, image data transmitted from the cloud server 10. As described above, the image data receiving unit 48 may receive image data associated with interval data.
[0136] In this embodiment, for example, the decode process execution unit 50 decodes image data received by the image data receiving unit 48 to generate a frame image (for example, a play image) represented by the image data.
[0137] In this embodiment, the frame image display control unit 52 causes the display unit 12d to display a frame image (for example, a play image) generated by the decode process executing unit 50, for example.
[0138] In this embodiment, the terminal-side traffic control unit 54 controls, for example, the data size of image data that is to be transmitted by the cloud server 10 .
[0139] As described above, the terminal side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the above-mentioned TT value, which indicates the time required to receive the image data transmitted from the cloud server 10.
[0140] As described above, the terminal-side traffic control unit 54 may specify the packet reception time (PadVspRTT) described above, which is the time from when the terminal 12 transmits operation data to when it receives the VSP associated with the operation data. Then, the terminal-side traffic control unit 54 may control the data size of the image data to be transmitted by the image data transmission unit 30 based on the packet reception time.
[0141] The terminal-side traffic control unit 54 may execute the above-mentioned bandwidth fluctuation tracking control. Then, the terminal-side traffic control unit 54 may update the control data stored in the terminal-side control data storage unit 40 to control data indicating the value B determined by executing the bandwidth fluctuation tracking control.
[0142] As described above in the bandwidth fluctuation tracking control, the terminal side traffic control unit 54 may control the data size of image data to be transmitted based on the packet reception time in receiving the latest VSP and the packet reception time in receiving at least one VSP before the latest one. Specifically, for example, the control shown in FIG. 4 as D=PD(FilPadVspRTT-EstBtmLatency) may be executed.
[0143] The terminal-side traffic control unit 54 may also execute the above-mentioned packet congestion reduction control. Then, the terminal-side traffic control unit 54 may update the control data stored in the terminal-side control data storage unit 40 to control data indicating the value B that is updated by executing the packet congestion reduction control.
[0144] As described above in the packet congestion reduction control, when it is determined based on a predetermined condition that the failure to receive a VSP continues, the terminal-side traffic control unit 54 may control to reduce the data size of the image data to be transmitted from the image data transmission unit 30. For example, as described above, when the number M of received VSPs in the most recent predetermined time t1 is smaller than a predetermined threshold value Th2, the packet congestion reduction control may be executed.
[0145] The terminal-side traffic control unit 54 may also hold a bitrate control mode flag indicating the bitrate control mode. For example, when the bitrate control mode flag has a value of 0, the bandwidth fluctuation tracking control may be performed, and when the bitrate control mode flag has a value of 1, the packet congestion reduction control may be performed.
[0146] In this embodiment, the transmission timing control unit 56 controls, for example, the transmission timing of the operation data.
[0147] The transmission timing control unit 56 may control the time from when the VSP is received to when the next operation data is transmitted. For example, the transmission timing control unit 56 may execute a first control to control the timing at which the operation data is transmitted so that the time from when the VSP is received to when the next operation data is transmitted achieves a first goal. For example, in the above example, the first goal corresponds to satisfying a predetermined condition, such as a state in which the value indicating the fluctuation of the DiffPadVsp value is less than a predetermined threshold value Th3 for a predetermined time t2 or more.
[0148] Furthermore, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data. For example, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data associated with the image data received by the image data receiving unit 48.
[0149] Here, for example, the transmission timing control unit 56 may execute a second control to control the timing at which the operation data is transmitted so that the time indicated by the interval data achieves a second target. For example, in the above example, the value of PRecvGStartTime becoming a predetermined value T2 (for example, 1 millisecond) corresponds to the second target.
[0150] Furthermore, as described above, the transmission timing control section 56 may start the second control in response to the achievement of the first target in the first control.
[0151] As described above, the transmission timing control section 56 may control the transmission period T of the operation data to be transmitted based on the interval data.
[0152] Furthermore, it is not necessary for the image data transmission unit 30 to transmit image data associated with interval data. The image data transmission unit 30 may transmit the interval data to the cloud server 10 independently of the image data. Then, the image data receiving unit 48 may receive the interval data transmitted in this manner. Then, the transmission timing control unit 56 may control the timing at which the next operation data is transmitted based on the interval data received from the cloud server 10 in this manner.
[0153] Furthermore, the transmission timing control section 56 may hold a padsync control mode flag indicating the mode of padsync control. For example, when the value of the padsync control mode flag is 0, the first control is executed, and when the value of the padsync control mode flag is 1, the second control is executed.
[0154] The transmission timing control unit 56 may output a transmission command to the operation data transmission unit 44 at the transmission period T determined as described above. Then, the operation data transmission unit 44 may transmit the operation data to the cloud server 10 in response to the reception of the transmission command.
[0155] Hereinafter, an example of the flow of processing in the bandwidth fluctuation tracking control performed in the terminal 12 according to this embodiment will be described with reference to the flow diagram illustrated in FIG.
[0156] First, the terminal side traffic control unit 54 waits until a predetermined execution timing related to the bandwidth fluctuation tracking control arrives (S101).
[0157] When a predetermined execution timing arrives, the terminal side traffic control unit 54 identifies the TT value for the latest image data received by the image data receiving unit 48 (S102).
[0158] Then, the terminal side traffic control unit 54 determines the above-mentioned value R based on the TT determined in the process shown in S102 (S103).
[0159] Then, the terminal side traffic control unit 54 identifies the latest value of PadVspRTT. (S104).
[0160] Then, the terminal side traffic control unit 54 identifies the absolute value V of the difference between the latest PadVspRTT value and the immediately preceding PadVspRTT value (S105).
[0161] Then, the terminal side traffic control unit 54 checks whether or not a state in which the absolute value V is less than Th1 has occurred N times consecutively (S106).
[0162] If it is confirmed that the state in which the absolute value V is less than Th1 has occurred N times consecutively (S106: Y), the terminal side traffic control unit 54 updates the value of EstBtmLatency (S107).
[0163] When it is confirmed that the state where the absolute value V is less than Th1 has not occurred N times consecutively If (S106: N) or the process shown in S107 is executed, the terminal side traffic control unit 54 identifies the value of FilPadVspRTT (S108).
[0164] Then, the terminal side traffic control unit 54 determines the above-mentioned value D based on the value of FilPadVspRTT determined in the process shown in S108 and the latest value of EstBtmLatency (S109).
[0165] Then, the terminal side traffic control unit 54 determines the value B based on the value R determined in the process shown in S103 and the value D determined in the process shown in S109 (S110).
[0166] Then, the terminal side traffic control unit 54 updates the control data stored in the terminal side control data storage unit 40 so that the set value B becomes the value B identified in the processing shown in S109 (S111), and returns to the processing shown in S101.
[0167] Hereinafter, an example of the flow of processing in the packet congestion reduction control performed in the terminal 12 according to this embodiment will be described with reference to the flow diagram illustrated in FIG.
[0168] First, the terminal side traffic control unit 54 waits until a predetermined execution timing related to the packet congestion reduction control arrives (S201).
[0169] When a predetermined execution timing arrives, the terminal-side traffic control unit 54 identifies the bit rate value B indicated by the control data stored in the terminal-side control data storage unit 40 (S202).
[0170] Then, the terminal side traffic control unit 54 checks whether the value B specified in the process shown in S202 is smaller than the lower limit b1 (S203).
[0171] If it is confirmed that the value B is smaller than the lower limit b1 (S203: Y), the process returns to S201.
[0172] If it is confirmed that the value B is not smaller than the lower limit b1 (S203:N), the terminal side traffic control unit 54 determines the new value B to be the value obtained by multiplying the value B determined in the process shown in S202 by a predetermined ratio r (r is less than 1) (S204).
[0173] Then, the terminal side traffic control unit 54 updates the control data stored in the terminal side control data storage unit 40 so that the set value B becomes the value B identified in the processing shown in S204 (S205), and returns to the processing shown in S201.
[0174] Hereinafter, an example of the flow of the bit rate control mode switching process performed in the terminal 12 according to this embodiment will be described with reference to the flow diagram shown in FIG.
[0175] First, the terminal side traffic control unit 54 waits until a predetermined execution timing for switching the bit rate control mode arrives (S301).
[0176] Here, for example, the execution timing may arrive at a predetermined cycle (for example, the transmission cycle T of the operation data). Also, the execution timing may arrive in response to the occurrence of a predetermined event (for example, the transmission of the operation data).
[0177] When a predetermined execution timing arrives, the terminal side traffic control unit 54 identifies the number M of VSP receptions in the most recent predetermined time t1 (S302).
[0178] Then, the terminal side traffic control unit 54 checks the current value of the bit rate control mode flag (S303).
[0179] If the confirmed value of the bit rate control mode flag is 0, the terminal side traffic control unit 54 checks whether the number of receptions M identified in the process shown in S302 is smaller than a predetermined threshold value Th2 (S304).
[0180] If the number of receptions M is equal to or greater than the predetermined threshold value Th2 (S304: N), the process returns to S301.
[0181] If the number of receptions M is smaller than the predetermined threshold value Th2 (S304: Y), the terminal side traffic control unit 54 changes the value of the held bit rate control mode flag to 1 (S305), and returns to the process shown in S301.
[0182] If the value of the current bit rate control mode flag confirmed in the process shown in S303 is 1, the terminal side traffic control unit 54 checks whether the number of receptions M identified in the process shown in S302 is greater than or equal to a predetermined threshold value Th2 (S306).
[0183] If the number of receptions M is smaller than the predetermined threshold value Th2 (S306: N), the process returns to S301.
[0184] If the number of receptions M is equal to or greater than the predetermined threshold value Th2 (S306: Y), the terminal side traffic control unit 54 changes the value of the held bit rate control mode flag to 0 (S307), and returns to the process shown in S301.
[0185] An example of the flow of processing in pad sync control performed in the terminal 12 according to this embodiment will be described below with reference to the flow diagram shown in FIG.
[0186] First, the transmission timing control section 56 waits until a predetermined execution timing related to pad sync control arrives (S401).
[0187] When a predetermined execution timing arrives, the transmission timing control unit 56 identifies the latest value of DiffPadVsp (S402).
[0188] Then, the transmission timing control section 56 checks the current value of the pad sync control mode flag (S403).
[0189] If the value of the confirmed pad sync control mode flag is 0, the transmission timing control section 56 determines a new transmission period T so that the value of DiffPadVsp specified in the process shown in S402 becomes T1 (S404).
[0190] Then, the transmission timing control section 56 determines the absolute value V of the difference between the latest DiffPadVsp value and the value T1 (S405).
[0191] Then, the transmission timing control section 56 identifies the time during which the state in which the absolute value V is less than the predetermined threshold value Th3 continues (S406).
[0192] Then, the transmission timing control section 56 checks whether the time specified in the process shown in S406 has reached a predetermined time (S407).
[0193] If it is not confirmed that the predetermined time has been reached (S407: N), the process returns to S401.
[0194] If it is confirmed that the predetermined time has been reached (S407: Y), the transmission timing control section 56 changes the value of the held pad sync control mode flag to 1 (S408), and returns to the process shown in S401.
[0195] If the current value of the pad sync control mode flag confirmed in the process shown in S403 is 1, the transmission timing control section 56 identifies the value of the interval data associated with the latest image data (S409).
[0196] Then, the transmission timing control section 56 determines a new transmission cycle T based on the value of the interval data identified in the process shown in S409 (S410).
[0197] Then, the transmission timing control section 56 determines the absolute value V of the difference between the latest value of DiffPadVsp and the value T1 (S411).
[0198] Then, the transmission timing control unit 56 checks whether the identified absolute value V is equal to or greater than a predetermined threshold value Th3 (S412).
[0199] If the absolute value V is not equal to or greater than the predetermined threshold value Th3 (S412: N), the process returns to S401.
[0200] If the absolute value V is equal to or greater than the predetermined threshold value Th3 (S412: Y), the transmission timing control section 56 changes the value of the held pad sync control mode flag to 0 (S413), and returns to the process shown in S401.
[0201] It should be noted that the present invention is not limited to the above-described embodiment.
[0202] For example, the interval data may be associated with the VSPs rather than being associated with the image data.
[0203] Furthermore, the frame image generating section 24 may generate frame images having a data size according to the bit rate.
[0204] Furthermore, the scope of application of the present invention is not limited to the cloud gaming system 1.
[0205] Furthermore, the scope of application of the present invention is not limited to computer networks 14 including mobile communication systems such as 4G and 5G. The present invention is also applicable to computer networks 14 in which wireless communication is performed by Wi-Fi (registered trademark) instead of wireless communication via mobile communication systems such as 4G and 5G.
[0206] Furthermore, the specific character strings and numerical values described above and the specific character strings and numerical values in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.
Claims
1. A video receiving device that sequentially receives image data representing frame images constituting a video from a video transmitting device, comprising: an operation data transmission unit that transmits operation data corresponding to an input operation of a user to the video transmitting device; an interval data receiving unit that receives, from the video transmitting device, interval data indicating a time from reception of the operation data at the video transmitting device to a start of generation of the frame image based on the operation data; a transmission timing control unit that controls a timing at which the next operation data is transmitted based on the interval data; A video receiving device comprising:
2. a packet receiving unit for receiving a packet associated with the operation data transmitted from the video transmitting device in response to start of generation of the frame image based on the operation data; the transmission timing control unit controls a time from a timing of receiving the packet to a timing of transmitting the next operation data.
2. The video receiving device according to claim 1.
3. the transmission timing control unit executes a first control for controlling a timing at which the operation data is transmitted so that a time from a timing at which the packet is received until a time at which the next operation data is transmitted achieves a first target, and a second control for controlling a timing at which the operation data is transmitted so that a time indicated by the interval data achieves a second target; the transmission timing control unit starts the second control in response to the first target being achieved in the first control.
3. The video receiving device according to claim 2.
4. the transmission timing control unit controls a transmission cycle of the operation data to be transmitted based on the interval data.
2. The video receiving device according to claim 1.
5. The moving image is generated according to a game play situation, The input operation is an input operation in playing the game.
2. The video receiving device according to claim 1.
6. a video receiving device which sequentially receives image data representing frame images constituting a video from a video transmitting device, and transmits operation data corresponding to an input operation by a user to the video transmitting device; receiving interval data from the video transmitting device, the interval data indicating a time from reception of the operation data at the video transmitting device to a start of generation of the frame image based on the operation data; controlling a timing when the next operation data is transmitted based on the interval data; A control method comprising:
7. A computer sequentially receives frame images constituting a moving image from a moving image transmitting device, transmitting operation data corresponding to an input operation of a user to the video transmitting device; receiving, from the video image transmitting device, interval data indicating a time from reception of the operation data at the video image transmitting device to start of generation of the frame image based on the operation data; a step of controlling a timing at which the next operation data is transmitted based on the interval data; A program characterized by executing the above.
Citation Information
Patent Citations
Game system, game device, game data distribution- transmitting device, image synchronizing system, and information recording medium
JP2000107453A
Distribution control system, distribution control method, and program
JP2016048898A
Game moving image distribution apparatus, game moving image distribution method and game moving image distribution program
JP2019164792A
