Transmission device, transmission method, and program

Abstract

A transmission device, a transmission method, and a program are provided for transmitting image data with appropriately set reference destinations while suppressing network load. An acquisition unit (26) sequentially acquires frame images drawn in a frame buffer. An encoding unit (28) sequentially encodes the encoding units and generates image data, each encoding unit constituting some or all of the frame images. A data transmission unit (30) sequentially transmits the image data. After transmitting the image data, a transmission failure probability estimation unit (38) estimates a transmission failure probability that is the probability of transmission failure of the image data. A reference destination determination unit (42) determines, based on the transmission failure probability of the image data, one or more reference destinations to be used as reference destinations for a given encoding unit from among the encoding units related to the transmitted image data. The encoding unit (28) encodes the given encoding unit with reference to the one or more reference destinations, thereby generating image data.

Description

Technical Field

[0001] The present invention relates to a transmission device, a transmission method, and a program. Background Art

[0002] In the cloud game service technology that has attracted much attention in recent years, frame images depicting game situations in a video game are generated in a cloud server. Then, the image data obtained by encoding the frame images is transmitted from the cloud server to a terminal, and the frame images obtained by decoding the image data are displayed on the terminal. This series of processes is repeatedly executed so that a moving image depicting the game situation in the video game is displayed on the terminal.

[0003] In addition, there has been a technology for compressing the size of image data generated by encoding an image by setting a reference object. For example, the image data obtained by encoding a frame image into a P frame (predictive-frame) that requires a reference object is smaller in size than the image data obtained by encoding the frame image into an I frame (intra frame) that does not require a reference object. Summary of the Invention

[0004] [Technical Problem]

[0005] Due to the influence of server or network overload or unstable connection to the server or network, in some cases, the terminal cannot receive image data. For example, the image data does not reach the terminal or the terminal receives all or part of the corrupted image data.

[0006] Here, when the terminal previously fails to receive the image data that is the reference object for the image data, even when the terminal successfully receives the image data that requires a reference object, the terminal cannot decode the successfully received image data, and as a result, the transmission of the image data performed by the cloud server is wasted.

[0007] However, when always transmitting image data that does not require a reference object and not transmitting the image data that requires a reference object to prevent such wasteful image data transmission, the data size of the transmitted image data is large, and as a result, a load is imposed on the network.

[0008] Note that this generally applies not only to the case of providing a cloud game service, but also to the case where a transmission device corresponding to the above cloud server transmits a moving image.

[0009] In view of the above actual situation, the present invention has been made, and its object is to provide a transmission device, a transmission method, and a program that can achieve the transmission of image data with an appropriate reference object with a small network load.

[0010] [Solution to the Problem]

[0011] To solve the above problems, according to the present invention, there is provided a transmission device, including: an acquisition unit configured to sequentially acquire frame images drawn in a frame buffer; an encoding processing unit configured to sequentially encode encoding units and generate image data, each of the encoding units corresponding to a part or all of one of the frame images; a transmission unit configured to sequentially transmit the image data; a transmission failure probability estimation unit configured to estimate a transmission failure probability after the transmission of the image data, the transmission failure probability being the probability of failure of the transmission of the image data; and a reference object determination unit configured to determine one or more reference objects to be used as a reference object for a given encoding unit from the encoding units related to the transmitted image data with reference to the transmission failure probability of the image data, wherein the encoding processing unit encodes the given encoding unit with reference to the one or more reference objects to generate the image data.

[0012] In one aspect of the present invention, the transmission failure probability estimation unit estimates the transmission failure probability with reference to the time elapsed from the start of the transmission of the image data to the current moment.

[0013] In addition, in one aspect of the present invention, the transmission failure probability estimation unit estimates the transmission failure probability with reference to the effective bandwidth of the communication path for the image data.

[0014] In addition, in one aspect of the present invention, the transmission failure probability estimation unit estimates the transmission failure probability with reference to the number of the transmitted image data for which corresponding acknowledgments have not been received and for which a predetermined timeout period has not elapsed since the transmission.

[0015] In addition, in one aspect of the present invention, the transmission device further includes: a candidate holding unit configured to hold a plurality of candidate data corresponding to the encoding units related to the transmitted image data, wherein the reference object determination unit determines the one or more reference objects from the encoding units corresponding to the candidate data.

[0016] In this aspect, the transmission device further includes: a candidate management unit configured to hold the candidate data in the candidate holding unit, wherein when the number of the candidate data held in the candidate holding unit reaches an upper limit, the candidate management unit holds new candidate data in the candidate holding unit and deletes one piece of the candidate data held in the candidate holding unit from the candidate holding unit.

[0017] In addition, the candidate management unit may delete candidate data determined according to a ratio from the candidate holding unit, where the ratio is the ratio of the number of coding unit blocks for which intra prediction coding is performed to the number of coding unit blocks included in one of the coding units.

[0018] Optionally, the candidate management unit may delete candidate data from the candidate holding unit, and the deleted candidate data is determined according to whether each of the frame images including the coding unit is an image immediately following a scene change.

[0019] Optionally, the candidate management unit may delete candidate data determined according to the coding order of the coding unit from the candidate holding unit.

[0020] Optionally, the candidate management unit may delete candidate data determined according to the order in which the coding unit is referenced from the candidate holding unit.

[0021] Optionally, the candidate management unit may perform control such that a terminal that is a transmission destination of the image data holds candidate data that is a copy of the candidate data held in the candidate holding unit.

[0022] In this regard, whenever the candidate data held in the candidate holding unit changes, the candidate management unit may send an update notification indicating the change to the terminal, so that the candidate data held in the terminal and the candidate data held in the candidate holding unit are synchronized with each other.

[0023] In addition, in one aspect of the present invention, the reference object determination unit determines the one or more reference objects according to the degree of smallness of the data size of the image data, where the image data is generated by referring to one of the coding units related to the transmitted image data and encoding the given coding unit.

[0024] In addition, according to the present invention, there is provided a transmission method, including: an acquisition step of sequentially acquiring frame images drawn in a frame buffer; an encoding process step of sequentially encoding encoding units and generating image data, each of the encoding units corresponding to a part or all of one of the frame images; a transmission step of sequentially transmitting the image data; a transmission failure probability estimation step of estimating, after the transmission of the image data, a transmission failure probability that is the probability of the transmission failure of the image data; and a reference object determination step of determining, with reference to the transmission failure probability of the image data, one or more reference objects to be used as a reference object for a given encoding unit from among the encoding units related to the transmitted image data, wherein, in the encoding process step, the given encoding unit is encoded by referring to the one or more reference objects, thereby generating the image data.

[0025] In addition, according to the present invention, there is provided a program for causing a computer to perform the following operations: an acquisition process of sequentially acquiring frame images drawn in a frame buffer; an encoding process step of sequentially encoding encoding units and generating image data, each of the encoding units corresponding to a part or all of one of the frame images; a transmission process of sequentially transmitting the image data; a transmission failure probability estimation process of estimating, after the transmission of the image data, a transmission failure probability that is the probability of the transmission failure of the image data; and a reference object determination process of determining, with reference to the transmission failure probability of the image data, one or more reference objects to be used as a reference object for a given encoding unit from among the encoding units related to the transmitted image data, wherein, in the encoding process step, the given encoding unit is encoded by referring to the one or more reference objects, thereby generating the image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram showing an exemplary overall configuration of a cloud game system according to an embodiment of the present invention.

[0027] Figure 2 is a schematic diagram showing exemplary candidate management data.

[0028] Figure 3 is a schematic diagram showing exemplary statistical data.

[0029] Figure 4 is an explanatory diagram schematically showing an exemplary general relationship among a transmission failure probability, the number of transmissions sent in parallel, an effective bandwidth, and an acknowledgment (Ack) response time.

[0030] Figure 5 is a chart showing an exemplary determination rule for values of frame scoring data.

[0031] Figure 6 is a diagram showing exemplary decision rules for values of situation scoring data.

[0032] Figure 7 is an explanatory diagram showing an example of how to determine a reference object in this embodiment.

[0033] Figure 8A is a functional block diagram showing exemplary functions implemented in a cloud server according to an embodiment of the present invention.

[0034] Figure 8B is a functional block diagram showing exemplary functions implemented in a terminal according to an embodiment of the present invention.

[0035] Figure 9 is a flowchart showing an exemplary processing flow executed in a cloud server according to an embodiment of the present invention.

[0036] Figure 10 is a flowchart showing an exemplary processing flow executed in a cloud server according to an embodiment of the present invention.

[0037] Figure 11 is a flowchart showing an exemplary processing flow executed in a cloud server according to an embodiment of the present invention. Detailed implementation

[0038] Figure 1 is a schematic diagram showing an exemplary overall configuration of a cloud game system 1 according to an embodiment of the present invention. As Figure 1 shown, the cloud game system 1 according to this embodiment includes a cloud server 10 and a terminal 12, and each of the cloud server 10 and the terminal 12 mainly includes a computer. The cloud server 10 and the terminal 12 are connected to a computer network 14 such as the Internet, so that the cloud server 10 and the terminal 12 can communicate with each other.

[0039] The cloud server 10 according to this embodiment is, for example, a server computer configured to execute a video game program based on a cloud game service. The cloud server 10 transmits a moving image of a game situation depicted in the video game to the terminal 12 used by a user playing the video game.

[0040] As Figure 1 shown, for example, the cloud server 10 includes a processor 10a, a storage unit 10b, a communication unit 10c, and an encoding / decoding unit 10d.

[0041] The processor 10a is, for example, a program control device such as a central processing unit (CPU), and performs various types of information processing according to a program stored in the storage unit 10b. The processor 10a according to the present embodiment further includes a graphics processing unit (GPU), which is configured to draw an image in a frame buffer according to a graphics command or data provided by the CPU.

[0042] The storage unit 10b is, for example, a storage element such as a read-only memory (ROM), a random access memory (RAM), or a solid state drive (SSD). The storage unit 10b stores, for example, a program executed by the processor 10a. In addition, the storage unit 10b according to the present embodiment has a frame buffer, and the GPU included in the processor 10a draws an image in this frame buffer.

[0043] The communication unit 10c is, for example, a communication interface for sending / receiving data to / from a computer such as the terminal 12 via the computer network 14.

[0044] The encoding / decoding unit 10d includes, for example, an encoder and a decoder. The encoder encodes an input image to generate image data indicating the image. In addition, the decoder decodes the input image data to output the image indicated by the image data.

[0045] The terminal 12 according to the present embodiment is, for example, a computer used by a user who uses a cloud game service, such as a video game console, a personal computer, a tablet device, or a smartphone.

[0046] As Figure 1 shown, the terminal 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an encoding / decoding unit 12d, an operation unit 12e, and a display unit 12f.

[0047] The processor 12a is, for example, a program control device such as a CPU, and performs various types of information processing according to a program stored in the storage unit 12b. The processor 12a according to the present embodiment further includes a GPU, which is configured to draw an image in a frame buffer according to a graphics command or data provided by the CPU.

[0048] The storage unit 12b is, for example, a storage element such as a ROM, a RAM, or an SSD. The storage unit 12b stores, for example, a program executed by the processor 12a. In addition, the storage unit 12b according to the present embodiment has a frame buffer, and the GPU included in the processor 12a draws an image in this frame buffer.

[0049] The communication unit 12c is, for example, a communication interface for sending / receiving data to / from a computer such as the cloud server 10 via the computer network 14.

[0050] The encoding / decoding unit 12d includes, for example, an encoder and a decoder. The encoder encodes an input image to generate image data indicating the image. In addition, the decoder decodes the input image data to output the image indicated by the image data.

[0051] The operation unit 12e is, for example, an operation component for performing an input operation on the processor 12a.

[0052] The display unit 12f is, for example, a display device such as a liquid crystal display or an organic electroluminescence (EL) display.

[0053] Note that the terminal 12 does not necessarily need to include a GPU and a frame buffer.

[0054] When a video game-related operation is performed on the terminal 12 according to the present embodiment through the operation unit 12e, an operation signal indicating the operation is sent from the terminal 12 to the cloud server 10. Then, video game processing based on the operation signal is performed in the cloud server 10. Then, a game image, which is a frame image depicting a game situation in the video game affected by the operation signal, is generated, and the game image is drawn in the frame buffer of the cloud server 10. In the present embodiment, the video game processing and the game image generation are repeatedly performed.

[0055] Next, the cloud server 10 sequentially acquires the game images (play images) drawn in the frame buffer and generates image data depicting the game images. Then, the cloud server 10 sends the generated image data to the terminal 12. Subsequently, the terminal 12 controls the display unit 12f to display the game images generated by decoding the image data received from the cloud server 10. In this way, in the present embodiment, a series of game images are displayed on the display unit 12f.

[0056] In the present embodiment, for example, the cloud server 10 sends image data that does not require a reference object (such as an I-frame) or image data that requires a reference object (such as a P-frame) to the terminal 12. Note that here, the image data related to the reference object can be an I-frame or a P-frame.

[0057] In addition, in the present embodiment, up to a predetermined number (for example, 10) of frame images or image data generated with reference to the frame images are held in the memory of the cloud server 10 as candidate data.

[0058] Figure 2 is a diagram showing an exemplary candidate management data for managing candidate data. As Figure 2As shown, the candidate management data includes, for example, a frame number, frame format data, a transmission confirmation flag, a transmission failure probability data, a scene change flag, a reference count data, an intra-frame prediction coding rate data, and a frame score data.

[0059] The candidate management data is data associated with the candidate data, and like the frame number included in the candidate management data, the frame number of the frame image corresponding to the candidate data is set.

[0060] Figure 2 The frame format data included in the candidate management data shown is data indicating the frame format of the image data generated with reference to the frame image. For example, when the image data is an I-frame, the value I is set, and when the image data is a P-frame, the value P is set.

[0061] Included in Figure 2 The transmission confirmation flag included in the candidate management data shown is a flag indicating whether the transmission of the image data generated with reference to the frame image is successful (or failed).

[0062] In this embodiment, when the image data is successfully received, the terminal 12 sends an acknowledgment (Ack) indicating the successful reception of the image data to the cloud server 10. For example, the acknowledgment is associated with the frame number. The value of the transmission confirmation flag is initially 0. Further, when the cloud server 10 receives the acknowledgment, the value of the candidate management data including the frame number associated with the acknowledgment is changed to 1.

[0063] Included in Figure 2 The transmission failure probability data included in the candidate management data shown is data indicating the transmission failure probability of the image data generated with reference to the frame image (hereinafter referred to as "transmission failure probability").

[0064] In this embodiment, for example, it is determined that the form of the function for estimating the transmission failure probability is a function of three variables with parameters, and these parameters include the number of services waiting for a response (hereinafter referred to as "the number of transmissions sent in parallel"), the effective bandwidth of the image data communication path, and the time elapsed from the start of the transmission of the image data to the current moment. Here, for example, the number of transmissions sent in parallel corresponds to the number of image data items for which the corresponding acknowledgment has not been received and which have not exceeded a predetermined timeout time since the start of the transmission.

[0065] Figure 3 is a chart showing exemplary statistical data for determining the form of the above function. As Figure 3 shown, the statistical data includes, for example, a frame number, a transmission success / failure flag, a data on the number of transmissions sent in parallel, an effective bandwidth data, and an Ack response time data.

[0066] In this embodiment, for example, when the cloud server 10 receives an acknowledgement, statistical data including the frame number associated with the acknowledgement is newly registered. For example, the value of the transmission success / failure flag of the statistical data is set to 1.

[0067] In addition, in this embodiment, when a predetermined timeout period has elapsed since the start of transmitting image data, statistical data including the frame number of the image data is newly registered. For example, the value of the transmission success / failure flag of the statistical data is set to 0.

[0068] In this embodiment, the number of transmissions in parallel and the effective bandwidth are continuously monitored. In addition, the value of the number of transmissions in parallel and the value of the effective bandwidth at the time of registering the statistical data are set as the values of the number of transmissions in parallel and the effective bandwidth included in the statistical data.

[0069] In addition, in this embodiment, the specific moment when the transmission of image data starts is recorded. In addition, when the cloud server 10 receives an acknowledgement, the value of the Ack response time data indicating the time from the start of transmission of the image data corresponding to the acknowledgement to the reception of the acknowledgement (hereinafter referred to as "Ack response time") is set as the value of the Ack response time data included in the statistical data corresponding to the image data. Note that for statistical data with a transmission success / failure flag having a value of 0, the value of the Ack response time data may not be set. In addition, for example, for statistical data with a transmission success / failure flag having a value of 0, virtual Ack response time data having a value of the maximum Ack response time value or a predetermined multiple of the maximum Ack response time value may be set.

[0070] In addition, in this embodiment, reference is made to Figure 3 the statistical data illustrated in, and a predetermined rule is used to determine the form of the function for estimating the transmission failure probability. Here, for example, statistical methods such as multiple logistic regression may be used. In this way, in this embodiment, the form of the function is determined with reference to the analysis result of the statistical data.

[0071] Note that for the image data for which the transmission failure probability is to be estimated, since an acknowledgement has not been received yet, the Ack response time itself cannot be measured. Therefore, in this embodiment, although the time elapsed from the start of transmission of the image data to the current moment is different from the Ack response time, since the transmission failure probability of the image data is based on the Ack response time, the transmission failure probability can be indirectly estimated based on the time elapsed from the start of transmission of the image data to the current moment by referring to the statistical information of the Ack response time. In this regard, the Ack response time cannot be processed in the same way as the number of transmissions in parallel and the effective bandwidth.

[0072] In view of the above, in this embodiment, the correspondence between the time elapsed from the start of image data transmission to the current time and the transmission failure probability can be determined with reference to the statistical information of the Ack response time data. For example, the form of a function corresponding to a typical value (such as an average value or a maximum value) or a value in the Ack response time data that is higher than a predetermined percentage (such as 10%) can be determined. For example, the form of a function can be determined such that when the value indicating the time elapsed from the start of image data transmission to the current moment exceeds the typical value, the transmission failure probability increases sharply in an exponential form.

[0073] Figure 4 is an explanatory diagram schematically showing an exemplary general relationship among the transmission failure probability, the number of transmissions sent in parallel, the effective bandwidth, and the Ack response time.

[0074] In Figure 4 the transmission failure probability is represented by a solid line, the effective bandwidth is represented by a dashed line, and the Ack response time is represented by a dotted line.

[0075] Assume that the reason for the failure of image data transmission is the congestion of the image data communication path, which causes the image data to be discarded in the middle of the communication path. In this case, in the congested state, the amount of image data exceeding the processing capacity of communication devices such as switches or routers on the communication path is input to the communication device, resulting in buffer overflow of the communication device. Thus, the excess image data is discarded.

[0076] As Figure 4 shown, as the number of transmissions sent in parallel increases, the effective bandwidth of the communication path reaches its limit. In addition, as the number of transmissions sent in parallel further increases, the length of time that the image data stays in the buffer of a communication device such as a switch or router increases (i.e., the queuing time increases), resulting in an increase in the Ack response time.

[0077] In Figure 4 the range A1 shown, where the effective bandwidth of the communication path has not reached its limit, for example, the queue in the buffer of a communication device such as a switch or router is 0. Next, in Figure 4 the range A2 shown, the effective bandwidth has reached its limit, and the Ack response time increases.

[0078] Then, as the number of transmissions sent in parallel further increases, buffer overflow occurs, resulting in image data discard. In Figure 4 the range A3 shown, the Ack response time has reached its limit, and the transmission failure probability increases sharply.

[0079] Since the above relationship generally holds, it can be said that the number of transmissions sent in parallel, the effective bandwidth, and the time elapsed since the start of transmission of the image data corresponding to the Ack response time can be combined and used as parameters of a function for estimating the transmission failure probability.

[0080] Note that in Figure 4 it seems that only the number of transmissions published in parallel can be referred to for estimating the transmission failure probability. However, in reality, since transcending control based on programmable buffer thresholds / priorities, input / output arbitration control using random numbers, etc. are performed in the buffer flow control of communication devices such as switches or routers, Figure 4 the relationship shown in

[0081] does not always hold.

[0082] In addition, in the present embodiment, each time new statistical data is registered, the form of the function for estimating the transmission failure probability is determined so that the form of the function is updated accordingly.

[0083] In addition, in response to the update of the function form, the value of the transmission failure probability data included in the candidate management data is updated with reference to the updated function.

[0084] Note that the form of the function for estimating the transmission failure probability and its determination method are not particularly limited to any type. For example, the form of the function for estimating the transmission failure probability can be determined to be a function of two variables having two parameters including the current effective bandwidth value / maximum effective bandwidth value and the time elapsed since the start of image data transmission to the current time / maximum Ack response time value.

[0085] In addition, when estimating the transmission failure probability, the number of transmissions sent in parallel at the start of image data transmission can be used instead of the current value of the number of transmissions sent in parallel as described above. In addition, the effective bandwidth at the start of image data transmission can be used instead of the current value of the effective bandwidth as described above.

[0086] Included in Figure 2The scene change flag in the candidate management data shown is a flag indicating whether the frame image is the first frame image after a scene change. For example, when the frame image is the first frame image after a scene change, the value is set to 1, otherwise the value is set to 0.

[0087] Here, for example, when the user is playing a video game, frame images can be generated that are associated with scene change data based on whether there is a scene change from the previous frame. For example, when there is a scene change from the previous frame, the frame image can be associated with scene change data with a value of 1, otherwise it can be associated with scene change data with a value of 0. Thus, the scene change flag value can be set based on the scene change data.

[0088] For example, the frame image associated with the scene change data is generated by a drawing program such as a video game program included in the cloud server 10. Note that the frame image associated with the scene change data can be generated by a game engine.

[0089] In addition, based on the comparison result between the frame image and the frame image of the frame before the frame image, it can be determined whether there is a scene change from the previous frame. Then, the scene change flag value can be set based on the determined result.

[0090] For example, when the value indicating the scene change amount is greater than a predetermined value, it can be determined that there is a scene change from the previous frame, and conversely, it can be determined that there is no scene change from the previous frame.

[0091] Here, for example, an index value indicating the degree of difference or similarity between the previous frame image and the frame image can be used as the value indicating the scene change amount. For example, when the peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) is less than a predetermined value, it can be determined that there is no scene change, and conversely, it can be determined that there is a scene change. Optionally, the amount of change in motion estimation (ME) can be used as the scene change amount. For example, when the value of the ME change amount is greater than a predetermined value, it can be determined that there is a scene change, and conversely, it can be determined that there is no scene change. Optionally, a value indicating the degree of audio discontinuity between the time when the previous frame image is displayed and the time when the frame image of the frame is displayed can be used as the value indicating the scene change amount. For example, when the value is greater than a predetermined value, it can be determined that there is a scene change, and conversely, it can be determined that there is no scene change.

[0092] In addition, a reference reservation rule can be referred to in order to derive and indicate the amount of change in the score value of the content of the frame image to determine whether there is a scene change from the previous frame. For example, it is possible to refer to the amount of change in the score value calculated based on whether the scene change time has arrived or information such as the type of image texture represented in the frame, the distribution of feature points, depth information, the number of objects, the use of mipmap textures for three-dimensional (3D) graphics at each level, the level of detail (LOD), the use of subdivision at each level, the number of characters and symbols, or the type of the represented scene to determine whether there is a scene change. Here, for example, a score value indicating the priority of spatial and temporal details can be calculated.

[0093] Figure 2 The reference count data included in the candidate management data shown is data indicating the number of times of being used as a reference for other image data. In the present embodiment, for example, each time the frame image is referred to, the value of the reference count data included in the candidate management data corresponding to the frame image is incremented by 1.

[0094] Included in Figure 2 The intra-frame prediction coding data included in the candidate management data shown is data indicating the ratio of the number of coded unit blocks that have undergone intra-frame prediction coding to the number of coded unit blocks included in the frame image (hereinafter referred to as "intra-frame prediction coding rate"). In Figure 2 the example, the intra-frame prediction coding rate is expressed as a percentage. For example, in the case where all the coded unit blocks included in the frame image have undergone intra-frame prediction coding, the intra-frame prediction coding data value is set to 100.

[0095] Included in Figure 2 The frame score data included in the candidate management data shown is data indicating the score of the frame image. In the present embodiment, for example, the value of the reference frame score data is referred to determine the candidate data to be deleted from the memory.

[0096] In the present embodiment, for example, the value of the frame score data is determined by referring to the value of the scene change flag, the value of the reference count data, and the value of the intra-frame prediction coding rate data.

[0097] Figure 5 is a diagram showing an exemplary determination rule for the value of the frame score data.

[0098] For example, an initial value a1 with a predetermined value can be first set as the value of the frame score data of the candidate management data corresponding to the new candidate data.

[0099] Then, in the case where the value of the scene change flag of the candidate management data is 1, a predetermined value a2 can be added to the frame score data.

[0100] In addition, for the candidate management data, each time the value of the reference count data increases by 1, a predetermined value a3 can be added to the frame score data.

[0101] In addition, when the value of the intra-prediction coding rate data of the candidate management data in a frame is equal to or greater than a predetermined threshold th, a predetermined value a4 can be added to the frame score data.

[0102] In addition, for the existing candidate management data, each time image data is generated by referring to a new frame image, the frame score data can be subtracted by a predetermined value a5.

[0103] Then, in this embodiment, when the number of candidate data stored in the memory reaches the upper limit and candidate data corresponding to new image data needs to be stored, the candidate data corresponding to the candidate management data with the minimum frame score data value is deleted from the memory. In addition, the candidate management data is also deleted. In this way, the candidate data stored in the memory is updated. Here, when there are multiple pieces of candidate management data with the minimum frame score data value, the candidate data corresponding to the candidate management data corresponding to the most recently referenced frame image (least recently used: LRU) can be preferentially stored in the memory. In this way, the candidate data to be deleted from the memory can be determined according to the order in which the corresponding frame images are referenced.

[0104] In addition, in this embodiment, when a frame image is generated immediately after a scene change, the candidate data corresponding to the frame image before the frame image is deleted from the memory. In addition, the candidate management data corresponding to the candidate data is also deleted.

[0105] Note that when a frame image is generated immediately after a scene change and the intra-prediction coding rate of the frame image is greater than a pre-set value, the candidate data corresponding to the frame image before the frame image can be deleted from the memory. Also in this case, the candidate management data corresponding to the candidate data is deleted.

[0106] In this embodiment, the candidate data segments stored in the memories of the cloud server 10 and the terminal 12 are synchronized with each other. For example, each time the candidate data stored in the memory of the cloud server 10 is changed, an update notification indicating the change is sent to the terminal 12, and the terminal 12 changes the candidate data stored in the memory of the terminal 12 (a copy of the candidate data stored in the memory of the cloud server 10) to match the candidate data stored in the cloud server 10.

[0107] Then, in this embodiment, the reference object of the frame image to be newly encoded is determined from the frame image corresponding to the candidate data.

[0108] Figure 6A diagram showing an exemplary determination rule for values of situation scoring data for determining a reference object.

[0109] First, for each coding unit block (e.g., macroblock) included in a frame image to be newly coded, a predetermined number of frame images corresponding to candidate data are identified in an order starting from the frame image that achieves the minimum data size through reference and coding. At this time, in order to reduce the amount of calculation and processing time, some frame images corresponding to candidate data can be selected by referring to the motion vectors of the coding unit blocks. Then, a predetermined number of frame images can be identified from the selected frame images.

[0110] Then, in this embodiment, for each of the following situations, the data size of the image data generated by coding a new frame image under the same quantization parameter (QP) condition is calculated, for example: (1) the situation where no other frame images are referenced, (2) the situation where a single frame image is referenced, and (3) the situation where multiple frame images are referenced. In situation (2), for each frame image identified for any coding unit block as described above, the data size of the situation of referencing that frame image is calculated. In the case where the number of frame images identified for any coding unit block as described above is n, the data size in each of the n situations is calculated. In situation (3), the data size in the case where each coding unit block references multiple frame images identified as described above is calculated. For example, in the case where the number of coding unit blocks is m and two frame images are identified for each coding unit block, the data size in each of 2 to the power of m situations is calculated.

[0111] Then, for each of the above situations, first, a value b1 obtained by normalizing based on the data size calculated for the situation is determined as the value of the situation score for the situation. Here, for a situation where the calculated data size is small, a larger value can be set as value b1.

[0112] Then, a predetermined value b2 is added to the value of the situation score for the situation of referencing a frame image corresponding to candidate management data with a transmission confirmation flag having a value of 1. Value b2 is preferably a sufficiently large value.

[0113] Then, for each situation, a value b3 based on the transmission failure probability data value of the candidate management data corresponding to the frame image referenced in the situation is added to the value of the situation score. Here, value b3 is less than value b2. Note that value b3 can also be added to the value of the situation score for the situation of candidate management data with a transmission confirmation flag having a value of 0.

[0114] For example, in the case of referring to a frame image corresponding to candidate management data having a transmission failure probability data value less than a predetermined value, the predetermined value b3 can be added to the value of the case score. In addition, for example, a value b3 that decreases as the transmission failure probability data value increases can be added to the value of the case score. In addition, for example, in the case of referring to a plurality of frame images, a representative value (e.g., an average value) of the values of the transmission failure probability data corresponding to the plurality of frame images can be referred to determine the value b3.

[0115] Then, in the present embodiment, for example, referring to the case scores determined for each case as described above, a case corresponding to the encoding scheme of the new frame image can be determined. For example, the case with the highest case score is determined as the case corresponding to the encoding scheme of the new frame image.

[0116] Then, the new frame image is encoded by the encoding scheme corresponding to the determined case, and new image data is generated.

[0117] Here, in the case determined to be the above case (1), no frame image is used as a reference, and new image data (e.g., an I frame) that does not require a reference is generated by referring to the new frame image.

[0118] In the case determined to be case (2), the new frame image is encoded by referring to the single frame image referred to in the case, and new image data is thereby generated.

[0119] In the case determined to be case (3), the new frame image is encoded for each of the plurality of coding unit blocks by referring to the frame images referred to in the case, and new image data is thereby generated.

[0120] As described above, in the present embodiment, when generating image data that requires a reference, the new frame image is encoded by referring to the frame image corresponding to the candidate data determined with reference to the reference transmission failure probability, and new image data is thereby generated. Specifically, for example, the new frame image is encoded by referring to the frame image corresponding to the candidate data having a transmission failure probability of 0 or relatively smaller than the transmission failure probability of other candidate data, thereby generating new image data.

[0121] For example, assume that as Figure 7 shown, six pieces of image data are continuously generated, and the situation of whether the third P frame and subsequent P frames starting from the left have been successfully transmitted is unknown. Note that in Figure 7Among them, the I-frame is represented as I, and the P-frame is represented as P. In this case, the rightmost frame image is usually encoded by referring to the previous P-frame. However, in this embodiment, the rightmost frame image is encoded by referring to the leftmost I-frame that has been confirmed to have been successfully transmitted or the second P-frame starting from the left. In some cases, the rightmost frame image is encoded by referring to the leftmost I-frame and the second P-frame starting from the left. Alternatively, in some cases, the rightmost frame image is encoded as an I-frame without a reference frame image.

[0122] Due to overload or the influence of unstable connection with the cloud server 10 or the computer network 14, the terminal 12 fails to receive image data in some cases. For example, the image data does not reach the terminal 12 or the terminal 12 receives all or part of the damaged image data.

[0123] Here, even if the image data that requires a reference object is successfully received, when the image data serving as the reference object for the said image data fails to be received, the terminal 12 cannot decode the successfully received image data, resulting in a waste of the transmission of the said image data by the cloud server 10.

[0124] However, when preventing such wasteful image data transmission by always transmitting image data that does not require a reference object and not transmitting image data that requires a reference object, the size of the transmitted image data becomes larger, and as a result, a load is imposed on the computer network 14.

[0125] In this embodiment, as described above, a new frame image is encoded by referring to the frame image corresponding to the image data with a low transmission failure probability, thereby generating new image data, and the new image data is transmitted. Thus, according to this embodiment, image data with an appropriate reference object can be transmitted using a smaller network load.

[0126] Although in the above description, the encoding unit, candidate data, reference object, etc. are frame images, the encoding unit, candidate data, reference object, etc. can be strips formed by dividing the frame image.

[0127] Note that in the case where a transmission failure occurs within the unit of the encoding unit block during the image data decoding process on the terminal 12 and the decoded image is thus partially damaged, techniques described in, for example, the international patent PCT / JP2019 / 050907 are used to recover the said image.

[0128] In addition, in this embodiment, even when transmitting a moving image using a connectionless protocol such as the User Datagram Protocol (UDP), a handshake required for estimating the transmission failure probability is also performed.

[0129] Now, the functions implemented in the cloud game system 1 according to this embodiment and the processes executed in the cloud server 10 will be further described.

[0130] Figure 8A and Figure 8B is a functional block diagram showing example functions implemented in the cloud game system 1 according to this embodiment. Note that in the cloud game system 1 according to this embodiment, it is not necessary to implement all the functions shown in Figure 8A and Figure 8B and functions other than those shown in Figure 8A and Figure 8B can be implemented (for example, functions that execute video game processing or game image generation based on operation signals, etc.).

[0131] As Figure 8A shown, the cloud server 10 according to this embodiment functionally includes, for example, a server-side candidate holding unit 20, a candidate management data storage unit 22, a statistical data storage unit 24, an acquisition unit 26, an encoding processing unit 28, a data transmission unit 30, a monitoring unit 32, a signal receiving unit 34, a statistical data management unit 36, a transmission failure probability estimation unit 38, a candidate management unit 40, and a reference object determination unit 42.

[0132] The server-side candidate holding unit 20, the candidate management data storage unit 22, and the statistical data storage unit 24 are implemented mainly by the storage unit 10b. The acquisition unit 26 and the encoding processing unit 28 are implemented mainly by the encoding / decoding unit 10d. The data transmission unit 30 and the signal receiving unit 34 are implemented mainly by the communication unit 10c. The monitoring unit 32, the statistical data management unit 36, the transmission failure probability estimation unit 38, and the reference object determination unit 42 are implemented mainly by the processor 10a. The candidate management unit 40 is implemented mainly by the processor 10a and the communication unit 10c.

[0133] In addition, the above functions are implemented by the processor 10a executing a program installed on the cloud server 10 as a computer, and this program includes instructions corresponding to the above functions. This program is provided to the cloud server 10 through a computer-readable information storage medium such as an optical disc, a magnetic disk, a magnetic tape, a magneto-optical disc, or a flash memory, or through, for example, the Internet.

[0134] In addition, as Figure 8B shown, the terminal 12 according to this embodiment functionally includes, for example, a data receiving unit 50, a decoding processing unit 52, a display control unit 54, a signal transmission unit 56, a candidate synchronization unit 58, and a terminal-side candidate holding unit 60.

[0135] The data receiving unit 50 and the signal transmitting unit 56 are implemented mainly by the communication unit 12c. The decoding processing unit 52 is implemented mainly by the encoding / decoding unit 12d. The display control unit 54 is implemented mainly by the processor 12a and the display unit 12f. The candidate synchronization unit 58 is implemented mainly by the processor 12a and the communication unit 12c. The terminal-side candidate holding unit 60 is implemented mainly by the storage unit 12b.

[0136] In addition, the above functions are implemented by the processor 12a executing a program installed on the terminal 12 which is a computer, and the program includes instructions corresponding to the above functions. The program is provided to the terminal 12 through a computer-readable information storage medium such as an optical disc, a magnetic disk, a magnetic tape, a magneto-optical disc, or a flash memory, or through the Internet, for example.

[0137] The server-side candidate holding unit 20 of the present embodiment holds, for example, a plurality of candidate data corresponding to coding units related to the transmitted image data.

[0138] The candidate management data storage unit 22 of the present embodiment stores, for example, Figure 2 candidate management data associated with the corresponding candidate data shown.

[0139] The statistical data storage unit 24 of the present embodiment stores, for example, Figure 3 the statistical data exemplified in

[0140] The acquisition unit 26 of the present embodiment sequentially acquires, for example, frame images (game images in the above example) drawn in the frame buffer.

[0141] The encoding processing unit 28 of the present embodiment sequentially encodes each coding unit corresponding to part or all of the frame images acquired by the acquisition unit 26, for example, and generates image data. Here, the coding unit is the frame image in the above example. In this case, the candidate data as described above is the frame image or the image data generated with reference to the frame image. In addition, as described above, the coding unit may be a stripe obtained by dividing the frame image. In this case, the candidate data as described above may be the stripe or the image data generated with reference to the stripe.

[0142] The data transmitting unit 30 of the present embodiment sequentially transmits the image data generated by the encoding processing unit 28, for example, to the terminal 12.

[0143] The monitoring unit 32 of the present embodiment monitors, for example, the effective bandwidth of the communication path for the image data and the number of transmissions issued in parallel. Here, as described above, the number of transmissions issued in parallel corresponds to, for example, the number of pieces of image data for which the corresponding acknowledgments have not been received and which have not exceeded a predetermined timeout period since transmission.

[0144] The signal receiving unit 34 of this embodiment receives a signal such as an acknowledgment (Ack) indicating successful reception of image data from the terminal 12.

[0145] The statistical data management unit 36 of this embodiment generates new statistical data and stores the generated statistical data in the statistical data storage unit 24 when, for example, an acknowledgment is received or a predetermined timeout period has elapsed since the start of transmission of the image data. As described above, in the case where an acknowledgment is received, statistical data with a transmission success / failure flag having a value of 1 is generated. In addition, in the case where a predetermined timeout period has elapsed since the start of transmission of the image data, statistical data with a transmission success / failure flag having a value of 0 is generated.

[0146] In addition, the values of the transmission count data and the effective bandwidth data for parallel transmission are set according to the monitoring result of the monitoring unit 32. For example, these values can be set according to the monitoring result when generating statistical data or the monitoring result when transmitting image data.

[0147] In addition, for the statistical data with a value of 1 for the transmission success / failure flag, the value of the Ack response time data is set as described above. For the statistical data with a value of 0 for the transmission success / failure flag, the value of the virtual Ack response time data or the Ack response time data value may not be set as described above.

[0148] The transmission failure probability estimation unit 38 of this embodiment estimates the transmission failure probability after transmitting the image data, which is, for example, the probability that the image data transmission fails. Here, the transmission failure probability estimation unit 38 can estimate the transmission failure probability with reference to the time elapsed from the start of transmission of the image data to the current time. In addition, the transmission failure probability estimation unit 38 can estimate the transmission failure probability with reference to the effective bandwidth of the communication path of the image data. In addition, the transmission failure probability estimation unit 38 can estimate the transmission failure probability with reference to the number of transmitted image data for which the corresponding acknowledgment has not been received and the time since transmission has not exceeded a predetermined timeout period (i.e., the parallel transmission count as described above).

[0149] For example, as described above, the transmission failure probability estimation unit 38 can update the form of the function used to estimate the transmission failure probability when storing new statistical data in the statistical data storage unit 24. Then, the transmission failure probability estimation unit 38 can estimate the transmission failure probability of the image data with reference to the updated function, the current value of the parallel transmission count, the current value of the effective bandwidth, and the time elapsed from the start of transmission of the image data to the current time.

[0150] Here, the transmission failure probability estimation unit 38 can estimate the transmission failure probability of the image data corresponding to each of the multiple pieces of candidate management data stored in the candidate management data storage unit 22. In addition, instead of the current value of the parallel publication transmission number, the transmission number of the parallel publication at the start of the transmission of the image data can be used. In addition, instead of the current effective bandwidth value, the effective bandwidth at the start of the transmission of the image data can be used.

[0151] Then, the transmission failure probability estimation unit 38 can update the value of the transmission failure probability data included in each of the multiple pieces of candidate management data stored in the candidate management data storage unit 22 to the value indicating the transmission failure probability estimated in this way.

[0152] Note that the transmission failure probability estimation unit 38 can set the value of the transmission failure probability data of the candidate management data with a transmission confirmation flag value of 1 to 0.

[0153] In response to the generation of the image data of a new coding unit, the candidate management unit 40 of the present embodiment holds new candidate data corresponding to the coding unit in the server - side candidate holding unit 20, for example.

[0154] In addition, the candidate management unit 40 generates new candidate management data corresponding to the candidate data. Then, the candidate management unit 40 stores the generated candidate management data in the candidate management data storage unit 22. Here, the frame number of the frame image corresponding to the candidate management data is set as the frame number of the candidate management data. Here, in the case where the coding unit is a strip, a number indicating the coding order can be set instead of the frame number.

[0155] In addition, a value indicating the frame format of the coding unit corresponding to the candidate management data is set as the frame format data of the candidate management data. For example, in the case where the coding result obtains an I - frame, I is set as the value of the frame format data, and in the case where the coding result obtains a P - frame, P is set as the value of the frame format data.

[0156] In addition, the value of the transmission confirmation flag of the candidate management data is set to 0. After that, when the signal receiving unit 34 receives a confirmation indicating the successful reception of the image data corresponding to the candidate management data, the candidate management unit 40 updates the value of the transmission confirmation flag of the candidate management data to 1.

[0157] In addition, a predetermined initial value is set for the transmission failure probability data of the candidate management data. Thereafter, when the transmission failure probability estimation unit 38 estimates the transmission failure probability of the image data corresponding to the candidate management data, the candidate management unit 40 updates the value of the transmission failure probability data of the candidate management data to the estimated transmission failure probability value.

[0158] In addition, for each frame image, the candidate management unit 40 determines whether the frame image is an image immediately following a scene change as described above. Then, the candidate management unit 40 sets a value to the scene change flag value of the candidate management data based on the determined result.

[0159] In addition, the value of the reference count data of the candidate management data is set to 0. Whenever the coding unit corresponding to the candidate management data is referenced, the candidate management unit 40 increments the value of the reference count data of the candidate management data by 1.

[0160] In addition, for the coding unit corresponding to the candidate management data, the candidate management unit 40 identifies the ratio (intra prediction coding rate) of the number of coding unit blocks that have undergone intra prediction coding to the number of coding unit blocks included in the coding unit. Then, the value indicating the identified intra prediction coding rate is set to the value of the intra prediction coding rate data of the candidate management data.

[0161] In addition, as Figure 5 shown, the value calculated with reference to the scene change flag value, reference count data value, and intra prediction coding rate data value of the candidate management data is set as the frame score data of the candidate management data. In addition, as described above, for the candidate management data already stored in the candidate management data storage unit 22, each time new image data is generated, the value of the frame score data decreases.

[0162] Then, as described above, when the number of candidate data held in the server - side candidate holding unit 20 reaches the upper limit, the candidate management unit 40 holds new candidate data in the server - side candidate holding unit 20 and deletes one piece of candidate data held in the server - side candidate holding unit 20 from the server - side candidate holding unit 20.

[0163] For example, as described above, the candidate management unit 40 deletes, from the server - side candidate holding unit 20, candidate data corresponding to, for example, the candidate management data having the minimum frame score data value. In addition, the candidate management unit 40 deletes the candidate management data from the candidate management data storage unit 22.

[0164] In this way, candidate data determined based on the intra-prediction coding rate of the reference frame can be deleted. In addition, candidate data determined based on whether the frame image including the coding unit is an image immediately following a scene change can be deleted. In addition, candidate data determined based on the coding order of the coding unit can be deleted. In addition, candidate data determined based on the order in which the coding unit is referenced can be deleted.

[0165] In addition, the candidate management unit 40 performs control such that the terminal-side candidate holding unit 60 of the terminal 12 holds candidate data that is a copy of the candidate data held in the server-side candidate holding unit 20. Here, the candidate management unit 40 can send an update notification indicating the change to the terminal 12 each time the candidate data held in the server-side candidate holding unit 20 changes, so that the candidate data held in the terminal-side candidate holding unit 60 (a copy of the candidate data held in the server-side candidate holding unit 20) is synchronized with the candidate data held in the server-side candidate holding unit 20.

[0166] The reference object determination unit 42 of the present embodiment determines, for example, one or more reference objects from the coding units related to the transmitted image data as the reference objects for a given coding unit (for example, the latest coding unit). The reference object determination unit 42 can determine the reference object with reference to the transmission failure probability of the image data. In addition, the reference object determination unit 42 can determine the reference object based on the degree of smallness of the data size of the image data generated by encoding the given coding unit by referring to the coding units related to the transmitted image data. In addition, the reference object determination unit 42 can determine the reference object from the coding units corresponding to the candidate data held in the server-side candidate holding unit 20.

[0167] As described above, for each of the following cases, the reference object determination unit 42 calculates the data size of the image data generated by encoding the new coding unit under the same QP condition, for example: (1) the case of not referring to other coding units, (2) the case of referring to a single coding unit, and (3) the case of referring to multiple coding units. Then, as Figure 6 shown, the reference object determination unit 42 determines the case corresponding to the coding scheme of the given coding unit based on the degree of smallness of the data size, whether the referenced coding unit has been transmitted, or the transmission failure probability of the referenced coding unit.

[0168] In this case, the encoding processing unit 28 can encode the given coding unit by referring to one or more reference objects corresponding to the case determined by the reference object determination unit 42, thereby generating image data. For example, the encoding processing unit 28 can encode the given coding unit by the coding scheme corresponding to the case determined by the reference object determination unit 42, thereby generating image data.

[0169] The data receiving unit 50 of this embodiment receives, for example, image data transmitted from the cloud server 10.

[0170] The decoding processing unit 52 of this embodiment decodes the image data received by the data receiving unit 50, for example, to generate coding units. For example, the decoding processing unit 52 decodes the image data by a decoding scheme paired with the coding scheme corresponding to the situation determined by the reference object determining unit 42.

[0171] The display control unit 54 of this embodiment controls the display unit 12f of the terminal 12 to display, for example, the coding units generated by the decoding processing unit 52.

[0172] When the reception of the image data has been successful, the signal sending unit 56 of this embodiment sends, for example, a signal such as an acknowledgment indicating the successful reception of the image data to the cloud server 10.

[0173] The candidate synchronization unit 58 of this embodiment receives, for example, an update notification given by the candidate management unit 40. In addition, the candidate synchronization unit 58 changes the candidate data held in the terminal - side candidate holding unit 60 according to the update notification. In this way, the candidate data held in the terminal 12 and the candidate data held in the cloud server 10 are synchronized with each other.

[0174] As described above, the terminal - side candidate holding unit 60 holds candidate data synchronized with the server - side candidate holding unit 20. The candidate data held in the terminal - side candidate holding unit 60 is used by the decoding processing unit 52 for decoding.

[0175] Here, with reference to Figure 9 the flowchart of [reference] describes an exemplary processing flow executed in the cloud server 10 of this embodiment when an acknowledgment is received or when a predetermined timeout period has elapsed since the start of the transmission of the image data.

[0176] In addition, the statistical data management unit 36 generates new statistical data and stores the generated statistical data in the statistical data storage unit 24 (S101).

[0177] Then, the transmission failure probability estimation unit 38 updates the form of the function used to estimate the transmission failure probability (S102).

[0178] Subsequently, the transmission failure probability estimation unit 38 updates the value of the transmission failure probability data included in each of the multiple pieces of candidate management data stored in the candidate management data storage unit 22 (S103). Then, the processing in this processing example ends.

[0179] Next, with reference to Figure 10The flowchart describes an exemplary processing flow for generating and transmitting image data based on coding units executed in the cloud server 10 according to this embodiment.

[0180] First, for each of the various situations described above, the reference object determination unit 42 calculates the data size (S201) of the image data generated by encoding the coding unit under the same QP condition.

[0181] Next, the reference object determination unit 42 determines the value b1 of the situation score corresponding to the data size calculated in the processing of S201 for each situation (S202).

[0182] Then, for the situation where the coding unit referred to the candidate management data with the transmission confirmation flag having a value of 1 is referenced, the reference object determination unit 42 adds a predetermined value b2 to the value of the situation score (S203).

[0183] Subsequently, for each situation, the reference object determination unit 42 adds the value b3 to the value of the situation score (S204), where the value b3 is based on the value of the transmission failure probability data of the candidate management data corresponding to the coding unit referred to in the situation.

[0184] Subsequently, the reference object determination unit 42 determines the situation corresponding to the coding scheme of the coding unit by referring to the value of the situation score for each situation (S205).

[0185] Then, the encoding processing unit 28 encodes the coding unit using the coding scheme corresponding to the situation determined in the processing of S205, thereby generating new image data (S206).

[0186] Subsequently, the data transmission unit 30 transmits the image data generated in the processing of S206 to the terminal 12 (S207).

[0187] Then, the candidate management unit 40 generates candidate management data corresponding to the image data generated in the processing of S206 (S208).

[0188] Subsequently, the candidate management unit 40 checks whether the number of candidate data held in the server - side candidate holding unit 20 has reached a predetermined upper limit (e.g., 10) (S209).

[0189] In the case where the number of candidate data has reached the upper limit (S209: Yes), the candidate management unit 40 determines the candidate data to be deleted by referring to the value of the frame score data included in the candidate management data (S210).

[0190] Next, the candidate management unit 40 deletes the candidate data determined in the process of S210 from the server - side candidate holding unit 20, and deletes the candidate management data corresponding to the candidate data from the candidate management data storage unit 22 (S211). In this case, as described above, an update notification is sent from the cloud server 10 to the terminal 12, and the corresponding candidate data is deleted from the terminal - side candidate holding unit 60.

[0191] When it is confirmed in the process of S209 that the number of candidate data does not reach the predetermined upper limit (S209: No), or when the process of S211 ends, the candidate management unit 40 holds the candidate data corresponding to the image data generated in the process of S206 in the server - side candidate holding unit 20. In addition, the candidate management unit 40 stores the candidate management data generated in the process of S208 in the candidate management data storage unit 22 (S212). Then, the process in this processing example ends. In this case, as described above, an update notification is sent from the cloud server 10 to the terminal 12, and the corresponding candidate data is held in the terminal - side candidate holding unit 60.

[0192] Next, with reference to Figure 11 the flowchart, an exemplary processing flow executed in the cloud server 10 according to the present embodiment in response to the generation of new candidate management data is described.

[0193] First, for the coding unit related to the image data corresponding to the candidate management data, the candidate management unit 40 increases the value of the reference - count data of the candidate management data corresponding to its reference object by 1, and increases the value of the frame - scoring data by the above - mentioned predetermined value a3 (S301).

[0194] Then, the candidate management unit 40 checks whether the value of the scene - change flag included in the candidate management data is 1 (S302).

[0195] When the value is 1 (S302: Yes), the candidate management unit 40 deletes the candidate data related to the frames before the frame corresponding to the coding unit, which is related to the image data corresponding to the candidate management data, from the server - side candidate holding unit 20. In this case, as described above, an update notification is sent from the cloud server 10 to the terminal 12, and the corresponding candidate data is deleted from the terminal - side candidate holding unit 60. In addition, the candidate management unit 40 deletes the candidate management data corresponding to the deleted candidate data from the candidate management data storage unit 22 (S303).

[0196] Then, for candidate management data other than the said candidate management data, the candidate management unit 40 decreases the value of the frame score data by the above-mentioned predetermined value a5 (S304). Then, the processing in this processing example ends. In addition, when it is confirmed in the processing of S302 that the value of the scene change flag included in the candidate management data is 0 (S302: No), for candidate management data other than the said candidate management data, the candidate management unit 40 decreases the value of the frame score data by the above-mentioned predetermined value a5 (S304). Then, the processing in this processing example ends.

[0197] During the processing in S302 of this processing example, it is possible to check whether the following conditions are satisfied: the value of the scene change flag included in the candidate management data is 1, and the value of the intra-prediction coding rate data included in the said candidate management data is greater than a preset value. Then, the processing in S303 and S304 can be executed when it is confirmed that the said conditions are satisfied. Then, when it is confirmed that the said conditions are not satisfied, the processing in S304 can be executed without executing the processing in S303.

[0198] In addition, in the present embodiment, the terminal-side candidate holding unit 60 can hold a number of frame images or candidate data exceeding a predetermined number that is the upper limit in the server-side candidate holding unit 20, and the candidate data is image data generated with reference to the frame images. In this way, even in the case of a delay in the synchronization process, an overflow of the terminal-side candidate holding unit 60 can be prevented.

[0199] Note that the present invention is not limited to the above embodiments.

[0200] For example, some functions implemented in the cloud server 10 can be implemented in the terminal 12.

[0201] In addition, in the present embodiment, the upper limit of the number of candidate data held in the server-side candidate holding unit 20 can be variable instead of a predetermined value.

[0202] In addition, the application scope of the present invention is not limited to the cloud game system 1. For example, the present invention is also applicable to a game system including a server installed at home and a terminal connected to the server through a home network, a public network, a network of a mobile phone operator, etc. That is, the computer network 14 can be a home network, a public network, a network of a mobile phone operator, etc. In addition, the server and the terminal can be connected to each other in a wired or wireless manner.

[0203] In addition, the type of video game to which the present invention is applied is not particularly limited to any type, and the present invention is also applicable to a game system configured to execute a virtual reality (VR) or augmented reality (AR) game, for example.

[0204] In addition, the application scope of the present invention is not limited to video games, and the present invention is generally applicable to the case where a moving image is transmitted by a transmission device corresponding to the above-described cloud server 10.

[0205] In addition, the specific strings and numerical values in the above text and the specific strings and numerical values in the drawings are illustrative, and the present invention is not limited to these strings and numerical values.

Claims

1. A transmission device, comprising: An acquisition unit configured to sequentially acquire frame images drawn in a frame buffer; An encoding processing unit configured to sequentially encode encoding units and generate image data, each of the encoding units corresponding to a part or all of one of the frame images; A transmission unit configured to sequentially transmit the image data; A transmission failure probability estimation unit configured to estimate a transmission failure probability after the transmission of the image data, the transmission failure probability being the probability of failure of the transmission of the image data; And A reference object determination unit configured to determine one or more reference objects to be used as a reference object for a given encoding unit from the encoding units related to the transmitted image data with reference to the transmission failure probability of the image data, wherein the encoding processing unit encodes the given encoding unit by referring to the one or more reference objects, thereby generating the image data.

2. The transmission device according to claim 1, wherein the transmission failure probability estimation unit estimates the transmission failure probability by referring to the time elapsed from the start of the transmission of the image data to the current moment and the statistical information of the Ack response time data.

3. The transmission device according to claim 1 or 2, wherein the transmission failure probability estimation unit estimates the transmission failure probability by referring to the effective bandwidth of the communication path for the image data.

4. The transmission device according to claim 1 or 2, wherein the transmission failure probability estimation unit estimates the transmission failure probability by referring to the number of the transmitted image data for which corresponding acknowledgments have not been received and for which a predetermined timeout period has not elapsed since the transmission.

5. The transmission device according to claim 1 or 2, further comprising: A candidate holding unit configured to hold a plurality of candidate data corresponding to the encoding units related to the transmitted image data, wherein the reference object determination unit determines the one or more reference objects from the encoding units corresponding to the candidate data.

6. The transmission device according to claim 5, further comprising: A candidate management unit configured to hold the candidate data in the candidate holding unit, wherein when the number of the candidate data held in the candidate holding unit reaches an upper limit, the candidate management unit holds new candidate data in the candidate holding unit and deletes one piece of candidate data held in the candidate holding unit from the candidate holding unit.

7. The transmission device according to claim 6, wherein the candidate management unit deletes the candidate data determined according to a ratio, the ratio being the ratio of the number of encoding unit blocks subjected to intra-frame prediction encoding to the number of encoding unit blocks included in one of the encoding units.

8. The transmission device according to claim 6, wherein The candidate management unit deletes the candidate data from the candidate holding unit, and the deleted candidate data is determined according to whether each of the frame images including the encoding unit is an image immediately after a scene change.

9. The transmission device according to claim 6, wherein the candidate management unit deletes candidate data determined according to the encoding order of the encoding unit from the candidate holding unit.

10. The transmission device according to claim 6, wherein the candidate management unit deletes candidate data determined according to the order in which the encoding unit is referenced from the candidate holding unit.

11. The transmission device according to claim 6, wherein the candidate management unit performs control such that a terminal that is a transmission destination of the image data holds candidate data that is a copy of the candidate data held in the candidate holding unit.

12. The transmission device according to claim 11, wherein, Whenever the candidate data held in the candidate holding unit changes, the candidate management unit sends an update notification indicating the change to the terminal, so that the candidate data held in the terminal and the candidate data held in the candidate holding unit are synchronized with each other.

13. The transmission device according to claim 1 or 2, wherein the reference object determination unit determines the one or more reference objects according to the degree of smallness of the data size of the image data, and the image data is generated by referring to one of the encoding units related to the transmitted image data and encoding the given encoding unit.

14. A transmission method, comprising: an acquisition step of sequentially acquiring frame images drawn in a frame buffer; an encoding process step of sequentially encoding encoding units and generating image data, each of the encoding units corresponding to a part or all of one of the frame images; a transmission step of sequentially transmitting the image data; a transmission failure probability estimation step of estimating a transmission failure probability that is the probability of transmission failure of the image data after the transmission of the image data; and a reference object determination step of determining one or more reference objects to be used as a reference object for a given encoding unit from the encoding units related to the transmitted image data with reference to the transmission failure probability of the image data, wherein, in the encoding process step, the given encoding unit is encoded by referring to the one or more reference objects, thereby generating the image data.

15. A program for causing a computer to perform the following operations: an acquisition process of sequentially acquiring frame images drawn in a frame buffer; an encoding process process of sequentially encoding encoding units and generating image data, each of the encoding units corresponding to a part or all of one of the frame images; a transmission process of sequentially transmitting image data; a transmission failure probability estimation process of estimating a transmission failure probability that is the probability of transmission failure of the image data after the transmission of the image data; and a reference object determination process of determining one or more reference objects to be used as a reference object for a given encoding unit from the encoding units related to the transmitted image data with reference to the transmission failure probability of the image data, wherein, in the encoding process process, the given encoding unit is encoded by referring to the one or more reference objects, thereby generating the image data.

Images